LOW-LATENCY, HIGH-BANDWIDTH USE CASES FOR NAHANNI / IVSHMEM

Transcription

1 1 LOW-LATENCY, HIGH-BANDWIDTH USE CASES FOR NAHANNI / IVSHMEM Cam Macdonell, Xiaodi Ke, Adam Wolfe Gordon, Paul Lu University of Alberta [email protected] KVM Forum 2011 August 16, 2011

2 2 Contents 1. For the QEMU/KVM developer Nahanni/ivshmem included as of QEMU , August VMs for Web services Memcached: up to 29% lower inter-vm latencies on workloads 3. VMs for computational science Order-of-magnitude lower latency and higher bandwidth on MPI microbenchmarks Up to 30% faster on MPI application benchmarks (GAMESS, SPEC MPI2007)

3 3 Part 1: What is Nahanni / ivshmem? User-level library for data movement OS bypass: No guest or host OS involvement Memcached, DDI: pointer-based structured data MPI: message passing, stream data Nahanni device looks like a graphics card Guest OS driver (for initialization only) No impact on guests that do not load the driver New Nahanni virtual PCI device in QEMU -device ivshmem,shm=<name>,size=1024! Creates POSIX shared memory on host Cam Macdonell s Ph.D.

4 Host-Guest, Inter-VM Shared Memory 4

5 5 Host-Guest Use Case Copying a file from host into a guest VM Nahanni is faster than Netcat, SCP-HPN, 9P

6 6 Part 2: Web Services in the Cloud Many companies use the cloud for their Web servers Reddit and FourSquare are on Amazon EC2 Memcached is a key-value cache for databases, etc. Used by Facebook, Twitter, others Conclusion: Nahanni reduces look-up latency by 29% on a read-mostly Yahoo Cloud Serving Benchmark (YCSB) workload, for co-located VMs M.Sc. thesis and NetDB 11 paper by Adam Wolfe Gordon

7 7 Nahanni Memcached + YCSB 1 million 1 KB records Each VM ran 12 million operations; 36 million in total Each VM ran 2,000 operations per second

8 8 Nahanni Memcached + YCSB VM 3 can do look-up using pointers and synchronization 1 million 1 KB records Each VM ran 12 million operations; 36 million in total Each VM ran 2,000 operations per second

9 9 Nahanni vs. virtio/bridging (br) vs. virtio/vhost (vh) Average Total Read Latency (ms) No cache (br) memcached (br) Nahanni (br) No cache (vh) memcached (vh) Nahanni (vh) Cache Type CASSANDRA_READ CACHE_READ CACHE_WRITE

10 10 Nahanni vs. virtio/bridging (br) vs. virtio/vhost (vh) Average Total Read Latency (ms) No cache (br) memcached (br) Nahanni (br) No cache (vh) memcached (vh) Nahanni (vh) Cache Type CASSANDRA_READ CACHE_READ CACHE_WRITE Workload: 29% lower latency

11 11 Nahanni vs. virtio/bridging (br) vs. virtio/vhost (vh) Average Total Read Latency (ms) No cache (br) memcached (br) Nahanni (br) No cache (vh) memcached (vh) Nahanni (vh) Cache Type CASSANDRA_READ CACHE_READ CACHE_WRITE Workload: 29% lower latency Read Ops: 73% lower latency

12 12 Part 3: Computational Science in VMs Your HPC application is likely to have an Message- Passing Interface (MPI) version We developed the MPI-Nahanni user-level library Port of MPICH2-Nemesis For co-located VM instances Conclusion: MPI-Nahanni has order-of-magnitude lower latency and higher bandwidth; applications are up to 30% faster. M.Sc. thesis of Xiaodi Ke, Ph.D. thesis of Cam Macdonell

13 13 NetPIPE 2-sided Bandwidth (Nahanni vs. vhost) ($!!! (#!!! *+,-./0/112 *+, *+,-./0/112 *+, (!!!! )!!! &!!! $!!! #!!!!! $ "# #%& #' (&' (#)' *755/879:2;7<=>675? (* )* &$* %(#*

14 14 NetPIPE 2-sided Bandwidth (Nahanni vs. vhost)!"((!((( +,-./ , , )+!"#' $(( %(( #(( "(( (!!"#$ ()(" " %&"'( ()(# # #&"(! ()($ $ '%"#) ()!% *&"#!!% ()&! %'!"'( ()%( &" :;3<8=>?786@ #)'"#!!%'"!+ %#!"$!)"! ")&* "'% #)$(

15 15 NetPIPE 2-sided Bandwidth (full results) D/19G2960C*DH5;<F ($!!! (#!!! (!!!! )!!! &!!! $!!! #!!! *+,-./0/112 *+, *+,-7829:; *+,-54<=;6 *+, >*-*+,-.;?;525.4>*-*+,-54<=;6 *+,-7829:; *+,-54<=;6.4>*-*+,-.;?;525.4>*-*+,-54<=;6!! $ "# #%& #' (&' (#)' *;55/:;@A2B;CDE6;5F (* )* &$* %(#*

16 16 NetPIPE 2-sided Latency (Nahanni vs. vhost) (&! (!! )*+,-./.001 )*+,2/345 )*+,2/345 $&! "&! "!! %&! %!! &!!! " # $" %"# &%" ) :6;<=564> "' #' $"'

17 17 NetPIPE 2-sided Latency (full results) F.5:0;DBG4:;E (&! (!! $&! $!! "&! "!! %&! %!! &! )*+,-./.001 )*+,2/345 )*+,67189: )*+,43;<:5-3=),)*+,-:>:414-3=),)*+,43;<:5 )*+,-./.001 )*+,2/345 )*+,67189: )*+,43;<:5-3=),)*+,-:>:414-3=),)*+,43;<:5!! " # $" %"# &%" ):44.9:?@1A:BCD5:4E "' #' $"'

18 18 SPEC MPI2007 SPEC MPI2007 is an industry benchmark for MPI We ran 9 of 13 applications from the medium input set Part of Cam Macdonell s Ph.D. thesis Conclusions: Performance benefit of MPI-Nahanni grows as the number of processes grows Improvement is proportional to time spent in MPI functions

19 SPEC MPI2007 (8x8) 19

20 % MPI time (y) vs. % Improvement (x) 20

21 % MPI time (y) vs. % Improvement (x) 21

22 % MPI time (y) vs. % Improvement (x) 22

23 23 GAMESS: Non-MPI Communications GAMESS Quantum Chemistry Not MPI; DDI ported to Nahanni

24 24 Concluding Remarks Nahanni / ivshmem is an alternative to other mechanisms (e.g., virtual network, virtio+vhost) for inter-vm, intra-host IPC OS bypass: does not modify or use data movement paths in host or guest VM Supports pointers, non-stream data too Web: low latency for client-server, structured data 29% lower on a YCSB + Nahanni memcached workload Computational science: low latency and high bandwidth for message-passing applications No changes to MPI code. Up to 30% faster on full applications. [email protected]