Description of problem: After the guest is migrated or the memorystate saved/restored the performance of the VM drops significantly. The most notable appears to be increased latency for syscalls on simple read, write, open and close operations. In realworld applications we've seen throughput drop around 25-30%. Version-Release number of selected component (if applicable): -HOST/Hypervisor: Fedora 18 - Kernel 3.7.9-205.fc18.x86_64 qemu-kvm-1.2.2-6.fc18.x86_64 -Guest/VM Fedora 17 - Kernel 3.7.9-101.fc17.x86_64 How reproducible: Always Steps to Reproduce: Method A. (If you ahve 2 hypervisors) 1. Migrate or livemigrate vm to another host 2. Run Benchmarks Method B. (Single Host using Libvirt) 1. virsh save <domain> filename 2. virsh restore filename 3. run benchmarks Actual results: Performance of the VM drops significantly after migration or save/restore. Expected results: Performance same pre/post migration or save/restore Additional info: -Rebooting the OS of the guest/vm restores the performance. -Different Guest Kernel versions on same Distro have no effect. Benchmarking Results ----- PRE Migration/SAVE ----- build-imagemagick Average: 7.19 Seconds Timed PHP Compilation 5.2.9 Average: 17.22 Seconds unixbench-5.1.2/pgms/syscall 10 close COUNT|31588933|1|lps Simple syscall: 0.1075 microseconds Simple read: 0.1905 microseconds Simple write: 0.1902 microseconds Simple stat: 0.6113 microseconds Simple open/close: 1.5888 microseconds ----- POST Migration/Restore ----- build-imagemagick Average: 10.10 Seconds Timed PHP Compilation 5.2.9 Average: 20.60 Seconds unixbench-5.1.2/pgms/syscall 10 close COUNT|21387180|1|lps Simple syscall: 0.1076 microseconds Simple read: 0.3366 microseconds Simple write: 0.3398 microseconds Simple stat: 0.6010 microseconds Simple open/close: 2.0010 microseconds
Chris, please provide the domain XML (sudo virsh dumpxml $vmname) and /var/log/libvirt/qemu/$vmname.log on the source host and destination host.
Cole, I dont have a second Fedora Hypervisor at the moment to do the live migration, but I have reproduced using the SAVE/RESTORE (Method B). Here are the logs and requested info. If you think a live migrate test will help further this along I can install another hypervisor, but this should be sufficient. ----------------------PRE SAVE---------------------- <domain type='kvm' id='2'> <name>fedora17</name> <uuid>119e14ba-aea6-ef32-05ac-871927a7491c</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>2</vcpu> <os> <type arch='x86_64' machine='pc-1.2'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/vms/fedora17.raw'/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <controller type='usb' index='0'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <interface type='network'> <mac address='52:54:00:40:30:44'/> <source network='default'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <input type='mouse' bus='ps2'/> <graphics type='vnc' port='5946' autoport='no' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='cirrus' vram='9216' heads='1'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_t:s0:c534,c844</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c534,c844</imagelabel> </seclabel> </domain> ----------------------POST RESTORE---------------------- <domain type='kvm' id='3'> <name>fedora17</name> <uuid>119e14ba-aea6-ef32-05ac-871927a7491c</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>2</vcpu> <os> <type arch='x86_64' machine='pc-1.2'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/vms/fedora17.raw'/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <controller type='usb' index='0'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <interface type='network'> <mac address='52:54:00:40:30:44'/> <source network='default'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <input type='mouse' bus='ps2'/> <graphics type='vnc' port='5946' autoport='no' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='cirrus' vram='9216' heads='1'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_t:s0:c496,c559</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c496,c559</imagelabel> </seclabel> </domain> -------------/var/log/libvirt/qemu/fedora17.log------------------------ 2013-04-03 14:24:32.683+0000: starting up LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -name fedora17 -S -M pc-1.2 -enable-kvm -m 4096 -smp 2,sockets=2,cores=1,threads=1 -uuid 119e14ba-aea6-ef32-05ac-871927a7491c -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/fedora17.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/vms/fedora17.raw,if=none,id=drive-virtio-disk0,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:40:30:44,bus=pci.0,addr=0x3 -vnc 0.0.0.0:46 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 2013-04-03 14:58:38.984+0000: shutting down qemu: terminating on signal 15 from pid 1251 2013-04-03 14:58:54.084+0000: starting up LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -name fedora17 -S -M pc-1.2 -enable-kvm -m 4096 -smp 2,sockets=2,cores=1,threads=1 -uuid 119e14ba-aea6-ef32-05ac-871927a7491c -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/fedora17.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/vms/fedora17.raw,if=none,id=drive-virtio-disk0,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=25 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:40:30:44,bus=pci.0,addr=0x3 -vnc 0.0.0.0:46 -vga cirrus -incoming fd:22 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There is a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 18 kernel bugs. Fedora 18 has now been rebased to 3.11.4-101.fc18. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 19, and are still experiencing this issue, please change the version to Fedora 19. If you experience different issues, please open a new bug report for those.
Fixed by upstream QEMU commit fc1c4a5 (migration: drop MADVISE_DONT_NEED for incoming zero pages, 2013-10-24). The commit however is only safe to backport to QEMU 1.6 (F20).
qemu-1.6.1-3.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/qemu-1.6.1-3.fc20
Package qemu-1.6.1-3.fc20: * should fix your issue, * was pushed to the Fedora 20 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing qemu-1.6.1-3.fc20' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2013-23618/qemu-1.6.1-3.fc20 then log in and leave karma (feedback).
qemu-1.6.1-3.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report.