Hide Forgot
This is a place holder to keep track of things we need to deal with for 7.2->earlier migration issues Paolo's: Serial thr_ipending APIC wait_for_sipi
Things the static checker flagged that will be a problem for 7.2->7.1 migration: $ ./scripts/vmstate-static-checker.py -s /var/tmp/rhev-7.1z.txt -d /var/tmp/rhev-7.2-rc2.txt --reverse Section "ich9-ahci" Description "ide_bus/error": version error: 2 > 1 Section "ich9-ahci", Description "ide_bus/error": expected field "retry_sector_num", while dest has no further fields Section "ich9-ahci", Description "ahci port": expected field "port.ifs[0]", got "port_state"; skipping rest Section "sdhci-pci" does not exist in dest Section "pci-serial-2x" Description "serial": Entry "Subsections" missing Section "base-sysbus-fdc" Description "fdc": Entry "Subsections" missing Section "base-sysbus-fdc", Description "fdrive": Subsection "fdrive/perpendicular" not found Section "i8042" Description "pckbd": Entry "Subsections" missing Section "isa-serial" Description "serial": Entry "Subsections" missing Section "sysbus-ahci" Description "ide_bus/error": version error: 2 > 1 Section "sysbus-ahci", Description "ide_bus/error": expected field "retry_sector_num", while dest has no further fields Section "sysbus-ahci", Description "ahci port": expected field "port.ifs[0]", got "port_state"; skipping rest Section "generic-sdhci" does not exist in dest Section "fw_cfg_io" does not exist in dest Section "intel-iommu" does not exist in dest Section "SUNW,fdtwo" Description "fdc": Entry "Subsections" missing Section "SUNW,fdtwo", Description "fdrive": Subsection "fdrive/perpendicular" not found Section "mc146818rtc" Description "mc146818rtc": Entry "Subsections" missing Section "fw_cfg_mem" does not exist in dest Section "pci-serial" Description "serial": Entry "Subsections" missing Section "isa-fdc" Description "fdc": Entry "Subsections" missing Section "isa-fdc", Description "fdrive": Subsection "fdrive/perpendicular" not found Section "apic" Description "apic": Entry "Subsections" missing Section "sysbus-fdc" Description "fdc": Entry "Subsections" missing Section "sysbus-fdc", Description "fdrive": Subsection "fdrive/perpendicular" not found Section "pci-serial-4x" Description "serial": Entry "Subsections" missing Section "kvm-apic" Description "apic": Entry "Subsections" missing Section "apic-common" Description "apic": Entry "Subsections" missing
From the list above, we can ignore: ich9-ahci (ahci, migration was unsupported) *sysbus* SUNW*
Serial should be okay now. Failures to migrate due to subsections should really be cases where it's safer to abort migration rather than continue with invalid state on 7.1. apic/kvm-apic/apic-common was tuned to never generate the subsection in practice for TCG, and never at all for KVM. fw_cfg - Section "fw_cfg_io" does not exist in dest. The device type changed from fw_cfg to fw_cfg_io, but vmstate_fw_cfg still has .name="fw_cfg" in 7.2. Should be good unless I'm missing something. mc146818rtc can cause failures where it's overall fine to continue on the destination. We can disable the subsection altogether (return 0 from rtc_irq_reinject_on_ack_count_needed) with small or no loss. Same for floppy. I would disable them for simplicity instead of wasting time debugging later. pckbd was tuned to never generate the subsection in practice. But we can disable the subsection altogether just to err on the side of safety. Some other differences seem to be missing from the list: - cpu_common/exception_index which is okay (only produced if tcg_enabled()) - ide_bus/error for the PIIX4 IDE - known breakage, but only if migrating a VM paused due to I/O error, where the IDE disks are the ones having the error state. Fix hinted in commit message d12b9ff2a4e21587e5b180008c88a0e6e1ec8ac5, but I would just ignore it as it's really a corner case. - parallel: we do not support parallel so it's okay to have it break 7.2->7.1 migration.
(In reply to Paolo Bonzini from comment #5) > Serial should be okay now. Failures to migrate due to subsections should > really be cases where it's safer to abort migration rather than continue > with invalid state on 7.1. I'm not sure about that. I don't yet understand the detail of all of the subsections in the serial code (hw/char/serial.c) but some seem to be as simple as the xmit/receive fifos. That seems to say that even just outputting some text on a console and migrating at just the wrong point would trigger a failed migration, and that doesn't seem acceptable. Dave
The alternative would be data loss on the receiving side, which seems worse. It can be documented in the release notes. I'm more worried about why some differences are missing from the report, even though the ones I remembered are okay.
(In reply to Paolo Bonzini from comment #5) > Some other differences seem to be missing from the list: > > - cpu_common/exception_index which is okay (only produced if tcg_enabled()) cpu_common was not visited by the code which dumps the json; I'll have to check why that is. > - ide_bus/error for the PIIX4 IDE - known breakage, but only if migrating a > VM paused due to I/O error, where the IDE disks are the ones having the > error state. Fix hinted in commit message > d12b9ff2a4e21587e5b180008c88a0e6e1ec8ac5, but I would just ignore it as it's > really a corner case. This has been flagged: Section "ich9-ahci" Description "ide_bus/error": version error: 2 > 1 Section "ich9-ahci", Description "ide_bus/error": expected field "retry_sector_num", while dest has no further fields > - parallel: we do not support parallel so it's okay to have it break > 7.2->7.1 migration. parallel didn't even get compiled.
Thanks for the explanation. Regarding ide_bus/error, I didn't understand that the subsection is flagged only once. In this case we don't support ich9-ahci migration, but piix-ide also has the same subsection and we do support it. Do you guys agree on comment 5 (apart from David's mentioned doubts regarding the serial port)?
Juan told me cpustate isn't registered via vmstate, but via the old method. I'm guessing it's the same for ide. The ide one didn't show up not because it's duplicated, btw. e.g. sysbus has ide too, and that is reported. My view on this is that we'll just have to audit each device specifically to figure whether it'll be safe to migrate by fudging the subsection's 'needed' state. We could also fail migration once and report that the failure was due to a missing subsection, and that a subsequent migration may not have such a state. However, that's cruel to everyone. Failing migration if we could have let it go on without any side effects isn't also optimal -- but then the subsection would not have been introduced in the first place. If we determine that a subsection won't be emitted at all for our use-cases (x86, kvm only), we can proceed, but I think we still keep ourselves open to the possibility of surprises where migration doesn't succeed once in a while due to the state in a new subsection.
> My view on this is that we'll just have to audit each device specifically to > figure whether it'll be safe to migrate by fudging the subsection's 'needed' > state. Note that there is no new device state. This is all old device state that we happened not to migrate. In all cases it's okay to drop the subsection altogether (.needed returns false). We have never migrated the state and the sky won't fall down if we keep not migrating it. In some cases, however (I think serial only) it may be better to have "surprises where migration doesn't succeed once in a while due to the state in a new subsection". In fact, this was the reason why subsections were introduced in the first place, to have backwards migration fail only in very specific and hopefully rare cases: commit 5ee84c3375f9dff8fdee7209195c8c5327a92551 Author: Juan Quintela <quintela> Date: Mon Jul 26 21:38:45 2010 +0200 ide: fix migration in the middle of a bmdma transfer It reintroduces Revert "ide save/restore pio/atapi cmd transfer fields and io buffer" but using subsections. Added bonus is the addition of ide_dummy_transfer_stop to transfer_end_table, that was missing. Signed-off-by: Juan Quintela <quintela> Signed-off-by: Anthony Liguori <aliguori.com> --- commit 50641c5ce258220fc63ba5be1afd1a37f023aba9 Author: Juan Quintela <quintela> Date: Mon Jul 26 21:38:44 2010 +0200 ide: fix migration in the middle of pio operation Signed-off-by: Juan Quintela <quintela> Signed-off-by: Anthony Liguori <aliguori.com>
BTW, I checked 2.1 ... 2.3 and didn't find anything else than the three items above. Also, PIIX IDE indeed uses vmstate_register.
(In reply to Paolo Bonzini from comment #7) > The alternative would be data loss on the receiving side, which seems worse. > It can be documented in the release notes. But that receive side loss already happens when migrating at the moment? If so then we just document that if you use the new machine type that serial is better protected against it. Doesn't our emulated serial port already fail like a real serial port in that a slow guest will lose data if it doesn't pluck stuff off the fifo quickly enough? And anyway, what device are you going to connect to your emulated serial port that's going to survive migration without being interrupted. I'm happy that failing on a migration problem with block storage is the right thing to do; we should never return corrupt memory or corrupt data on storage; serial I'm a bit less fussy about, since serial itself isn't exactly built for reliability.
Fair enough! Then we can tweak the .needed functions for serial as well, with a version comparison against 7.2.0.
I've created bz's for each of the components described above we think are a problem: serial: bug 1215087 mc146818rtc: 1215088 floppy: 1215091 pckbd: 1215092
Now with a brew build for 2.3.0-based qemu available, I re-ran the static checker against 7.0, 7.1 and 7.2. These are the results for 7.0 <-> 7.2 checks. qemu 1.5.3 -> 2.3.0 (rhel / rhev 7.0 -> rhev 7.2) with machine type 7.0 $ ~/src/qemu-kvm/scripts/vmstate-static-checker.py -s 1.5.3-86.el7/rhel70-70.json -d 2.3.0-1.el7/rhev72-70.json Section "ich9-ahci", Description "ahci port": expected field "port_state", got "port.ifs[0]"; skipping rest Section "xlnx,ps7-usb" does not exist in dest Section "sysbus-ahci", Description "ahci port": expected field "port_state", got "port.ifs[0]"; skipping rest Section "sysbus-ehci-usb" does not exist in dest Section "exynos4210-ehci-usb" does not exist in dest -------------- qemu 2.3.0 -> 1.5.3 (rhev 7.2 -> rhel / rhev 7.0) with machine type 7.0 $ ~/src/qemu-kvm/scripts/vmstate-static-checker.py -s 1.5.3-86.el7/rhel70-70.json -d 2.3.0-1.el7/rhev72-70.json --reverse Section "ich9-ahci" Description "ide_bus/error": version error: 2 > 1 Section "ich9-ahci", Description "ide_bus/error": expected field "retry_sector_num", while dest has no further fields Section "ich9-ahci", Description "ahci port": expected field "port.ifs[0]", got "port_state"; skipping rest Section "e1000-82544gc" does not exist in dest Section "secondary-vga" does not exist in dest Section "sdhci-pci" does not exist in dest Section "pci-serial-2x" Description "serial": Entry "Subsections" missing Section "base-sysbus-fdc" does not exist in dest Section "hda-audio" does not exist in dest Section "qxl" Description "vga": Entry "Subsections" missing Section "PIIX4_PM" Description "piix4_pm": Entry "Subsections" missing Section "PIIX4_PM", Description "piix4_pm": expected field "acpi_pci_hotplug.hotplug_select", while dest has no further fields Section "intel-hda-generic" does not exist in dest Section "i8042" Description "pckbd": Entry "Subsections" missing Section "isa-serial" Description "serial": Entry "Subsections" missing Section "usb-redir", Description "usb-redir-ep": Subsection "usb-redir-ep/stream-state" not found Section "sysbus-ahci" Description "ide_bus/error": version error: 2 > 1 Section "sysbus-ahci", Description "ide_bus/error": expected field "retry_sector_num", while dest has no further fields Section "sysbus-ahci", Description "ahci port": expected field "port.ifs[0]", got "port_state"; skipping rest Section "generic-sdhci" does not exist in dest Section "fw_cfg_io" does not exist in dest Section "VGA" Description "vga": Entry "Subsections" missing Section "intel-iommu" does not exist in dest Section "SUNW,fdtwo" Description "fdc": Entry "Subsections" missing Section "SUNW,fdtwo", Description "fdrive": Subsection "fdrive/perpendicular" not found Section "mc146818rtc" Description "mc146818rtc": Entry "Subsections" missing Section "e1000-82545em" does not exist in dest Section "e1000-82540em" does not exist in dest Section "fw_cfg_mem" does not exist in dest Section "pci-serial" Description "serial": Entry "Subsections" missing Section "isa-fdc" Description "fdc": Entry "Subsections" missing Section "isa-fdc", Description "fdrive": Subsection "fdrive/perpendicular" not found Section "apic" Description "apic": Entry "Subsections" missing Section "qxl-vga" Description "vga": Entry "Subsections" missing Section "sysbus-fdc" Description "fdc": Entry "Subsections" missing Section "sysbus-fdc", Description "fdrive": Subsection "fdrive/perpendicular" not found Section "ICH9-LPC" Description "ich9_pm": Entry "Subsections" missing Section "pci-serial-4x" Description "serial": Entry "Subsections" missing Section "kvm-apic" Description "apic": Entry "Subsections" missing Section "e1000" Description "e1000": Entry "Subsections" missing Section "apic-common" Description "apic": Entry "Subsections" missing
Results of 7.1 <-> 7.2 checks with 7.0 machine type: qemu 2.1.2 -> 2.3.0 (rhev 7.1 -> rhev 7.2) with machine type 7.0 $ ~/src/qemu-kvm/scripts/vmstate-static-checker.py -s 2.1.2-23.el7_1.3/rhev71-70.json -d 2.3.0-1.el7/rhev72-70.json Section "ich9-ahci", Description "ahci port": expected field "port_state", got "port.ifs[0]"; skipping rest Section "xlnx,ps7-usb" does not exist in dest Section "sysbus-ahci", Description "ahci port": expected field "port_state", got "port.ifs[0]"; skipping rest Section "fusbh200-ehci-usb" does not exist in dest Section "tegra2-ehci-usb" does not exist in dest Section "sysbus-ehci-usb" does not exist in dest Section "exynos4210-ehci-usb" does not exist in dest -------------- qemu 2.3.0 -> 2.1.2 (rhev 7.2 -> rhev 7.1) with machine type 7.0 $ ~/src/qemu-kvm/scripts/vmstate-static-checker.py -s 2.1.2-23.el7_1.3/rhev71-70.json -d 2.3.0-1.el7/rhev72-70.json --reverse Section "ich9-ahci" Description "ide_bus/error": version error: 2 > 1 Section "ich9-ahci", Description "ide_bus/error": expected field "retry_sector_num", while dest has no further fields Section "ich9-ahci", Description "ahci port": expected field "port.ifs[0]", got "port_state"; skipping rest Section "sdhci-pci" does not exist in dest Section "pci-serial-2x" Description "serial": Entry "Subsections" missing Section "base-sysbus-fdc" Description "fdc": Entry "Subsections" missing Section "base-sysbus-fdc", Description "fdrive": Subsection "fdrive/perpendicular" not found Section "i8042" Description "pckbd": Entry "Subsections" missing Section "isa-serial" Description "serial": Entry "Subsections" missing Section "sysbus-ahci" Description "ide_bus/error": version error: 2 > 1 Section "sysbus-ahci", Description "ide_bus/error": expected field "retry_sector_num", while dest has no further fields Section "sysbus-ahci", Description "ahci port": expected field "port.ifs[0]", got "port_state"; skipping rest Section "generic-sdhci" does not exist in dest Section "fw_cfg_io" does not exist in dest Section "intel-iommu" does not exist in dest Section "SUNW,fdtwo" Description "fdc": Entry "Subsections" missing Section "SUNW,fdtwo", Description "fdrive": Subsection "fdrive/perpendicular" not found Section "mc146818rtc" Description "mc146818rtc": Entry "Subsections" missing Section "fw_cfg_mem" does not exist in dest Section "pci-serial" Description "serial": Entry "Subsections" missing Section "isa-fdc" Description "fdc": Entry "Subsections" missing Section "isa-fdc", Description "fdrive": Subsection "fdrive/perpendicular" not found Section "apic" Description "apic": Entry "Subsections" missing Section "sysbus-fdc" Description "fdc": Entry "Subsections" missing Section "sysbus-fdc", Description "fdrive": Subsection "fdrive/perpendicular" not found Section "pci-serial-4x" Description "serial": Entry "Subsections" missing Section "kvm-apic" Description "apic": Entry "Subsections" missing Section "apic-common" Description "apic": Entry "Subsections" missing --------------
Finally, results of 7.1 <-> 7.2 checks with 7.1 machine type: qemu 2.1.2 -> 2.3.0 (rhev 7.1 -> rhev 7.2) with machine type 7.1 $ ~/src/qemu-kvm/scripts/vmstate-static-checker.py -s 2.1.2-23.el7_1.3/rhev71-71.json -d 2.3.0-1.el7/rhev72-71.json Section "ich9-ahci", Description "ahci port": expected field "port_state", got "port.ifs[0]"; skipping rest Section "xlnx,ps7-usb" does not exist in dest Section "sysbus-ahci", Description "ahci port": expected field "port_state", got "port.ifs[0]"; skipping rest Section "fusbh200-ehci-usb" does not exist in dest Section "tegra2-ehci-usb" does not exist in dest Section "sysbus-ehci-usb" does not exist in dest Section "exynos4210-ehci-usb" does not exist in dest -------------- qemu 2.3.0 -> 2.1.2 (rhev 7.2 -> rhev 7.1) with machine type 7.1 $ ~/src/qemu-kvm/scripts/vmstate-static-checker.py -s 2.1.2-23.el7_1.3/rhev71-71.json -d 2.3.0-1.el7/rhev72-71.json --reverse Section "ich9-ahci" Description "ide_bus/error": version error: 2 > 1 Section "ich9-ahci", Description "ide_bus/error": expected field "retry_sector_num", while dest has no further fields Section "ich9-ahci", Description "ahci port": expected field "port.ifs[0]", got "port_state"; skipping rest Section "sdhci-pci" does not exist in dest Section "pci-serial-2x" Description "serial": Entry "Subsections" missing Section "base-sysbus-fdc" Description "fdc": Entry "Subsections" missing Section "base-sysbus-fdc", Description "fdrive": Subsection "fdrive/perpendicular" not found Section "i8042" Description "pckbd": Entry "Subsections" missing Section "isa-serial" Description "serial": Entry "Subsections" missing Section "sysbus-ahci" Description "ide_bus/error": version error: 2 > 1 Section "sysbus-ahci", Description "ide_bus/error": expected field "retry_sector_num", while dest has no further fields Section "sysbus-ahci", Description "ahci port": expected field "port.ifs[0]", got "port_state"; skipping rest Section "generic-sdhci" does not exist in dest Section "fw_cfg_io" does not exist in dest Section "intel-iommu" does not exist in dest Section "SUNW,fdtwo" Description "fdc": Entry "Subsections" missing Section "SUNW,fdtwo", Description "fdrive": Subsection "fdrive/perpendicular" not found Section "mc146818rtc" Description "mc146818rtc": Entry "Subsections" missing Section "fw_cfg_mem" does not exist in dest Section "pci-serial" Description "serial": Entry "Subsections" missing Section "isa-fdc" Description "fdc": Entry "Subsections" missing Section "isa-fdc", Description "fdrive": Subsection "fdrive/perpendicular" not found Section "apic" Description "apic": Entry "Subsections" missing Section "sysbus-fdc" Description "fdc": Entry "Subsections" missing Section "sysbus-fdc", Description "fdrive": Subsection "fdrive/perpendicular" not found Section "pci-serial-4x" Description "serial": Entry "Subsections" missing Section "kvm-apic" Description "apic": Entry "Subsections" missing Section "apic-common" Description "apic": Entry "Subsections" missing --------------
OK, so I think I've posted all the simple PC devices: rtc serial kbd floppy that leaves the rdma (1210715) code that I'm waiting to go in upstream. The APIC changes are belived to only be with kernel_irqchip=off which doesn't normally happen. So other than waiting for RDMA I think we're good.
I test rdma migration: 1) boot guest on 7.2 host, do ping-pong migration 7.2<->7.1 for 5 times 2) boot guest on 7.1 host, do ping-pong migration 7.1<->7.2 for 5 times RHEL7.2 host: kernel-3.10.0-304.el7.x86_64 qemu-kvm-rhev-2.3.0-18.el7.x86_64 RHEL7.1 host: kernel-3.10.0-229.13.1.el7.x86_64 qemu-kvm-rhev-2.1.2-23.el7_1.8.x86_64 RHEL7.2 guest: kernel-3.10.0-302.el7.x86_64 The result is as following: Host Machine type Result ------------------------------------------------ 7.2<->7.1 rhel7.1.0 pass 7.2<->7.1 rhel7.0.0 pass 7.2<->7.1 rhel6.6.0 pass 7.2<->7.1 rhel6.5.0 pass 7.1<->7.2 rhel7.1.0 pass 7.1<->7.2 rhel7.0.0 pass 7.1<->7.2 rhel6.6.0 pass 7.1<->7.2 rhel6.5.0 pass Steps to Test: 1. boot rhel7.2 guest on src host 2. boot the guest on dst host with "-incoming rdma:0:5800" 3. inside guest, do some operation: # dd if=/dev/random of=/home/file bs=1M count=4096 # netperf -H 10.66.9.152 -l 600 4. do rdma migration: (qemu) migrate_set_speed 40G (qemu) migrate_set_capability rdma-pin-all on (qemu) migrate rdma:192.168.0.21:5800 5. inside guest, ping other host, do reboot and shutdown. The command line is: /usr/libexec/qemu-kvm \ -name rhel7 \ -S -machine pc-i440fx-rhel7.1.0,accel=kvm,usb=off \ -realtime mlock=off \ -cpu SandyBridge \ -sandbox off \ -m 3840 \ -smp 8,cores=1,threads=1,sockets=8 \ -object memory-backend-ram,policy=bind,id=mem-0,size=256M,prealloc=yes,host-nodes=0 -numa node,memdev=mem-0,cpus=0,cpus=1 \ -object memory-backend-ram,policy=bind,id=mem-1,size=512M,prealloc=yes,host-nodes=0 -numa node,memdev=mem-1,cpus=2-3 \ -object memory-backend-ram,policy=bind,id=mem-2,size=1024M,prealloc=yes,host-nodes=0 -numa node,memdev=mem-2,cpus=4 \ -object memory-backend-ram,policy=bind,id=mem-3,size=2048M,prealloc=yes,host-nodes=0 -numa node,memdev=mem-3,cpus=5-7 \ -uuid 49a3438a-70a3-4ba8-92ce-3a05e0934608 \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/rhel7.monitor,server,nowait \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=utc,driftfix=slew \ -global kvm-pit.lost_tick_policy=discard \ -no-hpet \ -no-shutdown \ -global PIIX4_PM.disable_s3=1 \ -global PIIX4_PM.disable_s4=1 \ -boot order=c,menu=on,splash-time=3000,strict=on \ -device ich9-usb-ehci1,id=usb0,bus=pci.0,addr=0x5.0x7 \ -device ich9-usb-uhci1,masterbus=usb0.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 \ -device ich9-usb-uhci2,masterbus=usb0.0,firstport=2,bus=pci.0,addr=0x5.0x1 \ -device ich9-usb-uhci3,masterbus=usb0.0,firstport=4,bus=pci.0,addr=0x5.0x2 \ -device virtio-scsi-pci,id=scsi0,cmd_per_lun=234,bus=pci.0,addr=0x8 \ -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 \ -drive file=/usr/share/virtio-win/virtio-win.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= \ -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 \ -drive file=/usr/share/virtio-win/virtio-win_amd64.vfd,if=none,id=drive-fdc0-0-0,format=raw,cache=none \ -global isa-fdc.driveA=drive-fdc0-0-0 \ -drive file=/home/rhel7.2.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,serial=f65effa5-90a6-47f2-8487-a9f64c95d4f5,cache=none,discard=unmap,werror=stop,rerror=stop,aio=threads \ -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \ -drive file=/home/virtio-scsi-blk,if=none,id=drive-scsi-disk,format=qcow2,cache=writethrough,werror=stop,rerror=stop \ -device virtio-scsi-pci,id=scsi1,addr=0x13 \ -device scsi-hd,drive=drive-scsi-disk,bus=scsi1.0,id=data-disk2 \ -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 \ -chardev spicevmc,id=charchannel0,name=vdagent \ -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 \ -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channel/target/rhel7.org.qemu.guest_agent.0,server,nowait \ -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 \ -device usb-tablet,id=input0 \ -device intel-hda,id=sound0,bus=pci.0,addr=0x4 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -device intel-hda,id=sound1,bus=pci.0,addr=0x6 \ -device hda-micro,id=sound1-codec0,bus=sound1.0 \ -device intel-hda,id=sound2,bus=pci.0,addr=0x17 \ -device hda-output,id=sound2-codec0,bus=sound2.0,cad=0 \ -device ich9-intel-hda,id=sound3,bus=pci.0,addr=0xc \ -device hda-duplex,id=sound3-codec0,bus=sound3.0,cad=0 \ -chardev spicevmc,id=charredir0,name=usbredir \ -device usb-redir,chardev=charredir0,id=redir0 \ -chardev spicevmc,id=charredir1,name=usbredir \ -device usb-redir,chardev=charredir1,id=redir1 \ -chardev spicevmc,id=charredir2,name=usbredir \ -device usb-redir,chardev=charredir2,id=redir2 \ -chardev spicevmc,id=charredir3,name=usbredir \ -device usb-redir,chardev=charredir3,id=redir3 \ -device usb-host,id=hostdev0 \ -device pvpanic,ioport=1285 -msg timestamp=on \ -netdev tap,id=hostnet0,id=hostnet0,script=/etc/qemu-ifup \ -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=b6:af:42:b8:46:18,bus=pci.0,addr=0x14 \ -netdev tap,id=hostnet1,script=/etc/qemu-ifup \ -device e1000,netdev=hostnet1,id=virtio-net-pci1,mac=b6:2f:a8:85:72:6c,bus=pci.0,addr=0x15,multifunction=off \ -netdev tap,id=hostnet2,vhost=on,script=/etc/qemu-ifup \ -device rtl8139,netdev=hostnet2,id=virtio-net-pci2,mac=4e:63:28:bc:b1:25,bus=pci.0,addr=0x16,multifunction=off \ -netdev tap,id=hostnet3,vhost=on,script=/etc/qemu-ifup \ -device e1000-82540em,netdev=hostnet3,id=virtio-net-pci3,mac=4e:63:28:bc:b1:35,bus=pci.0,addr=0x18,multifunction=off \ -netdev tap,id=hostnet4,vhost=on,script=/etc/qemu-ifup \ -device e1000-82544gc,netdev=hostnet4,id=virtio-net-pci4,mac=4e:63:28:bc:b1:45,bus=pci.0,addr=0x19,multifunction=off \ -netdev tap,id=hostnet5,vhost=on,script=/etc/qemu-ifup \ -device e1000-82545em,netdev=hostnet5,id=virtio-net-pci5,mac=4e:63:28:bc:b1:55,bus=pci.0,addr=0x1a,multifunction=off \ -drive file=/home/ide-disk,if=none,id=drive-data-disk,format=raw,cache=none,aio=native,werror=stop,rerror=stop,copy-on-read=off,serial=fux-ide,media=disk \ -device ide-drive,drive=drive-data-disk,id=system-disk,logical_block_size=512,physical_block_size=512,min_io_size=32,opt_io_size=64,discard_granularity=512,ver=fuxc-ver,bus=ide.0,unit=0 \ -device ich9-usb-uhci6,id=uhci6,bus=pci.0,addr=0xa \ -device usb-kbd,id=kdb0,bus=uhci6.0 \ -device ich9-usb-uhci5,id=uhci5,bus=pci.0,addr=0xb \ -device usb-mouse,id=mouse0,bus=uhci5.0 \ -device nec-usb-xhci,id=xhci,bus=pci.0,addr=0xd \ -device usb-bot,id=bot1,bus=xhci.0 \ -drive file=/usr/share/virtio-win/virtio-win.iso,if=none,id=usb-cdrom1,format=raw \ -device scsi-cd,bus=bot1.0,scsi-id=0,lun=1,drive=usb-cdrom1,id=usb-cdrom1 \ -drive file=/home/bot-disk1,id=usb-disk1,if=none,format=qcow2 \ -device scsi-hd,bus=bot1.0,scsi-id=0,lun=0,drive=usb-disk1,id=usb-disk1 \ -device usb-ehci,id=ehci,bus=pci.0,addr=0xe \ -device usb-bot,id=bot2,bus=ehci.0 \ -drive file=/usr/share/virtio-win/virtio-win.iso,if=none,id=usb-cdrom2,format=raw \ -device scsi-cd,bus=bot2.0,scsi-id=0,lun=1,drive=usb-cdrom2,id=usb-cdrom2 \ -drive file=/home/bot-disk2,id=usb-disk2,if=none,format=qcow2 \ -device scsi-hd,bus=bot2.0,scsi-id=0,lun=0,drive=usb-disk2,id=usb-disk2 \ -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0xf \ -device usb-bot,id=bot3,bus=usb.0 \ -drive file=/usr/share/virtio-win/virtio-win.iso,if=none,id=usb-cdrom3,format=raw \ -device scsi-cd,bus=bot3.0,scsi-id=0,lun=1,drive=usb-cdrom3,id=usb-cdrom3 \ -drive file=/home/bot-disk3,id=usb-disk3,if=none,format=qcow2 \ -device scsi-hd,bus=bot3.0,scsi-id=0,lun=0,drive=usb-disk3,id=usb-disk3 \ -device ich9-usb-uhci3,id=uhci,bus=pci.0,addr=0x10 \ -device usb-storage,drive=drive-usb-0,id=usb-0,removable=on,bus=uhci.0,port=1 \ -drive file=/home/usb-uhci,if=none,id=drive-usb-0,media=disk,format=qcow2 \ -device ich9-usb-ehci1,id=ehci1,bus=pci.0,addr=0x11 \ -device usb-storage,drive=drive-usb-1,id=usb-1,removable=on,bus=ehci1.0,port=1 \ -drive file=/home/usb-ehci,if=none,id=drive-usb-1,media=disk,format=qcow2 \ -device nec-usb-xhci,id=xhci1,bus=pci.0,addr=0x12 \ -device usb-storage,drive=drive-usb-2,id=usb-2,removable=on,bus=xhci1.0,port=1 \ -drive file=/home/usb-xhci,if=none,id=drive-usb-2,media=disk,format=qcow2 \ -device usb-ccid,id=ccid0 \ -watchdog ib700 -watchdog-action reset \ -device virtio-rng-pci,id=rng0,bus=pci.0,addr=0x1c \ -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x1d \ -monitor stdio \ -qmp tcp:0:4466,server,nowait -serial unix:/tmp/ttym,server,nowait \ -spice port=5910,addr=0.0.0.0,disable-ticketing,seamless-migration=on \ -k en-us \ -device qxl-vga,id=video0,ram_size=134217728,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2
(In reply to huiqingding from comment #21) > I test rdma migration: > 1) boot guest on 7.2 host, do ping-pong migration 7.2<->7.1 for 5 times > 2) boot guest on 7.1 host, do ping-pong migration 7.1<->7.2 for 5 times > > RHEL7.2 host: > kernel-3.10.0-304.el7.x86_64 > qemu-kvm-rhev-2.3.0-18.el7.x86_64 > > RHEL7.1 host: > kernel-3.10.0-229.13.1.el7.x86_64 > qemu-kvm-rhev-2.1.2-23.el7_1.8.x86_64 > > RHEL7.2 guest: > kernel-3.10.0-302.el7.x86_64 > > The result is as following: > Host Machine type Result > ------------------------------------------------ > 7.2<->7.1 rhel7.1.0 pass > 7.2<->7.1 rhel7.0.0 pass > 7.2<->7.1 rhel6.6.0 pass > 7.2<->7.1 rhel6.5.0 pass > 7.1<->7.2 rhel7.1.0 pass > 7.1<->7.2 rhel7.0.0 pass > 7.1<->7.2 rhel6.6.0 pass > 7.1<->7.2 rhel6.5.0 pass > > Steps to Test: > 1. boot rhel7.2 guest on src host > 2. boot the guest on dst host with "-incoming rdma:0:5800" > 3. inside guest, do some operation: > # dd if=/dev/random of=/home/file bs=1M count=4096 > # netperf -H 10.66.9.152 -l 600 > 4. do rdma migration: > (qemu) migrate_set_speed 40G > (qemu) migrate_set_capability rdma-pin-all on > (qemu) migrate rdma:192.168.0.21:5800 > 5. inside guest, ping other host, do reboot and shutdown. > > The command line is: > /usr/libexec/qemu-kvm \ > -name rhel7 \ > -S -machine pc-i440fx-rhel7.1.0,accel=kvm,usb=off \ > -realtime mlock=off \ > -cpu SandyBridge \ > -sandbox off \ > -m 3840 \ > -smp 8,cores=1,threads=1,sockets=8 \ > -object > memory-backend-ram,policy=bind,id=mem-0,size=256M,prealloc=yes,host-nodes=0 > -numa node,memdev=mem-0,cpus=0,cpus=1 \ > -object > memory-backend-ram,policy=bind,id=mem-1,size=512M,prealloc=yes,host-nodes=0 > -numa node,memdev=mem-1,cpus=2-3 \ > -object > memory-backend-ram,policy=bind,id=mem-2,size=1024M,prealloc=yes,host-nodes=0 > -numa node,memdev=mem-2,cpus=4 \ > -object > memory-backend-ram,policy=bind,id=mem-3,size=2048M,prealloc=yes,host-nodes=0 > -numa node,memdev=mem-3,cpus=5-7 \ > -uuid 49a3438a-70a3-4ba8-92ce-3a05e0934608 \ > -no-user-config \ > -nodefaults \ > -chardev > socket,id=charmonitor,path=/var/lib/libvirt/qemu/rhel7.monitor,server,nowait > \ > -mon chardev=charmonitor,id=monitor,mode=control \ > -rtc base=utc,driftfix=slew \ > -global kvm-pit.lost_tick_policy=discard \ > -no-hpet \ > -no-shutdown \ > -global PIIX4_PM.disable_s3=1 \ > -global PIIX4_PM.disable_s4=1 \ > -boot order=c,menu=on,splash-time=3000,strict=on \ > -device ich9-usb-ehci1,id=usb0,bus=pci.0,addr=0x5.0x7 \ > -device > ich9-usb-uhci1,masterbus=usb0.0,firstport=0,bus=pci.0,multifunction=on, > addr=0x5 \ > -device ich9-usb-uhci2,masterbus=usb0.0,firstport=2,bus=pci.0,addr=0x5.0x1 \ > -device ich9-usb-uhci3,masterbus=usb0.0,firstport=4,bus=pci.0,addr=0x5.0x2 \ > -device virtio-scsi-pci,id=scsi0,cmd_per_lun=234,bus=pci.0,addr=0x8 \ > -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 \ > -drive > file=/usr/share/virtio-win/virtio-win.iso,if=none,id=drive-ide0-1-0, > readonly=on,format=raw,serial= \ > -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 \ > -drive > file=/usr/share/virtio-win/virtio-win_amd64.vfd,if=none,id=drive-fdc0-0-0, > format=raw,cache=none \ > -global isa-fdc.driveA=drive-fdc0-0-0 \ > -drive > file=/home/rhel7.2.qcow2,if=none,id=drive-virtio-disk0,format=qcow2, > serial=f65effa5-90a6-47f2-8487-a9f64c95d4f5,cache=none,discard=unmap, > werror=stop,rerror=stop,aio=threads \ > -device > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk0, > id=virtio-disk0,bootindex=1 \ > -drive > file=/home/virtio-scsi-blk,if=none,id=drive-scsi-disk,format=qcow2, > cache=writethrough,werror=stop,rerror=stop \ > -device virtio-scsi-pci,id=scsi1,addr=0x13 \ > -device scsi-hd,drive=drive-scsi-disk,bus=scsi1.0,id=data-disk2 \ > -chardev pty,id=charserial0 -device > isa-serial,chardev=charserial0,id=serial0 \ > -chardev spicevmc,id=charchannel0,name=vdagent \ > -device > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0, > name=com.redhat.spice.0 \ > -chardev > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channel/target/rhel7.org. > qemu.guest_agent.0,server,nowait \ > -device > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1, > name=org.qemu.guest_agent.0 \ > -device usb-tablet,id=input0 \ > -device intel-hda,id=sound0,bus=pci.0,addr=0x4 \ > -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ > -device intel-hda,id=sound1,bus=pci.0,addr=0x6 \ > -device hda-micro,id=sound1-codec0,bus=sound1.0 \ > -device intel-hda,id=sound2,bus=pci.0,addr=0x17 \ > -device hda-output,id=sound2-codec0,bus=sound2.0,cad=0 \ > -device ich9-intel-hda,id=sound3,bus=pci.0,addr=0xc \ > -device hda-duplex,id=sound3-codec0,bus=sound3.0,cad=0 \ > -chardev spicevmc,id=charredir0,name=usbredir \ > -device usb-redir,chardev=charredir0,id=redir0 \ > -chardev spicevmc,id=charredir1,name=usbredir \ > -device usb-redir,chardev=charredir1,id=redir1 \ > -chardev spicevmc,id=charredir2,name=usbredir \ > -device usb-redir,chardev=charredir2,id=redir2 \ > -chardev spicevmc,id=charredir3,name=usbredir \ > -device usb-redir,chardev=charredir3,id=redir3 \ > -device usb-host,id=hostdev0 \ > -device pvpanic,ioport=1285 -msg timestamp=on \ > -netdev tap,id=hostnet0,id=hostnet0,script=/etc/qemu-ifup \ > -device > virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=b6:af:42:b8:46:18, > bus=pci.0,addr=0x14 \ > -netdev tap,id=hostnet1,script=/etc/qemu-ifup \ > -device > e1000,netdev=hostnet1,id=virtio-net-pci1,mac=b6:2f:a8:85:72:6c,bus=pci.0, > addr=0x15,multifunction=off \ > -netdev tap,id=hostnet2,vhost=on,script=/etc/qemu-ifup \ > -device > rtl8139,netdev=hostnet2,id=virtio-net-pci2,mac=4e:63:28:bc:b1:25,bus=pci.0, > addr=0x16,multifunction=off \ > -netdev tap,id=hostnet3,vhost=on,script=/etc/qemu-ifup \ > -device > e1000-82540em,netdev=hostnet3,id=virtio-net-pci3,mac=4e:63:28:bc:b1:35, > bus=pci.0,addr=0x18,multifunction=off \ > -netdev tap,id=hostnet4,vhost=on,script=/etc/qemu-ifup \ > -device > e1000-82544gc,netdev=hostnet4,id=virtio-net-pci4,mac=4e:63:28:bc:b1:45, > bus=pci.0,addr=0x19,multifunction=off \ > -netdev tap,id=hostnet5,vhost=on,script=/etc/qemu-ifup \ > -device > e1000-82545em,netdev=hostnet5,id=virtio-net-pci5,mac=4e:63:28:bc:b1:55, > bus=pci.0,addr=0x1a,multifunction=off \ > -drive > file=/home/ide-disk,if=none,id=drive-data-disk,format=raw,cache=none, > aio=native,werror=stop,rerror=stop,copy-on-read=off,serial=fux-ide, > media=disk \ > -device > ide-drive,drive=drive-data-disk,id=system-disk,logical_block_size=512, > physical_block_size=512,min_io_size=32,opt_io_size=64, > discard_granularity=512,ver=fuxc-ver,bus=ide.0,unit=0 \ > -device ich9-usb-uhci6,id=uhci6,bus=pci.0,addr=0xa \ > -device usb-kbd,id=kdb0,bus=uhci6.0 \ > -device ich9-usb-uhci5,id=uhci5,bus=pci.0,addr=0xb \ > -device usb-mouse,id=mouse0,bus=uhci5.0 \ > -device nec-usb-xhci,id=xhci,bus=pci.0,addr=0xd \ > -device usb-bot,id=bot1,bus=xhci.0 \ > -drive > file=/usr/share/virtio-win/virtio-win.iso,if=none,id=usb-cdrom1,format=raw \ > -device scsi-cd,bus=bot1.0,scsi-id=0,lun=1,drive=usb-cdrom1,id=usb-cdrom1 \ > -drive file=/home/bot-disk1,id=usb-disk1,if=none,format=qcow2 \ > -device scsi-hd,bus=bot1.0,scsi-id=0,lun=0,drive=usb-disk1,id=usb-disk1 \ > -device usb-ehci,id=ehci,bus=pci.0,addr=0xe \ > -device usb-bot,id=bot2,bus=ehci.0 \ > -drive > file=/usr/share/virtio-win/virtio-win.iso,if=none,id=usb-cdrom2,format=raw \ > -device scsi-cd,bus=bot2.0,scsi-id=0,lun=1,drive=usb-cdrom2,id=usb-cdrom2 \ > -drive file=/home/bot-disk2,id=usb-disk2,if=none,format=qcow2 \ > -device scsi-hd,bus=bot2.0,scsi-id=0,lun=0,drive=usb-disk2,id=usb-disk2 \ > -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0xf \ > -device usb-bot,id=bot3,bus=usb.0 \ > -drive > file=/usr/share/virtio-win/virtio-win.iso,if=none,id=usb-cdrom3,format=raw \ > -device scsi-cd,bus=bot3.0,scsi-id=0,lun=1,drive=usb-cdrom3,id=usb-cdrom3 \ > -drive file=/home/bot-disk3,id=usb-disk3,if=none,format=qcow2 \ > -device scsi-hd,bus=bot3.0,scsi-id=0,lun=0,drive=usb-disk3,id=usb-disk3 \ > -device ich9-usb-uhci3,id=uhci,bus=pci.0,addr=0x10 \ > -device > usb-storage,drive=drive-usb-0,id=usb-0,removable=on,bus=uhci.0,port=1 \ > -drive file=/home/usb-uhci,if=none,id=drive-usb-0,media=disk,format=qcow2 \ > -device ich9-usb-ehci1,id=ehci1,bus=pci.0,addr=0x11 \ > -device > usb-storage,drive=drive-usb-1,id=usb-1,removable=on,bus=ehci1.0,port=1 \ > -drive file=/home/usb-ehci,if=none,id=drive-usb-1,media=disk,format=qcow2 \ > -device nec-usb-xhci,id=xhci1,bus=pci.0,addr=0x12 \ > -device > usb-storage,drive=drive-usb-2,id=usb-2,removable=on,bus=xhci1.0,port=1 \ > -drive file=/home/usb-xhci,if=none,id=drive-usb-2,media=disk,format=qcow2 \ > -device usb-ccid,id=ccid0 \ > -watchdog ib700 -watchdog-action reset \ > -device virtio-rng-pci,id=rng0,bus=pci.0,addr=0x1c \ > -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x1d \ > -monitor stdio \ > -qmp tcp:0:4466,server,nowait -serial unix:/tmp/ttym,server,nowait \ > -spice port=5910,addr=0.0.0.0,disable-ticketing,seamless-migration=on \ > -k en-us \ > -device > qxl-vga,id=video0,ram_size=134217728,vram_size=67108864,vgamem_mb=16,bus=pci. > 0,addr=0x2 Excellent! Very nice mix of devices.
According to comment21 and comment22, set this issue as verified.