Bug 635954 (AssignedDevBlockMigr)
| Summary: | RFE: Assigned device should block migration | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | juzhang <juzhang> | |
| Component: | qemu-kvm | Assignee: | Alex Williamson <alex.williamson> | |
| Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | medium | |||
| Version: | 6.0 | CC: | amit.shah, bcao, gcosta, mhusnain, michen, mkenneth, tburke, virt-maint | |
| Target Milestone: | rc | Keywords: | FutureFeature | |
| Target Release: | 6.2 | |||
| Hardware: | All | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | qemu-kvm-0.12.1.2-2.132.el6 | Doc Type: | Enhancement | |
| Doc Text: |
Previously, when a virtual machine was associated with a specific hardware component on its host, attempted migrations of the virtual machine failed. This is fixed and the virtual machine migration option is now disabled while the virtual machine is associated with a hardware component on the host and becomes available once the virtual machine is free of all associated devices.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 701576 (view as bug list) | Environment: | ||
| Last Closed: | 2011-05-19 11:28:09 UTC | Type: | --- | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 580953, 701576 | |||
Can this be reproduced using libvirt tools? (In reply to comment #1) > Can this be reproduced using libvirt tools? Sorry reply late,just come back from vacation. Tested again using virt-manger,can't reproduce.migration failed with error "NO IOMMU found,unable to assign device.................." Tested on qemu-kvm-0.12.1.2-2.128.el6.x86_64 src host with PF nic #/usr/libexec/qemu-kvm -m 2G -smp 2 -cpu qemu64,+x2apic -balloon none -no-kvm-pit-reinjection -rtc base=localtime,clock=host,driftfix=slew -usbdevice tablet -drive file=/root/RHEL-Server-6.0-64.raw,if=none,format=raw,cache=none,werror=stop,rerror=stop,id=drive-ide0-0-0 -device ide-drive,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device rtl8139,netdev=hostnet0,mac=20:20:78:90:12:93 -boot c -monitor stdio -vnc :10 -device pci-assign,host=03:00.1,id=hostnet Destination Host with PF attached. /usr/libexec/qemu-kvm -M rhel6.0.0 -enable-kvm -m 1G -smp 1 -monitor stdio -rtc base=localtime -usbdevice tablet -drive file=/root/zhangjunyi/rhel6.0_64.qcow2,if=none,format=qcow2,werror=stop,rerror=stop,id=drive-virtio0-0-0,boot=on,cache=none -device ide-drive,drive=drive-virtio0-0-0,id=virtio0-0-0 -netdev tap,id=hostnet0,vhost=on -device rtl8139,netdev=hostnet0,id=net0,mac=20:20:20:56:42:19 -cpu qemu64,+x2apic -vnc :11 -boot c -qmp tcp:0:4445,server,nowait (qemu)migrate -d tcp:0:5555 Results: (1)check migration status in src host (qemu) info migrate Migration status: active transferred ram: 2073332 kbytes remaining ram: 44068 kbytes total ram: 2113920 kbytes (qemu) info migrate Migration status: failed (qemu) info migrate Migration status: failed (2)Destination host (qemu) Unknown savevm section or instance '0000:00:04.0/pci-assign' 0 load of migration failed I both tried PF and VF,both hit this problem. Hi,Alex This is expected results?assigned device indeed block migration,however,can we get more friendly message? Best Regards, Junyi (In reply to comment #8) > Tested on qemu-kvm-0.12.1.2-2.128.el6.x86_64 This test appears to be completely invalid... > src host with PF nic > > #/usr/libexec/qemu-kvm -m 2G -smp 2 -cpu qemu64,+x2apic -balloon none > -no-kvm-pit-reinjection -rtc base=localtime,clock=host,driftfix=slew -usbdevice > tablet -drive > file=/root/RHEL-Server-6.0-64.raw,if=none,format=raw,cache=none,werror=stop,rerror=stop,id=drive-ide0-0-0 > -device ide-drive,drive=drive-ide0-0-0,id=ide0-0-0 -netdev > tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device > rtl8139,netdev=hostnet0,mac=20:20:78:90:12:93 -boot c -monitor stdio -vnc :10 > -device pci-assign,host=03:00.1,id=hostnet > > Destination Host with PF attached. > > /usr/libexec/qemu-kvm -M rhel6.0.0 -enable-kvm -m 1G -smp 1 -monitor stdio -rtc different machine type, different memory size, different SMP... > base=localtime -usbdevice tablet -drive > file=/root/zhangjunyi/rhel6.0_64.qcow2,if=none,format=qcow2,werror=stop,rerror=stop,id=drive-virtio0-0-0,boot=on,cache=none different block device, different format, different interface... > -device ide-drive,drive=drive-virtio0-0-0,id=virtio0-0-0 -netdev > tap,id=hostnet0,vhost=on -device > rtl8139,netdev=hostnet0,id=net0,mac=20:20:20:56:42:19 -cpu qemu64,+x2apic -vnc > :11 -boot c -qmp tcp:0:4445,server,nowait different mac address on the rtl8139, and no assigned device. > > (qemu)migrate -d tcp:0:5555 I also don't see an incoming on the destination, so I hope all the above differences are a cut and paste error. > Results: > (1)check migration status in src host > (qemu) info migrate > Migration status: active > transferred ram: 2073332 kbytes > remaining ram: 44068 kbytes > total ram: 2113920 kbytes > (qemu) info migrate > Migration status: failed > (qemu) info migrate > Migration status: failed > (2)Destination host > (qemu) Unknown savevm section or instance '0000:00:04.0/pci-assign' 0 > load of migration failed > > I both tried PF and VF,both hit this problem. This is the expected failure mode. > Hi,Alex > > This is expected results?assigned device indeed block migration,however,can > we get more friendly message? Sorry, I tried to overhaul the entire qemu infrastructure upstream to allow better errors and more immediate failures and was NAK'd. This the expected error message. > > > > Destination Host without PF attached. > > > > /usr/libexec/qemu-kvm -M rhel6.0.0 -enable-kvm -m 1G -smp 1 -monitor stdio -rtc > > different machine type, different memory size, different SMP... > > > base=localtime -usbdevice tablet -drive > > file=/root/zhangjunyi/rhel6.0_64.qcow2,if=none,format=qcow2,werror=stop,rerror=stop,id=drive-virtio0-0-0,boot=on,cache=none > > different block device, different format, different interface... > > > -device ide-drive,drive=drive-virtio0-0-0,id=virtio0-0-0 -netdev > > tap,id=hostnet0,vhost=on -device > > rtl8139,netdev=hostnet0,id=net0,mac=20:20:20:56:42:19 -cpu qemu64,+x2apic -vnc > > :11 -boot c -qmp tcp:0:4445,server,nowait > > different mac address on the rtl8139, and no assigned device. > > > > > (qemu)migrate -d tcp:0:5555 > Junyi:sorry,I paste wrong destination CML. Destination without PF,should be this /usr/libexec/qemu-kvm -m 2G -smp 2 -cpu qemu64,+x2apic -balloon none -no-kvm-pit-reinjection -rtc base=localtime,clock=host,driftfix=slew -usbdevice tablet -drive file=/root/RHEL-Server-6.0-64.raw,if=none,format=raw,cache=none,werror=stop,rerror=stop,id=drive-ide0-0-0 -device ide-drive,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device rtl8139,netdev=hostnet0,mac=20:20:78:90:12:93 -boot c -monitor stdio -vnc :11 -incoming tcp:0:5555 > > Sorry, I tried to overhaul the entire qemu infrastructure upstream to allow > better errors and more immediate failures and was NAK'd. This the expected > error message. Alex,you want to improve error message,if not,I will set this bug as pass.according to your description,this is right results. Best Regards, Junyi (In reply to comment #10) > Junyi:sorry,I paste wrong destination CML. > Destination without PF,should be this Good, that looks better. > > > > Sorry, I tried to overhaul the entire qemu infrastructure upstream to allow > > better errors and more immediate failures and was NAK'd. This the expected > > error message. > Alex,you want to improve error message,if not,I will set this bug as > pass.according to your description,this is right results. If the migration is run in the foreground, without the -d, you'll see this in the monitor: cannot migrate with device '0000:00:04.0/pci-assign' However I don't see a way to print to the monitor when it's run in the background. Would it perhaps be better to send this to stderror? Also, I just noticed another issue, if the migration destination includes an assigned device, the destination process will segfault rather than finish as you show. I've sent a patch for this upstream, perhaps we should also include that for this bz to be complete. > If the migration is run in the foreground, without the -d, you'll see this in > the monitor: What's your run in the foreground means and without the -d? you means in the process of migration,hot-add pass-through nic? what's without -d means? I tried the following scenario (1) In the process of migrating,do hot-add pass-through nic (qemu)migrate -d tcp:0:5555 (qemu) device_add pci-assign,host=03:10.0,id=hostnet results: Source host,(qemu) info migrate Migration status: failed Destination host (qemu) Unknown savevm section or instance '0000:00:04.0/pci-assign' 0 load of migration failed (2)migrate without -d (qemu)migrate tcp:0:5555 Can't do any operation in soruce monitor (qemu) until migration completed. I can't see the message "cannot migrate with device '0000:00:04.0/pci-assign' " > Also, I just noticed another issue, if the migration destination includes an > assigned device, the destination process will segfault rather than finish as > you show. I've sent a patch for this upstream, perhaps we should also include > that for this bz to be complete. Yes,right,found (qemu) Segmentation fault (core dumped) you plan to fix this issue in this bug or plan to bug a separate bug? If you want to fix in this issue,I will change bug status form ON_QA to assigned.if you want fix it in separate issue,I will set this issue as verified and open new issue? (In reply to comment #12) > > > If the migration is run in the foreground, without the -d, you'll see this in > > the monitor: > What's your run in the foreground means and without the -d? > you means in the process of migration,hot-add pass-through nic? what's without > -d means? > I tried the following scenario > (1) In the process of migrating,do hot-add pass-through nic > (qemu)migrate -d tcp:0:5555 > (qemu) device_add pci-assign,host=03:10.0,id=hostnet > > results: > Source host,(qemu) info migrate > Migration status: failed > > Destination host > (qemu) Unknown savevm section or instance '0000:00:04.0/pci-assign' 0 > load of migration failed > > (2)migrate without -d > (qemu)migrate tcp:0:5555 > > Can't do any operation in soruce monitor (qemu) until migration completed. Forget this question,I can see it "(qemu) migrate tcp:0:5555 cannot migrate with device '0000:00:04.0/pci-assign" :) Let's fix it here, back to assigned. For tracking, the patches we already added in qemu-kvm-0.12.1.2-2.124.el6 are fine, we just need to add a couple more to make this complete. Please test this brew build to see if it meets your expectations: https://brewweb.devel.redhat.com/taskinfo?taskID=3040180 You should now always see an error message, the migration should fail immediately, and you should never see a segfault on the target. Thanks, Alex (In reply to comment #15) > Please test this brew build to see if it meets your expectations: > > https://brewweb.devel.redhat.com/taskinfo?taskID=3040180 > > You should now always see an error message, the migration should fail > immediately, and you should never see a segfault on the target. Thanks, > > Alex Tested on what you provided build,I tried three scenarios,two works,the last is failed.the follow is detailed infos. (1)I tried the migration destination includes an assigned device,assigned device indeed block migration and no core dump. (qemu) migrate -d tcp:0:5555 state blocked by non-migratable device '0000:00:04.0/pci-assign' (2)I also tried the migration destination without an assigned device,works too. (qemu) migrate -d tcp:0:5555 state blocked by non-migratable device '0000:00:04.0/pci-assign' (3)In the process of migration,hot-add an assigned device.failed. Source qemu #/usr/libexec/qemu-kvm -m 2G -smp 2 -cpu qemu64,+x2apic -balloon none -no-kvm-pit-reinjection -rtc base=localtime,clock=host,driftfix=slew -usbdevice tablet -drive file=/root/RHEL-Server-6.0-64.raw,if=none,format=raw,cache=none,werror=stop,rerror=stop,id=drive-ide0-0-0 -device ide-drive,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device rtl8139,netdev=hostnet0,mac=20:20:78:90:12:93 -boot c -monitor stdio -vnc :10 (qemu) migrate -d tcp:0:5555 Before migration finish,hot add an assigned device (qemu) device_add pci-assign,host=03:00.1,id=test1 (qemu) info migrate Migration status: active transferred ram: 2093784 kbytes remaining ram: 38800 kbytes total ram: 2113920 kbytes (qemu) info migrate Migration status: completed Destination qemu. #/usr/libexec/qemu-kvm -m 2G -smp 2 -cpu qemu64,+x2apic -balloon none -no-kvm-pit-reinjection -rtc base=localtime,clock=host,driftfix=slew -usbdevice tablet -drive file=/root/RHEL-Server-6.0-64.raw,if=none,format=raw,cache=none,werror=stop,rerror=stop,id=drive-ide0-0-0 -device ide-drive,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device rtl8139,netdev=hostnet0,mac=20:20:78:90:12:93 -boot c -monitor stdio -vnc :11 --incoming tcp:0:5555 QEMU 0.12.1 monitor - type 'help' for more information (qemu) Unknown savevm section or instance '0000:00:04.0/pci-assign' 0 load of migration failed Hi,Alex Do you want to improve scenario three? Best Regards, Junyi (In reply to comment #16) > (In reply to comment #15) > > Please test this brew build to see if it meets your expectations: > > > > https://brewweb.devel.redhat.com/taskinfo?taskID=3040180 > > > > You should now always see an error message, the migration should fail > > immediately, and you should never see a segfault on the target. Thanks, > > > > Alex > > Tested on what you provided build,I tried three scenarios,two works,the last is > failed.the follow is detailed infos. > > (1)I tried the migration destination includes an assigned device,assigned > device indeed block migration and no core dump. > (qemu) migrate -d tcp:0:5555 > state blocked by non-migratable device '0000:00:04.0/pci-assign' > > (2)I also tried the migration destination without an assigned device,works too. > (qemu) migrate -d tcp:0:5555 > state blocked by non-migratable device '0000:00:04.0/pci-assign' > > (3)In the process of migration,hot-add an assigned device.failed. > Source qemu > #/usr/libexec/qemu-kvm -m 2G -smp 2 -cpu qemu64,+x2apic -balloon none > -no-kvm-pit-reinjection -rtc base=localtime,clock=host,driftfix=slew -usbdevice > tablet -drive > file=/root/RHEL-Server-6.0-64.raw,if=none,format=raw,cache=none,werror=stop,rerror=stop,id=drive-ide0-0-0 > -device ide-drive,drive=drive-ide0-0-0,id=ide0-0-0 -netdev > tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device > rtl8139,netdev=hostnet0,mac=20:20:78:90:12:93 -boot c -monitor stdio -vnc :10 > (qemu) migrate -d tcp:0:5555 > Before migration finish,hot add an assigned device > (qemu) device_add pci-assign,host=03:00.1,id=test1 > (qemu) info migrate > Migration status: active > transferred ram: 2093784 kbytes > remaining ram: 38800 kbytes > total ram: 2113920 kbytes > (qemu) info migrate > Migration status: completed > > Destination qemu. > #/usr/libexec/qemu-kvm -m 2G -smp 2 -cpu qemu64,+x2apic -balloon none > -no-kvm-pit-reinjection -rtc base=localtime,clock=host,driftfix=slew -usbdevice > tablet -drive > file=/root/RHEL-Server-6.0-64.raw,if=none,format=raw,cache=none,werror=stop,rerror=stop,id=drive-ide0-0-0 > -device ide-drive,drive=drive-ide0-0-0,id=ide0-0-0 -netdev > tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device > rtl8139,netdev=hostnet0,mac=20:20:78:90:12:93 -boot c -monitor stdio -vnc :11 > --incoming tcp:0:5555 > QEMU 0.12.1 monitor - type 'help' for more information > (qemu) Unknown savevm section or instance '0000:00:04.0/pci-assign' 0 > load of migration failed > > Hi,Alex > > Do you want to improve scenario three? Thanks for testing. Scenario 3 is really not supportable right now and I think falls into a case where the raw monitor interface will allow you to attempt things that shouldn't be tried and are hopefully prevented using the libvirt interfaces. I think there are scenarios where you'd even run into problems hot adding or removing emulated devices, that are migrate-able, when a migration is already in progress. Verified on qemu-kvm-0.12.1.2-2.132.el6,passed. the steps are as same as comment17 except Scenario 3.
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.
New Contents:
Cause:
Device assignment ties a VM to a specific piece of hardware on the host, making VM migration impossible. However, nothing actually prevented a migration from being attempted.
Consequence:
VM with assigned device may attempt migration.
Fix:
Deny migration requests when an assigned device is present in the VM.
Result:
VMs with assigned devices will not allow a migration, ensuring the integrity of the VM. Migration is re-enabled if all assigned devices are removed from the VM.
Technical note updated. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.
Diffed Contents:
@@ -1,8 +1 @@
-Cause:
+Previously, when a virtual machine was associated with a specific hardware component on its host, attempted migrations of the virtual machine failed. This is fixed and the virtual machine migration option is now disabled while the virtual machine is associated with a hardware component on the host and becomes available once the virtual machine is free of all associated devices.-Device assignment ties a VM to a specific piece of hardware on the host, making VM migration impossible. However, nothing actually prevented a migration from being attempted.
-Consequence:
-VM with assigned device may attempt migration.
-Fix:
-Deny migration requests when an assigned device is present in the VM.
-Result:
-VMs with assigned devices will not allow a migration, ensuring the integrity of the VM. Migration is re-enabled if all assigned devices are removed from the VM.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0534.html An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0534.html |
Description of problem: Boot a guest with one physical assigned,then boot the guest in the listening mode without physical assigned.then do migration and Migration finished successful.I both tried local migration and remote migration,all successful.however,even migration finished,this physical nic still does not work in guest any more.it's very easy confused user. Version-Release number of selected component (if applicable): #uname -r 2.6.32-71.1.1.el6_0.x86_64 #rpm -q qemu-kvm qemu-kvm-0.12.1.2-2.113.el6_0.1.x86_64 How reproducible: Steps to Reproduce: 1.Unbind 82576 nic from host #lspci -n | grep 28:00.0 28:00.0 0200: 8086:10c9 (rev 01) echo "8086 10c9" >/sys/bus/pci/drivers/pci-stub/new_id echo 0000:28:00.0 >/sys/bus/pci/devices/0000\:28\:00.0/driver/unbind echo 0000:28:00.0 >/sys/bus/pci/drivers/pci-stub/bind 2.Start src VM with physical nic assigned #/usr/libexec/qemu-kvm -enable-kvm -m 2G -smp 2 -uuid `uuidgen` -monitor stdio -rtc base=localtime -drive file=/home/rhel6.0rc3_64.qcow2,format=qcow2,id=test,cache=none,boot=on,werror=stop,rerror=stop,if=none -device ide-drive,drive=test,id=test1 -boot c -qmp tcp:0:4445,server,nowait -vnc :10 -device pci-assign,host=28:00.0,id=hostnet -net none 3.Start dest waiting for migration without physical nic assigned #usr/libexec/qemu-kvm -enable-kvm -m 2G -smp 2 -uuid `uuidgen` -monitor stdio -rtc base=localtime -drive file=/root/nfs/rhel64.qcow2,format=qcow2,id=test,cache=none,boot=on,werror=stop,rerror=stop,if=none -device ide-drive,drive=test,id=test1 -boot c -qmp tcp:0:4445,server,nowait -vnc :10 -net none 4.Migrate src to dest Actual results: I both tried tried local migration and remote migration. 1. remote migration(src host pass-through 82576 nic to guest,dest host have not 82576 nic,even didn't add intel_iommu=on in kernel line) Migration was finished.after migration.checheck pci info via qemu on destination host.didn't find 82576 nic.however,in guest.found 82576 nic.however,82576 nic does not work. (qemu) info pci Bus 0, device 0, function 0: Host bridge: PCI device 8086:1237 id "" Bus 0, device 1, function 0: ISA bridge: PCI device 8086:7000 id "" Bus 0, device 1, function 1: IDE controller: PCI device 8086:7010 BAR4: I/O at 0xc000 [0xc00f]. id "" Bus 0, device 1, function 3: Bridge: PCI device 8086:7113 IRQ 9. id "" Bus 0, device 2, function 0: VGA controller: PCI device 1013:00b8 BAR0: 32 bit prefetchable memory at 0xf0000000 [0xf1ffffff]. BAR1: 32 bit memory at 0xf2000000 [0xf2000fff]. BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe]. id "" In guest #lspci | grep 82576 00:03.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 2. local migration Migration was finished.after migration.checheck pci info via qemu on destination host.didn't find 82576 nic.however,in guest.found 82576 nic.however,82576 nic does not work. Expected results: should be some prompts and migration failed.it's very easy confused user. Additional info: 1. If using the same steps except using emulation nic replace of pass-through nic.migration failed with message some like "(qemu) Unknown ramblock "0000:00:04.0/e1000.rom", cannot accept migration qemu: warning: error while loading state for instance 0x0 of device 'ram' load of migration failed" 2. If this is intended design for device pass-through,please close this bug.