Bug 670787
| Summary: | Hot plug the 14st VF to guest causes guest shut down | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | zhanghaiyan <yoyzhang> | |
| Component: | qemu-kvm | Assignee: | Alex Williamson <alex.williamson> | |
| Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | low | |||
| Version: | 6.1 | CC: | ajia, berrange, chayang, chrisw, ddutile, dyuan, eblake, juzhang, lihuang, llim, mkenneth, virt-maint, xen-maint | |
| Target Milestone: | rc | |||
| Target Release: | --- | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | qemu-kvm-0.12.1.2-2.134.el6 | Doc Type: | Bug Fix | |
| Doc Text: |
Cause:
Device assignment code consumes resources from a fixed pool for each memory ranges used by an assigned device.
Consequence:
When the resource pool is exhausted, the VM exits.
Fix:
Limit number of devices that may be assigned to a VM to avoid running out of resources. Limit set to 8 devices.
Result:
Adding assigned devices to a VM can no longer trigger an unexpected shutdown of the VM.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 678368 (view as bug list) | Environment: | ||
| Last Closed: | 2011-05-19 11:30:54 UTC | Type: | --- | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 580954 | |||
Please provide the QEMU logfile /var/log/libvirt/qemu/$GUEST.log You're not hitting the PCI slot limit, but there are other limits that might be hit. Hopefully QEMU gave some indication in the log. Also if possible try and get a stack trace from QEMU itself when it (likely) crashes. eg, before step 8, do # debuginfo-install qemu-kvm # gdb (gdb) attach ...PID of QEMU.. ..then run step 8... if GDB catches a crash, then run 'thread apply all bt' in GDB. # cat /var/log/libvirt/qemu/rhel6.log 2011-01-19 19:34:48.269: starting up LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.0.0 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name rhel6 -uuid 11b761ff-19f2-b83f-8de5-c3db28ed413c -nodefconfig -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/rhel6.monitor,server,nowait -mon chardev=monitor,mode=control -rtc base=utc -boot c -drive file=/var/lib/libvirt/images/test.img,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/var/lib/libvirt/images/fd.img,if=none,id=drive-fdc0-0-0,format=raw -global isa-fdc.driveA=drive-fdc0-0-0 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -vnc 127.0.0.1:0 -k en-us -vga cirrus -device AC97,id=sound0,bus=pci.0,addr=0x4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 19:34:48.275: 6201: debug : virCgroupNew:555 : New group /libvirt/qemu/rhel6 19:34:48.275: 6201: debug : virCgroupDetect:245 : Detected mount/mapping 0:cpu at /cgroup/cpu in 19:34:48.275: 6201: debug : virCgroupDetect:245 : Detected mount/mapping 1:cpuacct at /cgroup/cpuacct in 19:34:48.275: 6201: debug : virCgroupDetect:245 : Detected mount/mapping 2:cpuset at /cgroup/cpuset in 19:34:48.275: 6201: debug : virCgroupDetect:245 : Detected mount/mapping 3:memory at /cgroup/memory in 19:34:48.275: 6201: debug : virCgroupDetect:245 : Detected mount/mapping 4:devices at /cgroup/devices in 19:34:48.275: 6201: debug : virCgroupDetect:245 : Detected mount/mapping 5:freezer at /cgroup/freezer in 19:34:48.275: 6201: debug : virCgroupMakeGroup:497 : Make group /libvirt/qemu/rhel6 19:34:48.275: 6201: debug : virCgroupMakeGroup:509 : Make controller /cgroup/cpu/libvirt/qemu/rhel6/ 19:34:48.275: 6201: debug : virCgroupMakeGroup:509 : Make controller /cgroup/cpuacct/libvirt/qemu/rhel6/ 19:34:48.275: 6201: debug : virCgroupMakeGroup:509 : Make controller /cgroup/cpuset/libvirt/qemu/rhel6/ 19:34:48.275: 6201: debug : virCgroupMakeGroup:509 : Make controller /cgroup/memory/libvirt/qemu/rhel6/ 19:34:48.275: 6201: debug : virCgroupMakeGroup:509 : Make controller /cgroup/devices/libvirt/qemu/rhel6/ 19:34:48.275: 6201: debug : virCgroupMakeGroup:509 : Make controller /cgroup/freezer/libvirt/qemu/rhel6/ 19:34:48.276: 6201: debug : virCgroupSetValueStr:290 : Set value '/cgroup/cpu/libvirt/qemu/rhel6/tasks' to '6201' 19:34:48.281: 6201: debug : virCgroupSetValueStr:290 : Set value '/cgroup/cpuacct/libvirt/qemu/rhel6/tasks' to '6201' 19:34:48.288: 6201: debug : virCgroupSetValueStr:290 : Set value '/cgroup/cpuset/libvirt/qemu/rhel6/tasks' to '6201' 19:34:48.296: 6201: debug : virCgroupSetValueStr:290 : Set value '/cgroup/memory/libvirt/qemu/rhel6/tasks' to '6201' 19:34:48.304: 6201: debug : virCgroupSetValueStr:290 : Set value '/cgroup/devices/libvirt/qemu/rhel6/tasks' to '6201' 19:34:48.312: 6201: debug : virCgroupSetValueStr:290 : Set value '/cgroup/freezer/libvirt/qemu/rhel6/tasks' to '6201' 19:34:48.320: 6201: debug : qemudInitCpuAffinity:2327 : Setting CPU affinity 19:34:48.321: 6201: debug : qemuSecurityDACSetProcessLabel:547 : Dropping privileges of VM to 107:107 char device redirected to /dev/pts/1 create_userspace_phys_mem: Invalid argument assigned_dev_iomem_map: Error: create new mapping failed 2011-01-19 19:39:15.292: shutting down Ok, no need for the GDB trace, this is a clear QEMU bug.
It fails to create a new mapping for the 14th device in:
static void assigned_dev_iomem_map(PCIDevice *pci_dev, int region_num,
pcibus_t e_phys, pcibus_t e_size, int type)
And just does this totally bogus error handling
if (ret != 0) {
fprintf(stderr, "%s: Error: create new mapping failed\n", __func__);
exit(1);
}
It needs to report the error back to the monitor, not simply exit.
kvm_register_phys_mem() fails because we exceed the limit of 32 memory slots. Reporting the error back is not necessarily trivial either since at this point the device has already been assigned to the guest, and the guest is mapping the PCI BARs. Perhaps we need to pre-map the BARs in the initfn so we can find the error there. Trouble is the guest owns the address space at that point, so I'm not sure where the BAR should be assigned.
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.
New Contents:
Cause:
Device assignment code consumes resources from a fixed pool for each memory ranges used by an assigned device.
Consequence:
When the resource pool is exhausted, the VM exits.
Fix:
Limit number of devices that may be assigned to a VM to avoid running out of resources. Limit set to 8 devices.
Result:
Adding assigned devices to a VM can no longer trigger an unexpected shutdown of the VM.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0534.html An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0534.html |
Description of problem: Hot plug the 14st VF to guest causes guest shut down while the previous 13 VFs can be hotplugged to guest successfully Version-Release number of selected component (if applicable): - 2.6.32-94.el6.x86_64 - libvirt-0.8.7-2.el6.x86_64 - qemu-kvm-0.12.1.2-2.129.el6.x86_64 How reproducible: 3/3 Steps to Reproduce: 1. # rmmod igb 2. # modprobe igb max_vfs=7 3. # virsh nodedev-list --tree computer | +- net_lo_00_00_00_00_00_00 +- pci_0000_00_00_0 +- pci_0000_00_01_0 | | | +- pci_0000_03_00_0 | | | | | +- net_eth0_00_1b_21_39_8b_18 | | | +- pci_0000_03_00_1 | | | | | +- net_eth1_00_1b_21_39_8b_19 | | | +- pci_0000_03_10_0 | | | | | +- net_eth5_2a_cc_b2_a1_da_67 | | | +- pci_0000_03_10_1 | | | | | +- net_eth7_a2_1f_b9_53_1e_04 | | | +- pci_0000_03_10_2 | | | | | +- net_eth9_56_41_9c_f4_3c_15 | | | +- pci_0000_03_10_3 | | | | | +- net_eth8_ea_56_5d_1f_de_a2 | | | +- pci_0000_03_10_4 | | | | | +- net_eth10_5a_9c_c4_10_f5_a4 | | | +- pci_0000_03_10_5 | | | | | +- net_eth11_2a_a9_85_86_db_72 | | | +- pci_0000_03_10_6 | | | | | +- net_eth16_52_8f_73_02_5f_f9 | | | +- pci_0000_03_10_7 | | | | | +- net_eth13_5a_c4_2d_cb_42_8d | | | +- pci_0000_03_11_0 | | | | | +- net_eth12_66_1c_e7_92_4a_4b | | | +- pci_0000_03_11_1 | | | | | +- net_eth15_ae_36_43_f0_e1_6d | | | +- pci_0000_03_11_2 | | | | | +- net_eth18_72_eb_5b_7d_be_93 | | | +- pci_0000_03_11_3 | | | | | +- net_eth19_f2_12_af_86_39_00 | | | +- pci_0000_03_11_4 | | | | | +- net_eth14_b2_b6_cd_e0_8c_f2 | | | +- pci_0000_03_11_5 | | | +- net_eth17_b2_47_c6_22_8b_ec 4. Detach 14 VFs from host # for i in {0..7}; do virsh nodedev-dettach pci_0000_03_10_$i; done Device pci_0000_03_10_0 dettached Device pci_0000_03_10_1 dettached Device pci_0000_03_10_2 dettached Device pci_0000_03_10_3 dettached Device pci_0000_03_10_4 dettached Device pci_0000_03_10_5 dettached Device pci_0000_03_10_6 dettached Device pci_0000_03_10_7 dettached # for i in {0..5}; do virsh nodedev-dettach pci_0000_03_11_$i; done Device pci_0000_03_11_0 dettached Device pci_0000_03_11_1 dettached Device pci_0000_03_11_2 dettached Device pci_0000_03_11_3 dettached Device pci_0000_03_11_4 dettached Device pci_0000_03_11_5 dettached 5. Reset 14 VFs from host 6. Prepare VF*.xml for each VF # ls VF_* VF_10_0.xml VF_10_2.xml VF_10_4.xml VF_10_6.xml VF_11_0.xml VF_11_2.xml VF_11_4.xml VF_10_1.xml VF_10_3.xml VF_10_5.xml VF_10_7.xml VF_11_1.xml VF_11_3.xml VF_11_5.xml # cat VF_10_0.xml <hostdev mode='subsystem' type='pci'> <source> <address bus='3' slot='0x10' function='0'/> </source> </hostdev> 7. Hotplug the previous 13 VFs to guest # virsh start rhel6 Domain rhel6 started # for i in {0..7}; do virsh attach-device rhel6 VF_10_$i.xml; done Device attached successfully Device attached successfully Device attached successfully Device attached successfully Device attached successfully Device attached successfully Device attached successfully Device attached successfully # for i in {0..4}; do virsh attach-device rhel6 VF_11_$i.xml; done Device attached successfully Device attached successfully Device attached successfully Device attached successfully Device attached successfully 8. Hotplug the 14st VF to guest # virsh attach-device rhel6 VF_11_5.xml Device attached successfully Actual results: step7, all 13 VFs could be hotplugged to guest successfully and could get ip in guest step8, the guest is caused to shut down. Expected results: step8, the 14st VF should be able to hotplugged to guest successfully and could get ip in guest as the previous 13 VFs. Because guest could be able to support 32 pci slot, and before hot plug VFs to guest, existing pci device only occupied 5 pci slot. So should could hotplug 27 pci device to guest. For example, indeed I can hotplug 27 virtual network to guest successfully. Additional info: