Bug 1785052
| Summary: | QEMU crashed when detaching a VF from VM | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux Advanced Virtualization | Reporter: | jiyan <jiyan> |
| Component: | qemu-kvm | Assignee: | Amnon Ilan <ailan> |
| Status: | CLOSED DUPLICATE | QA Contact: | Virtualization Bugs <virt-bugs> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 8.2 | CC: | chayang, dyuan, jiyan, peterx, virt-maint, xuzhang, yafu |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-01-09 13:07:25 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Should be a duplicate of Bug 1782678 - qemu core dump after hot-unplugging the XXV710/XL710 PF with multifunction=on BTW:
For the env info:
# lspci |grep "82:00.1"
82:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
For the detach info:
# virsh dumpxml dell36 |grep "<interface" -A9
<interface type='network'> ************original one
<mac address='52:54:00:5d:e8:cb'/>
<source network='default' portid='d44d0426-65bd-48da-a71a-4c889dd93640' bridge='virbr0'/>
<target dev='vnet0'/>
<model type='e1000'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='hostdev' managed='yes'> ************inter.xml
<mac address='52:54:00:0e:09:6d'/>
<driver name='vfio'/>
<source>
<address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x1'/>
</source>
<model type='rtl8139'/>
<alias name='ua-9e6c3c9c-7b5b-4e1a-b498-439ac8cce0c3'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</interface>
<interface type='hostdev' managed='yes'>
<mac address='52:54:00:17:f3:2b'/>
<driver name='vfio'/>
<source>
<address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x3'/>
</source>
<model type='rtl8139'/>
<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<interface type='hostdev' managed='yes'>
<mac address='52:54:00:b8:80:67'/>
<driver name='vfio'/>
<source>
<address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x5'/>
</source>
<model type='rtl8139'/>
<alias name='hostdev1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</interface>
<interface type='hostdev' managed='yes'>
<mac address='52:54:00:1e:a3:5b'/>
<driver name='vfio'/>
<source>
<address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x7'/>
</source>
<model type='rtl8139'/>
<alias name='hostdev2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
</interface>
# virsh detach-device dell36 inter.xml
error: Failed to detach device from inter.xml
error: internal error: End of file from qemu monitor
So in the previous comment, the card is 82599, and there is no multifunction='on' for the detaching interface's address.
The issue above can also be reproduced on RHEL-8.1.1 AV. Version: qemu-kvm-4.1.0-20.module+el8.1.1+5309+6d656f05.x86_64 kernel-4.18.0-147.3.1.el8_1.x86_64 libvirt-5.6.0-10.module+el8.1.1+5309+6d656f05.x86_64 Hi Peter Reproduced on qemu-kvm-4.2.0-5.module+el8.2.0+5389+367d9739.x86_64. And fixed on qemu-kvm-4.2.0-5.el8.bz1782678.x86_64. Version: # rpm -qa libvirt qemu-kvm kernel qemu-kvm-4.2.0-5.module+el8.2.0+5389+367d9739.x86_64 kernel-4.18.0-167.el8.x86_64 libvirt-5.10.0-2.module+el8.2.0+5274+60f836b5.x86_64 Steps: 1. Env setup # virsh nodedev-dumpxml pci_0000_82_00_1 <device> <name>pci_0000_82_00_1</name> <path>/sys/devices/pci0000:80/0000:80:02.0/0000:82:00.1</path> <parent>pci_0000_80_02_0</parent> <driver> <name>ixgbe</name> </driver> <capability type='pci'> <class>0x020000</class> <domain>0</domain> <bus>130</bus> <slot>0</slot> <function>1</function> <product id='0x10fb'>82599ES 10-Gigabit SFI/SFP+ Network Connection</product> <vendor id='0x8086'>Intel Corporation</vendor> <capability type='virt_functions' maxCount='63'> <address domain='0x0000' bus='0x82' slot='0x10' function='0x1'/> <address domain='0x0000' bus='0x82' slot='0x10' function='0x3'/> <address domain='0x0000' bus='0x82' slot='0x10' function='0x5'/> </capability> <iommuGroup number='42'> <address domain='0x0000' bus='0x82' slot='0x00' function='0x1'/> </iommuGroup> <numa node='1'/> <pci-express> <link validity='cap' port='0' speed='5' width='8'/> <link validity='sta' speed='5' width='8'/> </pci-express> </capability> </device> # cat network.xml <network> <name>hostnet</name> <uuid>c1fb4ead-21b8-4d69-8ad9-669c55b3dfc7</uuid> <forward mode='hostdev' managed='yes'> <driver name='vfio'/> <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x1'/> </forward> </network> # virsh net-define network.xml Network hostnet defined from network.xml # virsh net-start hostnet Network hostnet started 2. Attach the VF to VM # virsh attach-interface testbz network hostnet Interface attached successfully # virsh dumpxml testbz |grep "<interface" -A9 ... <interface type='hostdev' managed='yes'> <mac address='52:54:00:e1:91:d1'/> <driver name='vfio'/> <source> <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x1'/> </source> <model type='rtl8139'/> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </interface> 3. Detach the VF ==> Crash # virsh detach-interface testbz --mac 52:54:00:e1:91:d1 --type hostdev error: Failed to detach interface error: internal error: End of file from qemu monitor 4. Install the scratch buid, and start VM until it is totall booted # rpm -qa qemu-kvm qemu-kvm-4.2.0-5.el8.bz1782678.x86_64 # virsh start testbz Domain testbz started 5. Attach the VF to VM again # virsh attach-interface testbz network hostnet Interface attached successfully # virsh dumpxml testbz |grep "<interface" -A9 <interface type='hostdev' managed='yes'> <mac address='52:54:00:32:0c:8a'/> <driver name='vfio'/> <source> <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x1'/> </source> <model type='rtl8139'/> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </interface> 6. Detach the VM ==> No crash here # virsh detach-interface testbz --mac 52:54:00:32:0c:8a --type hostdev Interface detached successfully Thanks jiyan. Then I'm marking the bz accordingly. *** This bug has been marked as a duplicate of bug 1782678 *** |
Description of problem: QEMU crashed when detaching a VF from VM Version-Release number of selected component (if applicable): kernel-4.18.0-167.el8.x86_64 qemu-kvm-4.2.0-4.module+el8.2.0+5220+e82621dc.x86_64 libvirt-5.10.0-1.module+el8.2.0+5135+ed3b2489.x86_64 How reproducible: 100% Steps to Reproduce: 1. Check env info # ethtool -i enp130s0f1 driver: ixgbe version: 5.1.0-k-rh8.2.0 firmware-version: 0x000161ae expansion-rom-version: bus-info: 0000:82:00.1 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes # virsh nodedev-dumpxml pci_0000_82_00_1 <device> <name>pci_0000_82_00_1</name> <path>/sys/devices/pci0000:80/0000:80:02.0/0000:82:00.1</path> <parent>pci_0000_80_02_0</parent> <driver> <name>ixgbe</name> </driver> <capability type='pci'> <class>0x020000</class> <domain>0</domain> <bus>130</bus> <slot>0</slot> <function>1</function> <product id='0x10fb'>82599ES 10-Gigabit SFI/SFP+ Network Connection</product> <vendor id='0x8086'>Intel Corporation</vendor> <capability type='virt_functions' maxCount='63'> <address domain='0x0000' bus='0x82' slot='0x10' function='0x1'/> <address domain='0x0000' bus='0x82' slot='0x10' function='0x3'/> <address domain='0x0000' bus='0x82' slot='0x10' function='0x5'/> <address domain='0x0000' bus='0x82' slot='0x10' function='0x7'/> <address domain='0x0000' bus='0x82' slot='0x11' function='0x1'/> <address domain='0x0000' bus='0x82' slot='0x11' function='0x3'/> <address domain='0x0000' bus='0x82' slot='0x11' function='0x5'/> <address domain='0x0000' bus='0x82' slot='0x11' function='0x7'/> <address domain='0x0000' bus='0x82' slot='0x12' function='0x1'/> </capability> <iommuGroup number='42'> <address domain='0x0000' bus='0x82' slot='0x00' function='0x1'/> </iommuGroup> <numa node='1'/> <pci-express> <link validity='cap' port='0' speed='5' width='8'/> <link validity='sta' speed='5' width='8'/> </pci-express> </capability> </device> 2. Prepare a shutdown VM and net # virsh domstate dell36 shut off # virsh net-dumpxml hostnet <network> <name>hostnet</name> <uuid>c1fb4ead-21b8-4d69-8ad9-669c55b3dfc7</uuid> <forward mode='hostdev' managed='yes'> <driver name='vfio'/> <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x1'/> <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x3'/> <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x5'/> <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x7'/> </forward> </network> 3. Start VM and attach VMs to VM after VM is ally booted # virsh console dell36 Connected to domain dell36 Escape character is ^] Red Hat Enterprise Linux 8.1 (Ootpa) Kernel 4.18.0-147.el8.x86_64 on an x86_64 # cat inter.xml <interface type='network'> <source network='hostnet'/> <alias name='ua-9e6c3c9c-7b5b-4e1a-b498-439ac8cce0c3'/> <mac address='52:54:00:0e:09:6d'/> </interface> # virsh attach-device dell36 inter.xml Device attached successfully # virsh attach-interface dell36 network hostnet Interface attached successfully # virsh attach-interface dell36 network hostnet Interface attached successfully # virsh attach-interface dell36 network hostnet Interface attached successfully # virsh attach-interface dell36 network hostnet error: Failed to attach interface error: internal error: network 'hostnet' requires exclusive access to interfaces, but none are available 4. Detach Vf from VM (Terminal 1) # virsh detach-device dell36 inter.xml error: Failed to detach device from inter.xml error: Unable to read from monitor: Connection reset by peer (Terminal 2) # gdb -p `pidof qemu-kvm` (gdb) c Continuing. Thread 4 "CPU 0/KVM" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fc35f3f3700 (LWP 27192)] 0x00005647b27f804d in notifier_remove (notifier=notifier@entry=0x5647b378db68) at util/notify.c:31 31 QLIST_REMOVE(notifier, node); (gdb) bt #0 0x00005647b27f804d in notifier_remove (notifier=notifier@entry=0x5647b378db68) at util/notify.c:31 #1 0x00005647b24d9b79 in kvm_irqchip_remove_change_notifier (n=n@entry=0x5647b378db68) at /usr/src/debug/qemu-kvm-4.2.0-4.module+el8.2.0+5220+e82621dc.x86_64/accel/kvm/kvm-all.c:1409 #2 0x00005647b251f028 in vfio_exitfn (pdev=<optimized out>) at /usr/src/debug/qemu-kvm-4.2.0-4.module+el8.2.0+5220+e82621dc.x86_64/hw/vfio/pci.c:3103 #3 0x00005647b266453b in pci_qdev_unrealize (dev=<optimized out>, errp=<optimized out>) at hw/pci/pci.c:1131 #4 0x00005647b2600771 in device_set_realized (obj=<optimized out>, value=<optimized out>, errp=0x0) at hw/core/qdev.c:932 #5 0x00005647b2712fbb in property_set_bool (obj=0x5647b378cf20, v=<optimized out>, name=<optimized out>, opaque=0x5647b34e1ba0, errp=0x0) at qom/object.c:2078 #6 0x00005647b2717483 in object_property_set_qobject (obj=0x5647b378cf20, value=<optimized out>, name=0x5647b28cca7d "realized", errp=0x0) at qom/qom-qobject.c:26 #7 0x00005647b2714ce9 in object_property_set_bool (obj=0x5647b378cf20, value=<optimized out>, name=0x5647b28cca7d "realized", errp=0x0) at qom/object.c:1336 #8 0x00005647b25e319b in acpi_pcihp_eject_slot (s=<optimized out>, bsel=<optimized out>, slots=slots@entry=128) at hw/acpi/pcihp.c:170 #9 0x00005647b25e3229 in pci_write (size=<optimized out>, data=128, addr=8, opaque=<optimized out>) at hw/acpi/pcihp.c:341 #10 pci_write (opaque=<optimized out>, addr=<optimized out>, data=128, size=<optimized out>) at hw/acpi/pcihp.c:332 #11 0x00005647b24cc087 in memory_region_write_accessor (mr=<optimized out>, addr=<optimized out>, value=<optimized out>, size=<optimized out>, shift=<optimized out>, mask=<optimized out>, attrs=...) at /usr/src/debug/qemu-kvm-4.2.0-4.module+el8.2.0+5220+e82621dc.x86_64/memory.c:483 #12 0x00005647b24ca2be in access_with_adjusted_size (addr=addr@entry=8, value=value@entry=0x7fc35f3f2508, size=size@entry=4, access_size_min=<optimized out>, access_size_max=<optimized out>, access_fn=0x5647b24cc010 <memory_region_write_accessor>, mr=0x5647b3e6d260, attrs=...) at /usr/src/debug/qemu-kvm-4.2.0-4.module+el8.2.0+5220+e82621dc.x86_64/memory.c:544 #13 0x00005647b24ce19c in memory_region_dispatch_write (mr=0x5647b3e6d260, addr=8, data=<optimized out>, op=<optimized out>, attrs=...) at /usr/src/debug/qemu-kvm-4.2.0-4.module+el8.2.0+5220+e82621dc.x86_64/memory.c:1475 #14 0x00005647b247b227 in flatview_write_continue (fv=0x7fc34c2bc320, addr=44552, attrs=..., buf=0x7fc3729bb000 "\200", len=4, addr1=<optimized out>, l=<optimized out>, mr=0x5647b3e6d260) at /usr/src/debug/qemu-kvm-4.2.0-4.module+el8.2.0+5220+e82621dc.x86_64/include/qemu/host-utils.h:164 #15 0x00005647b247b446 in flatview_write (fv=0x7fc34c2bc320, addr=44552, attrs=..., buf=0x7fc3729bb000 "\200", len=4) at /usr/src/debug/qemu-kvm-4.2.0-4.module+el8.2.0+5220+e82621dc.x86_64/exec.c:3169 #16 0x00005647b247f95f in address_space_write () at /usr/src/debug/qemu-kvm-4.2.0-4.module+el8.2.0+5220+e82621dc.x86_64/exec.c:3259 #17 0x00005647b24dd0b4 in kvm_cpu_exec (cpu=<optimized out>) at /usr/src/debug/qemu-kvm-4.2.0-4.module+el8.2.0+5220+e82621dc.x86_64/accel/kvm/kvm-all.c:2116 #18 0x00005647b24c1eee in qemu_kvm_cpu_thread_fn (arg=0x5647b352a990) at /usr/src/debug/qemu-kvm-4.2.0-4.module+el8.2.0+5220+e82621dc.x86_64/cpus.c:1318 #19 0x00005647b27eb1f4 in qemu_thread_start (args=0x5647b3550920) at util/qemu-thread-posix.c:519 #20 0x00007fc36d9052de in start_thread (arg=<optimized out>) at pthread_create.c:486 #21 0x00007fc36d636e83 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Actual result: As step-4 shows Expected result: VF should be detached successfully and VM should not crash Additional info: