Hide Forgot
+++ This bug was initially created as a clone of Bug #1000973 +++ Description of problem: libvirtd crash when starting guest after cold-plug interface with hostdev network Version-Release number of selected component (if applicable): libvirt-0.10.2-23.el6.x86_64 How reproducible: 100% Steps to Reproduce: # virsh start r6 Domain r6 started # virsh net-list --all Name State Autostart Persistent -------------------------------------------------- default active no yes hostdev-net1 active no yes # lspci|grep 11:10 11:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) # virsh net-dumpxml hostdev-net1 <network> <name>hostdev-net1</name> <uuid>c039ec41-79cd-9418-76bc-a74a7ce29715</uuid> <forward mode='hostdev' managed='yes'> <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x0'/> <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x1'/> </forward> </network> # virsh destroy r6 Domain r6 destroyed # cat vfpool.xml <interface type='network'> <source network='hostdev-net1'/> </interface> # virsh attach-device r6 vfpool.xml --config Device attached successfully # virsh start r6 error: Failed to start domain r6 error: End of file while reading data: Input/output error error: One or more references were leaked after disconnect from the hypervisor error: Failed to reconnect to the hypervisor Actual results: libvirtd crash Expected results: libvirtd works fine Additional info: --- Additional comment from Peter Krempa on 2013-08-29 10:58:03 CEST --- Fixed upstream: commit 50348e6edfa10ddb61929bf95a1c4820a9614e19 Author: Peter Krempa <pkrempa> Date: Tue Aug 27 19:06:18 2013 +0200 qemu: Remove hostdev entry when freeing the depending network entry When using a <interface type="network"> that points to a network with hostdev forwarding mode a hostdev alias is created for the network. This allias is inserted into the hostdev list, but is backed with a part of the network object that it is connected to. When a VM is being stopped qemuProcessStop() calls networkReleaseActualDevice() which eventually frees the memory for the hostdev object. Afterwards when the domain definition is being freed by virDomainDefFree() an invalid pointer is accessed by virDomainHostdevDefFree() and may cause a crash of the daemon. This patch removes the entry in the hostdev list before freeing the depending memory to avoid this issue. Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1000973
Verify this bug in build libvirt-1.1.1-3.el7.x86_64, the bug is fixed, so change the status to verify. steps: 1. prepare one healthy guest. # virsh list --all Id Name State ---------------------------------------------------- 4 r7 running 2. prepare one network with hostdev forward mode # virsh net-dumpxml hostdev-net1 <network connections='1'> <name>hostdev-net1</name> <uuid>6208c354-0713-4d90-b12c-a893546e44ae</uuid> <forward mode='hostdev' managed='yes'> <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x0'/> <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x1'/> <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x0'/> <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x1'/> </forward> </network> # virsh net-list --all Name State Autostart Persistent ---------------------------------------------------------- br1 inactive no yes default active yes yes hostdev-net1 active no yes 3. check the vf in the host # lspci|grep 82576 0e:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 0e:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 0f:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 0f:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 0f:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 0f:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 10:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 10:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 11:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 4. prepare one xml like the following one: # cat vfpool.xml <interface type='network'> <source network='hostdev-net1'/> </interface> 5. destroy the guest, and attach the interface to the guest with config option. # virsh destroy r7 Domain r7 destroyed # virsh attach-device r7 vfpool.xml --config Device attached successfully 6. check the libvirtd status # service libvirtd status Redirecting to /bin/systemctl status libvirtd.service libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled) Active: active (running) since 一 2013-09-02 01:30:46 EDT; 3min 32s ago Main PID: 3769 (libvirtd) CGroup: name=systemd:/system/libvirtd.service ├─1046 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf ├─3769 /usr/sbin/libvirtd └─4182 /usr/libexec/qemu-kvm -name r7 -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -bios /usr... 9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...14 9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...15 9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...16 9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...17 9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...18 9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...19 9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...20 9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...21 9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...24 9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...27 7. make sure the interface is cold-pluged. # virsh dumpxml r7|grep hostdev -A 5 <source network='hostdev-net1'/> <model type='rtl8139'/> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </interface> # ps -ef|grep qemu qemu 4182 1 42 01:34 ? 00:00:17 /usr/libexec/qemu-kvm -name r7 ...... -device pci-assign,configfd=23,host=0f:10.0,id=hostdev0,bus=pci.0,addr=0x4
hi, While I do some other operations, the libvirtd service crashed again. Seems like the same issue. Here is the reproduce steps: 1. prepare one healthy guest. # virsh list --all Id Name State ---------------------------------------------------- 4 r7 running 2. prepare one network with hostdev forward mode # virsh net-dumpxml hostdev-net1 <network connections='1'> <name>hostdev-net1</name> <uuid>6208c354-0713-4d90-b12c-a893546e44ae</uuid> <forward mode='hostdev' managed='yes'> <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x0'/> <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x1'/> <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x0'/> <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x1'/> </forward> </network> # virsh net-list --all Name State Autostart Persistent ---------------------------------------------------------- br1 inactive no yes default active yes yes hostdev-net1 active no yes 3. check the vf in the host # lspci|grep 82576 0e:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 0e:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 0f:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 0f:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 0f:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 0f:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 10:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 10:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 11:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 4. prepare one xml like the following one: # cat vfpool.xml <interface type='network'> <source network='hostdev-net1'/> </interface> 5. hot-plug the vf to the guest. # virsh attach-device r7 vfpool.xml Device attached successfully 6. make sure the vf is attached. # virsh dumpxml r7 ....... <interface type='network'> <mac address='52:54:00:67:c2:3f'/> <source network='hostdev-net1'/> <model type='rtl8139'/> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </interface> ...... 7. new one vf xml for detach, the content is same with the one in dumpxml. # cat vf-detach.xml <interface type='network'> <mac address='52:54:00:67:c2:3f'/> <source network='hostdev-net1'/> <model type='rtl8139'/> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </interface> 8. detach the vf from guest. # virsh detach-device r7 vf-detach.xml Device detached successfully 9. destroy the guest. # virsh destroy r7 error: Failed to destroy domain r7 error: End of file while reading data: Input/output error error: One or more references were leaked after disconnect from the hypervisor error: Failed to reconnect to the hypervisor Actual result: libvirtd crashed in step9. Expected result: libvirtd should be running.
Created attachment 792726 [details] libvirtd.log
Created attachment 792727 [details] backtrace
Upload the libvirtd.log and backtrace in comment 5 and comment 6. Change the bug status to "Assigned".
After contact with Jiri, file one new bug for the issue in comment 4. https://bugzilla.redhat.com/show_bug.cgi?id=1003537 So, change this bug to verify according to comment 3.
This request was resolved in Red Hat Enterprise Linux 7.0. Contact your manager or support representative in case you have further questions about the request.