Bug 1002669

Summary: libvirtd crash when starting guest after cold-plug interface with hostdev network
Product: Red Hat Enterprise Linux 7 Reporter: Peter Krempa <pkrempa>
Component: libvirtAssignee: Peter Krempa <pkrempa>
Status: CLOSED CURRENTRELEASE QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 7.0CC: acathrow, dyuan, honzhang, jdenemar, mzhan, shyu, xuzhang
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-1.1.1-3.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1000973 Environment:
Last Closed: 2014-06-13 13:16:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1000973    
Bug Blocks:    
Attachments:
Description Flags
libvirtd.log
none
backtrace none

Description Peter Krempa 2013-08-29 17:06:39 UTC 
+++ This bug was initially created as a clone of Bug #1000973 +++

Description of problem:
libvirtd crash when starting  guest after cold-plug interface with hostdev network


Version-Release number of selected component (if applicable):
libvirt-0.10.2-23.el6.x86_64 

How reproducible:
100%

Steps to Reproduce:
# virsh start r6
Domain r6 started

# virsh net-list --all
Name                 State      Autostart     Persistent
--------------------------------------------------
default              active     no            yes
hostdev-net1         active     no            yes

# lspci|grep 11:10
11:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)


# virsh net-dumpxml hostdev-net1
<network>
  <name>hostdev-net1</name>
  <uuid>c039ec41-79cd-9418-76bc-a74a7ce29715</uuid>
  <forward mode='hostdev' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x1'/>
  </forward>
</network>

# virsh destroy r6
Domain r6 destroyed

# cat vfpool.xml
<interface type='network'>
   <source network='hostdev-net1'/>
</interface>

# virsh attach-device r6 vfpool.xml --config
Device attached successfully

# virsh start r6
error: Failed to start domain r6
error: End of file while reading data: Input/output error
error: One or more references were leaked after disconnect from the hypervisor
error: Failed to reconnect to the hypervisor


Actual results:
libvirtd crash 

Expected results:
libvirtd works fine 

Additional info:

--- Additional comment from Peter Krempa on 2013-08-29 10:58:03 CEST ---

Fixed upstream:

commit 50348e6edfa10ddb61929bf95a1c4820a9614e19
Author: Peter Krempa <pkrempa>
Date:   Tue Aug 27 19:06:18 2013 +0200

    qemu: Remove hostdev entry when freeing the depending network entry
    
    When using a <interface type="network"> that points to a network with
    hostdev forwarding mode a hostdev alias is created for the network. This
    allias is inserted into the hostdev list, but is backed with a part of
    the network object that it is connected to.
    
    When a VM is being stopped qemuProcessStop() calls
    networkReleaseActualDevice() which eventually frees the memory for the
    hostdev object. Afterwards when the domain definition is being freed by
    virDomainDefFree() an invalid pointer is accessed by
    virDomainHostdevDefFree() and may cause a crash of the daemon.
    
    This patch removes the entry in the hostdev list before freeing the
    depending memory to avoid this issue.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1000973
Comment 3 Xuesong Zhang 2013-09-02 05:42:15 UTC 
Verify this bug in build libvirt-1.1.1-3.el7.x86_64, the bug is fixed, so change the status to verify.

steps:
1. prepare one healthy guest.
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 4     r7                             running

2. prepare one network with hostdev forward mode 
# virsh net-dumpxml hostdev-net1
<network connections='1'>
  <name>hostdev-net1</name>
  <uuid>6208c354-0713-4d90-b12c-a893546e44ae</uuid>
  <forward mode='hostdev' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x1'/>
  </forward>
</network>

# virsh net-list --all
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 br1                  inactive   no            yes
 default              active     yes           yes
 hostdev-net1         active     no            yes

3. check the vf in the host
# lspci|grep 82576
0e:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0e:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0f:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
10:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
10:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
11:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

4. prepare one xml like the following one:
# cat vfpool.xml 
<interface type='network'>
   <source network='hostdev-net1'/>
</interface>

5. destroy the guest, and attach the interface to the guest with config option.
# virsh destroy r7
Domain r7 destroyed

# virsh attach-device r7 vfpool.xml --config
Device attached successfully

6. check the libvirtd status
# service libvirtd status
Redirecting to /bin/systemctl status  libvirtd.service
libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
   Active: active (running) since 一 2013-09-02 01:30:46 EDT; 3min 32s ago
 Main PID: 3769 (libvirtd)
   CGroup: name=systemd:/system/libvirtd.service
           ├─1046 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf
           ├─3769 /usr/sbin/libvirtd
           └─4182 /usr/libexec/qemu-kvm -name r7 -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -bios /usr...

9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...14
9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...15
9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...16
9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...17
9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...18
9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...19
9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...20
9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...21
9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...24
9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...27

7. make sure the interface is cold-pluged.
# virsh dumpxml r7|grep hostdev -A 5
      <source network='hostdev-net1'/>
      <model type='rtl8139'/>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
    

# ps -ef|grep qemu
qemu      4182     1 42 01:34 ?        00:00:17 /usr/libexec/qemu-kvm -name r7 ...... -device pci-assign,configfd=23,host=0f:10.0,id=hostdev0,bus=pci.0,addr=0x4
Comment 4 Xuesong Zhang 2013-09-02 07:25:34 UTC 
hi, 

  While I do some other operations, the libvirtd service crashed again. Seems like the same issue. 

Here is the reproduce steps:
1. prepare one healthy guest.
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 4     r7                             running

2. prepare one network with hostdev forward mode 
# virsh net-dumpxml hostdev-net1
<network connections='1'>
  <name>hostdev-net1</name>
  <uuid>6208c354-0713-4d90-b12c-a893546e44ae</uuid>
  <forward mode='hostdev' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x1'/>
  </forward>
</network>

# virsh net-list --all
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 br1                  inactive   no            yes
 default              active     yes           yes
 hostdev-net1         active     no            yes

3. check the vf in the host
# lspci|grep 82576
0e:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0e:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0f:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
10:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
10:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
11:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

4. prepare one xml like the following one:
# cat vfpool.xml 
<interface type='network'>
   <source network='hostdev-net1'/>
</interface>

5. hot-plug the vf to the guest.
# virsh attach-device r7 vfpool.xml 
Device attached successfully

6. make sure the vf is attached.
# virsh dumpxml r7
.......
<interface type='network'>
      <mac address='52:54:00:67:c2:3f'/>
      <source network='hostdev-net1'/>
      <model type='rtl8139'/>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>
......

7. new one vf xml for detach, the content is same with the one in dumpxml.
# cat vf-detach.xml 
<interface type='network'>
      <mac address='52:54:00:67:c2:3f'/>
      <source network='hostdev-net1'/>
      <model type='rtl8139'/>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>

8. detach the vf from guest.
# virsh detach-device r7 vf-detach.xml 
Device detached successfully

9. destroy the guest.
# virsh destroy r7
error: Failed to destroy domain r7
error: End of file while reading data: Input/output error
error: One or more references were leaked after disconnect from the hypervisor
error: Failed to reconnect to the hypervisor


Actual result:
libvirtd crashed in step9.

Expected result:
libvirtd should be running.
Comment 5 Xuesong Zhang 2013-09-02 07:26:24 UTC 
Created attachment 792726 [details]
libvirtd.log
Comment 6 Xuesong Zhang 2013-09-02 07:27:08 UTC 
Created attachment 792727 [details]
backtrace
Comment 7 Xuesong Zhang 2013-09-02 07:28:28 UTC 
Upload the libvirtd.log and backtrace in comment 5 and comment 6.
Change the bug status to "Assigned".
Comment 8 Xuesong Zhang 2013-09-02 10:05:54 UTC 
After contact with Jiri, file one new bug for the issue in comment 4.
https://bugzilla.redhat.com/show_bug.cgi?id=1003537

So, change this bug to verify according to comment 3.
Comment 9 Ludek Smid 2014-06-13 13:16:35 UTC 
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.