RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1002669 - libvirtd crash when starting guest after cold-plug interface with hostdev network
Summary: libvirtd crash when starting guest after cold-plug interface with hostdev net...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Peter Krempa
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On: 1000973
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-08-29 17:06 UTC by Peter Krempa
Modified: 2014-06-18 00:54 UTC (History)
7 users (show)

Fixed In Version: libvirt-1.1.1-3.el7
Doc Type: Bug Fix
Doc Text:
Clone Of: 1000973
Environment:
Last Closed: 2014-06-13 13:16:35 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
libvirtd.log (346.20 KB, text/plain)
2013-09-02 07:26 UTC, Xuesong Zhang 		no flags Details
backtrace (7.48 KB, text/plain)
2013-09-02 07:27 UTC, Xuesong Zhang 		no flags Details
View All

Description Peter Krempa 2013-08-29 17:06:39 UTC 
+++ This bug was initially created as a clone of Bug #1000973 +++

Description of problem:
libvirtd crash when starting  guest after cold-plug interface with hostdev network


Version-Release number of selected component (if applicable):
libvirt-0.10.2-23.el6.x86_64 

How reproducible:
100%

Steps to Reproduce:
# virsh start r6
Domain r6 started

# virsh net-list --all
Name                 State      Autostart     Persistent
--------------------------------------------------
default              active     no            yes
hostdev-net1         active     no            yes

# lspci|grep 11:10
11:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)


# virsh net-dumpxml hostdev-net1
<network>
  <name>hostdev-net1</name>
  <uuid>c039ec41-79cd-9418-76bc-a74a7ce29715</uuid>
  <forward mode='hostdev' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x1'/>
  </forward>
</network>

# virsh destroy r6
Domain r6 destroyed

# cat vfpool.xml
<interface type='network'>
   <source network='hostdev-net1'/>
</interface>

# virsh attach-device r6 vfpool.xml --config
Device attached successfully

# virsh start r6
error: Failed to start domain r6
error: End of file while reading data: Input/output error
error: One or more references were leaked after disconnect from the hypervisor
error: Failed to reconnect to the hypervisor


Actual results:
libvirtd crash 

Expected results:
libvirtd works fine 

Additional info:

--- Additional comment from Peter Krempa on 2013-08-29 10:58:03 CEST ---

Fixed upstream:

commit 50348e6edfa10ddb61929bf95a1c4820a9614e19
Author: Peter Krempa <pkrempa>
Date:   Tue Aug 27 19:06:18 2013 +0200

    qemu: Remove hostdev entry when freeing the depending network entry
    
    When using a <interface type="network"> that points to a network with
    hostdev forwarding mode a hostdev alias is created for the network. This
    allias is inserted into the hostdev list, but is backed with a part of
    the network object that it is connected to.
    
    When a VM is being stopped qemuProcessStop() calls
    networkReleaseActualDevice() which eventually frees the memory for the
    hostdev object. Afterwards when the domain definition is being freed by
    virDomainDefFree() an invalid pointer is accessed by
    virDomainHostdevDefFree() and may cause a crash of the daemon.
    
    This patch removes the entry in the hostdev list before freeing the
    depending memory to avoid this issue.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1000973
Comment 3 Xuesong Zhang 2013-09-02 05:42:15 UTC 
Verify this bug in build libvirt-1.1.1-3.el7.x86_64, the bug is fixed, so change the status to verify.

steps:
1. prepare one healthy guest.
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 4     r7                             running

2. prepare one network with hostdev forward mode 
# virsh net-dumpxml hostdev-net1
<network connections='1'>
  <name>hostdev-net1</name>
  <uuid>6208c354-0713-4d90-b12c-a893546e44ae</uuid>
  <forward mode='hostdev' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x1'/>
  </forward>
</network>

# virsh net-list --all
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 br1                  inactive   no            yes
 default              active     yes           yes
 hostdev-net1         active     no            yes

3. check the vf in the host
# lspci|grep 82576
0e:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0e:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0f:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
10:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
10:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
11:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

4. prepare one xml like the following one:
# cat vfpool.xml 
<interface type='network'>
   <source network='hostdev-net1'/>
</interface>

5. destroy the guest, and attach the interface to the guest with config option.
# virsh destroy r7
Domain r7 destroyed

# virsh attach-device r7 vfpool.xml --config
Device attached successfully

6. check the libvirtd status
# service libvirtd status
Redirecting to /bin/systemctl status  libvirtd.service
libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
   Active: active (running) since 一 2013-09-02 01:30:46 EDT; 3min 32s ago
 Main PID: 3769 (libvirtd)
   CGroup: name=systemd:/system/libvirtd.service
           ├─1046 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf
           ├─3769 /usr/sbin/libvirtd
           └─4182 /usr/libexec/qemu-kvm -name r7 -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -bios /usr...

9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...14
9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...15
9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...16
9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...17
9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...18
9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...19
9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...20
9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...21
9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...24
9月 02 01:34:10 SRIOV2.qe.lab.eng.nay.redhat.com libvirtd[3769]: 2013-09-02 05:34:10.759+0000: 4181: debug...27

7. make sure the interface is cold-pluged.
# virsh dumpxml r7|grep hostdev -A 5
      <source network='hostdev-net1'/>
      <model type='rtl8139'/>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
    

# ps -ef|grep qemu
qemu      4182     1 42 01:34 ?        00:00:17 /usr/libexec/qemu-kvm -name r7 ...... -device pci-assign,configfd=23,host=0f:10.0,id=hostdev0,bus=pci.0,addr=0x4
Comment 4 Xuesong Zhang 2013-09-02 07:25:34 UTC 
hi, 

  While I do some other operations, the libvirtd service crashed again. Seems like the same issue. 

Here is the reproduce steps:
1. prepare one healthy guest.
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 4     r7                             running

2. prepare one network with hostdev forward mode 
# virsh net-dumpxml hostdev-net1
<network connections='1'>
  <name>hostdev-net1</name>
  <uuid>6208c354-0713-4d90-b12c-a893546e44ae</uuid>
  <forward mode='hostdev' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x1'/>
  </forward>
</network>

# virsh net-list --all
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 br1                  inactive   no            yes
 default              active     yes           yes
 hostdev-net1         active     no            yes

3. check the vf in the host
# lspci|grep 82576
0e:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0e:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0f:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
10:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
10:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
11:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

4. prepare one xml like the following one:
# cat vfpool.xml 
<interface type='network'>
   <source network='hostdev-net1'/>
</interface>

5. hot-plug the vf to the guest.
# virsh attach-device r7 vfpool.xml 
Device attached successfully

6. make sure the vf is attached.
# virsh dumpxml r7
.......
<interface type='network'>
      <mac address='52:54:00:67:c2:3f'/>
      <source network='hostdev-net1'/>
      <model type='rtl8139'/>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>
......

7. new one vf xml for detach, the content is same with the one in dumpxml.
# cat vf-detach.xml 
<interface type='network'>
      <mac address='52:54:00:67:c2:3f'/>
      <source network='hostdev-net1'/>
      <model type='rtl8139'/>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>

8. detach the vf from guest.
# virsh detach-device r7 vf-detach.xml 
Device detached successfully

9. destroy the guest.
# virsh destroy r7
error: Failed to destroy domain r7
error: End of file while reading data: Input/output error
error: One or more references were leaked after disconnect from the hypervisor
error: Failed to reconnect to the hypervisor


Actual result:
libvirtd crashed in step9.

Expected result:
libvirtd should be running.
Comment 5 Xuesong Zhang 2013-09-02 07:26:24 UTC 
Created attachment 792726 [details]
libvirtd.log
Comment 6 Xuesong Zhang 2013-09-02 07:27:08 UTC 
Created attachment 792727 [details]
backtrace
Comment 7 Xuesong Zhang 2013-09-02 07:28:28 UTC 
Upload the libvirtd.log and backtrace in comment 5 and comment 6.
Change the bug status to "Assigned".
Comment 8 Xuesong Zhang 2013-09-02 10:05:54 UTC 
After contact with Jiri, file one new bug for the issue in comment 4.
https://bugzilla.redhat.com/show_bug.cgi?id=1003537

So, change this bug to verify according to comment 3.
Comment 9 Ludek Smid 2014-06-13 13:16:35 UTC 
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.
Note You need to log in before you can comment on or make changes to this bug.