Bug 1012834

Summary: the virtual interface can't be hotpluged with command "attach-interface" while the guest contains VF
Product: Red Hat Enterprise Linux 6 Reporter: Xuesong Zhang <xuzhang>
Component: libvirtAssignee: Laine Stump <laine>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.5CC: dyuan, honzhang, jdenemar, jmiao, laine, mzhan, rbalakri
Target Milestone: rcKeywords: Upstream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-0.10.2-36.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1012824 Environment:
Last Closed: 2014-10-14 04:17:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1012824    
Bug Blocks:    

Description Xuesong Zhang 2013-09-27 08:48:48 UTC
+++ This bug was initially created as a clone of Bug #1012824 +++

Description
After hot-plug/cold-plug one hostdev network interface, can't attach the interface from default network. This issue is about command "attach-interface".

Version:
libvirt-1.1.1-5.el7.x86_64
qemu-kvm-1.5.3-3.el7.x86_64
kernel-3.10.0-23.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. prepare 2 types network on the SR-IOV host: one is default network, the other one is the hostdev network.
# virsh net-dumpxml hostnet
<network connections='2'>
  <name>hostnet</name>
  <uuid>c1fb4ead-21b8-4d69-8ad9-669c55b3dfc7</uuid>
  <forward mode='hostdev' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
  </forward>
</network>

# virsh net-dumpxml default
<network>
  <name>default</name>
  <uuid>74b8fbe6-87d3-4e75-90e9-0a92623e1040</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0' />
  <mac address='52:54:00:6d:3f:27'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254' />
    </dhcp>
  </ip>
</network>

2. prepare one running guest

3. prepare one guest, cold-plug the vf to the guest like following:
# virsh dumpxml a
......
<interface type='network'>
      <mac address='52:54:00:e6:1f:fe'/>
      <source network='hostnet'/>
      <model type='rtl8139'/>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
......

4. start the guest

5. hot-plug the interface of default network to the guest:
# virsh attach-interface a network default
error: Failed to attach interface
error: internal error: Unable to determine device index for network device

6. hot-plug the interface of hostdev network to the guest:
# virsh attach-interface a network hostnet
Interface attached successfully

7. destroy the guest, then edit the guest and delete the VF, start the guest again.

8. hot-plug the interface of default network to the guest:
# virsh attach-interface a network default
Interface attached successfully

9. hot-plug the interface of hostdev network to the guest:
# virsh attach-interface a network hostnet
Interface attached successfully

10. hot-plug the interface of default network to the guest:
# virsh attach-interface a network default
error: Failed to attach interface
error: internal error: Unable to determine device index for network device

Actual results:
In step5 and step10, after the guest cold-plug/hot-plug one VF, the interface of default network can't be hot-plug to the guest.

Expected results:
In step5 and step10, the interface of default network should be hot-plug successfully to the guest whether there is other type network interface or not.

Additional info:

Comment 2 Laine Stump 2013-10-03 11:30:38 UTC
Fix posted upstream:

https://www.redhat.com/archives/libvir-list/2013-October/msg00152.html

Comment 3 Laine Stump 2013-10-03 15:12:11 UTC
Pushed upstream:

commit 9881bfed2541faa428372b4513518b4b9ae1ab15
Author: Laine Stump <laine>
Date:   Thu Oct 3 13:02:35 2013 +0300

    qemu: check actual netdev type rather than config netdev type during init

Comment 9 Jincheng Miao 2014-05-20 09:50:54 UTC
Hi Laine,

I met a problem during testing this bug, destroying the guest which attached a VF interface will report an error:

# virsh attach-device rhel65 hostdev.xml 
Device attached successfully

# virsh destroy rhel65
error: Failed to destroy domain rhel65
error: operation failed: failed to kill qemu process with SIGTERM

# ps -ef | grep qemu
root     11258  4695  0 05:10 pts/2    00:00:00 grep qemu

some logs are:
2014-05-20 09:09:55.064+0000: 9794: warning : qemuProcessKill:4254 : Timed out waiting after SIGTERM to process 9908, sending SIGKILL
2014-05-20 09:10:00.067+0000: 9794: warning : qemuProcessKill:4286 : Timed out waiting after SIGKILL to process 9908
2014-05-20 09:10:00.067+0000: 9794: debug : virObjectRef:168 : OBJECT_REF: obj=0x7f7444122840
2014-05-20 09:10:00.067+0000: 9794: debug : virObjectUnref:135 : OBJECT_UNREF: obj=0x7f7444122840
2014-05-20 09:10:00.067+0000: 9794: error : qemuDomainDestroyFlags:2111 : operation failed: failed to kill qemu process with SIGTERM

Further information:
I try to attach a normal PCI device to guest, and no error happened.

Although I think this problem is not related to this bug fix, I just need your confirmation. If so, I will report a new one to track it.

Comment 10 Jincheng Miao 2014-05-22 06:41:02 UTC
(In reply to Jincheng Miao from comment #9)
> some logs are:
> 2014-05-20 09:09:55.064+0000: 9794: warning : qemuProcessKill:4254 : Timed
> out waiting after SIGTERM to process 9908, sending SIGKILL
> 2014-05-20 09:10:00.067+0000: 9794: warning : qemuProcessKill:4286 : Timed
> out waiting after SIGKILL to process 9908
> 2014-05-20 09:10:00.067+0000: 9794: debug : virObjectRef:168 : OBJECT_REF:
> obj=0x7f7444122840
> 2014-05-20 09:10:00.067+0000: 9794: debug : virObjectUnref:135 :
> OBJECT_UNREF: obj=0x7f7444122840
> 2014-05-20 09:10:00.067+0000: 9794: error : qemuDomainDestroyFlags:2111 :
> operation failed: failed to kill qemu process with SIGTERM

For the same machine in rhel7 libvirt-1.1.1-29.el7.x86_64, there is no this problem.

Comment 11 Jincheng Miao 2014-05-23 06:26:54 UTC
This SIGTERM problem is not related to this fix, I tested on another 82599 machine, and no happened.

In latest libvirt, there is no error like "Unable to determine device index for network device"

# virsh net-dumpxml hostnet
<network>
  <name>hostnet</name>
  <uuid>1d257516-0d82-70c9-d0c8-dcd8ee288175</uuid>
  <forward mode='hostdev' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x44' slot='0x10' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x44' slot='0x10' function='0x0'/>
  </forward>
</network>

# virsh edit rhel65
    <interface type='network'>
      <mac address='52:54:00:ed:8d:00'/>
      <source network='hostnet'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </interface>

# virsh start rhel65
Domain rhel65 started

# virsh attach-interface rhel65 network default
Interface attached successfully

# virsh destroy rhel65
Domain rhel65 destroyed

# virsh edit rhel65
Domain rhel65 XML configuration edited.

# virsh start rhel65
Domain rhel65 started

# virsh attach-interface rhel65 network default
Interface attached successfully

# virsh attach-interface rhel65 network hostnet
Interface attached successfully

# virsh attach-interface rhel65 network default
Interface attached successfully

attach-interface works well, so I choose to change the status to VERIFIED.

Comment 13 errata-xmlrpc 2014-10-14 04:17:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1374.html