Bug 971313

Created attachment 757573 [details]
libvirt debug log

> error: Unable to read from monitor: Connection reset by peer

Whenever you see this error, you need to gather the guest's qemu logfile, which is in /var/log/libvirt/qemu/$guestname.log ($guestname is rhel7 in this case).

Also, can you verify that the guest remains running properly *after* the device is detached the first time, right up until the 2nd attach attempt?

Created attachment 764884 [details]
guest qemu log

Attached guest qemu log

The state of guest become shutoff when the device detached the first time.

# virsh start rhel7
Domain rhel7 started

# virsh attach-device rhel7 vf.xml
Device attached successfully

# virsh detach-device rhel7 vf.xml
Device detached successfully

# virsh list
 Id    Name                           State
----------------------------------------------------
 4     rhel7                          running

wait for some moment , the guest shut off 

# virsh list
 Id    Name                           State
----------------------------------------------------

(In reply to hongming from comment #4)
> 
> Attached guest qemu log

That's an interesting log, but it doesn't look like it is /var/log/libvirt/qemu/rhel7.log.

The qemu logfile would contain things such as the qemu commandline that was used start start qemu, and any error messages that qemu generated after it was started.

The logfile that you've attached attachment 764884 [details] contains a lot of messages from libvirtd, followed by what looks like a very strange error message about the kernel being unable to write to a pci device. I don't recognize it, so I'm asking the qemu people for assistance.

Can you try this with selinux disabled to see if the behavior is different?

Also check for any new AVCs in /var/log/audit/audit.log.

If selinux is disabled , it works fine. the bug can't be reproduced.

If selinux is enforing , it can be reproduced.
# getenforce 
Enforcing

# virsh start rhel7
Domain rhel7 started

# virsh attach-device rhel7 vf.xml
Device attached successfully

# virsh detach-device rhel7 vf.xml
Device detached successfully

# cat /var/log/audit/audit.log|grep avc
type=AVC msg=audit(1372230114.643:139): avc:  denied  { write } for  pid=1711 comm="qemu-kvm" path="/sys/devices/pci0000:00/0000:00:1c.6/0000:0c:00.0/0000:0d:02.0/0000:0f:10.3/config" dev="sysfs" ino=27333 scontext=system_u:system_r:svirt_t:s0:c150,c740 tcontext=system_u:object_r:sysfs_t:s0 tclass=file

Created attachment 765495 [details]
audit log

This is the offending AVC:

type=AVC msg=audit(1372240465.252:542): avc:  denied  { write } for  pid=4129 comm="qemu-kvm" path="/sys/devices/pci0000:00/0000:00:1c.6/0000:0c:00.0/0000:0d:02.0/0000:0f:10.3/config" dev="sysfs" ino=27333 scontext=system_u:system_r:svirt_t:s0:c394,c836 tcontext=system_u:object_r:sysfs_t:s0 tclass=file

The fact that this works when the device is attached, but fails when the device is detached, implies that the selinux label on this resource is being "undone" too soon.

I have a theory about this that is a bit disturbing - when we send the device_del command to qemu, it returns almost immediately with success, but it hasn't *really* finished detaching the device. In the meantime, we happily proceed to reattach the device to the host driver, undo any cgroups that we had setup, and relabel everything in sysfs to prevent access by the qemu process.*But it may not be finished yet!*

So I think the solution to this problem is to implement a wait for the new qemu event that it produces when it is *really* finished with a device (does anyone remember the BZ number for the libvirt side of that?

I believe that this bug may have been fixed by the patch for Bug 984112, which is now available in the latest RHEL7 build- libvirt-1.1.1-1.el7. Can you please retest and see if that is the case.

If it is fixed, we should re-target this back to 7.0, then mark it as fixed in libvirt-1.1.1-1.el7.

QA testing for Bug 984112 ran into this same problem, so it seems it isn't yet fixed. See my comment in that bug.

They hit it when trying to reproduce the bug with an older package. In any case, this is supposed to be fixed by the patches for bug 984112 and it would be a bug in those patches if not. Thus, I'm moving this back to 7.0 with a TestOnly keyword.

This bug was blocked by bug 990987, I can verify it after bug 990987 fixed.

I can't reproduce it any more.

Version:
libvirt-1.1.1-15.el7.x86_64
qemu-kvm-rhev-1.5.3-21.el7.x86_64
kernel-3.10.0-61.el7.x86_64

1. Enable vfio module in kernel
[root@sriov1 ~]#  modprobe vfio_pci
[root@sriov1 ~]# lsmod|grep vfio
vfio_iommu_type1       17636  1 
vfio_pci               36474  1 
vfio                   20777  5 vfio_iommu_type1,vfio_pci
[root@sriov1 ~]#

2. Detach VF device from host
[root@sriov1 ~]# cat net.xml 
<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x10' function='0x4'/>
      </source>
    </hostdev>
[root@sriov1 ~]# 

[root@sriov1 ~]# virsh nodedev-dumpxml pci_0000_03_10_4
<device>
  <name>pci_0000_03_10_4</name>
  <path>/sys/devices/pci0000:00/0000:00:01.0/0000:03:10.4</path>
  <parent>pci_0000_00_01_0</parent>
  <driver>
    <name>igbvf</name>
  </driver>
...(snipped)

[root@sriov1 ~]# virsh nodedev-detach pci_0000_03_10_4
Device pci_0000_03_10_4 detached

[root@sriov1 ~]# virsh nodedev-dumpxml pci_0000_03_10_4
<device>
  <name>pci_0000_03_10_4</name>
  <path>/sys/devices/pci0000:00/0000:00:01.0/0000:03:10.4</path>
  <parent>pci_0000_00_01_0</parent>
  <driver>
    <name>vfio-pci</name>
  </driver>
...(snipped)

2. Repeat 3 times to attach/detach VF to domain r7
[root@sriov1 ~]# getenforce 
Enforcing
[root@sriov1 ~]# 

[root@sriov1 ~]# virsh attach-device r7 net.xml 
Device attached successfully

[root@sriov1 ~]# virsh detach-device r7 net.xml 
Device detached successfully

[root@sriov1 ~]# virsh attach-device r7 net.xml 
Device attached successfully

[root@sriov1 ~]# virsh detach-device r7 net.xml 
Device detached successfully

[root@sriov1 ~]# virsh attach-device r7 net.xml 
Device attached successfully

[root@sriov1 ~]# virsh detach-device r7 net.xml 
Device detached successfully

[root@sriov1 ~]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 4     r7                             running


We can get the expected results, and the bug 984112 has been verified.

This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.