Bug 1046598

This is another duplicate of Bug 1044595 (which is assigned to the kernel) - in short, legacy KVM device assignment has been disabled in RHEL7, but when libvirt tries to detect VFIO devidce assignment, it thinks it's not available because the vfio driver hasn't been loaded.

I'm marking this as a duplicate of 1044595. In the meantime, please run "modprobe vfio" prior to testing PCI device assignment - that should clear up the problem.

*** This bug has been marked as a duplicate of bug 1044595 ***

hi, Laine,

I'm sure I have load the vfio module manually before the hot-plug, it is not a duplicated bug.

Sorry for didn't any the load vfio steps in the bug description, here is my testing env with VFIO modules.

# lsmod |grep vfio
vfio_iommu_type1       17672  0 
vfio_pci               36569  0 
vfio                   24951  2 vfio_iommu_type1,vfio_pci
 
# cat /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts 
Y


(In reply to Laine Stump from comment #1)
> This is another duplicate of Bug 1044595 (which is assigned to the kernel) -
> in short, legacy KVM device assignment has been disabled in RHEL7, but when
> libvirt tries to detect VFIO devidce assignment, it thinks it's not
> available because the vfio driver hasn't been loaded.
> 
> I'm marking this as a duplicate of 1044595. In the meantime, please run
> "modprobe vfio" prior to testing PCI device assignment - that should clear
> up the problem.
> 
> *** This bug has been marked as a duplicate of bug 1044595 ***

What does

# ls -lZ /dev/vfio

(In reply to Miroslav Grepl from comment #5)
> What does
> 
> # ls -lZ /dev/vfio

Hi, Miroslav Grepl,

Thanks for your so quickly response, it's like the following one:

# ll -Z /dev/vfio/vfio 
crw-rw-rw-. root root system_u:object_r:vfio_device_t:s0 /dev/vfio/vfio
# ll -Z /dev/vfio
crw-rw-rw-. root root system_u:object_r:vfio_device_t:s0 vfio

I‘m sorry that paste the wrong log in comment 3, since there are several avc denied error in audit.log, the sealert analysis in comments 3 is not the right one.
Please reference the following right one:


SELinux is preventing /usr/libexec/qemu-kvm from write access on the file /sys/devices/pci0000:00/0000:00:19.0/config.

*****  Plugin catchall (100. confidence) suggests   **************************

If you believe that qemu-kvm should be allowed write access on the config file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# grep qemu-kvm /var/log/audit/audit.log | audit2allow -M mypol
# semodule -i mypol.pp

Additional Information:
Source Context                system_u:system_r:svirt_t:s0:c93,c472
Target Context                system_u:object_r:sysfs_t:s0
Target Objects                /sys/devices/pci0000:00/0000:00:19.0/config [ file
                              ]
Source                        qemu-kvm
Source Path                   /usr/libexec/qemu-kvm
Port                          <Unknown>
Host                          xuzhang3
Source RPM Packages           qemu-kvm-rhev-1.5.3-35.el7.x86_64
Target RPM Packages           
Policy RPM                    selinux-policy-3.12.1-112.el7.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     xuzhang3
Platform                      Linux xuzhang3 3.10.0-67.el7.x86_64 #1 SMP Tue Jan
                              7 18:01:25 EST 2014 x86_64 x86_64
Alert Count                   4
First Seen                    2014-01-10 14:41:50 CST
Last Seen                     2014-01-10 16:40:58 CST
Local ID                      6ae5410f-532f-4c38-b352-3212900a74bd

Raw Audit Messages
type=AVC msg=audit(1389343258.75:988): avc:  denied  { write } for  pid=13835 comm="qemu-kvm" path="/sys/devices/pci0000:00/0000:00:19.0/config" dev="sysfs" ino=10656 scontext=system_u:system_r:svirt_t:s0:c93,c472 tcontext=system_u:object_r:sysfs_t:s0 tclass=file


type=SYSCALL msg=audit(1389343258.75:988): arch=x86_64 syscall=recvmsg success=yes exit=EPERM a0=15 a1=7fffe4df5b90 a2=40000000 a3=0 items=0 ppid=1 pid=13835 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm=qemu-kvm exe=/usr/libexec/qemu-kvm subj=system_u:system_r:svirt_t:s0:c93,c472 key=(null)

Hash: qemu-kvm,svirt_t,sysfs_t,file,write

Just to make sure there are no other problems in libvirt causing the failure - have you verified that the attach-device is successful if you set selinux to permissive mode?

(In reply to Laine Stump from comment #8)
> Just to make sure there are no other problems in libvirt causing the failure
> - have you verified that the attach-device is successful if you set selinux
> to permissive mode?

Yeah, the hot-plug will be succussfully after I set selinux as permissive.

Please see the following steps:
# getenforce
Permissive

# virsh attach-device a Inter-hostdev.xml 
Device attached successfully

# cat Inter-hostdev.xml 
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address bus='0x00' slot='0x19' function='0x0'/>
  </source>
</hostdev>

Add another failed scenario here, it also caused by selinux deny.

package version:
kernel-3.10.0-73.el7.x86_64
qemu-kvm-rhev-1.5.3-38.el7.x86_64
libvirt-1.1.1-18.el7.x86_64
selinux-policy-3.12.1-118.el7.noarch

steps:
1. check the selinux status
# getenforce 
Enforcing

2. start one guest
[root@xuzhangtest2 xuzhang]# virsh start a
Domain a started

3. didn't load the vfio realted modules.
[root@xuzhangtest2 xuzhang]# lsmod|grep vfio

4. prepare one pci xml, specify the vfio driver in xml.
# cat hostdev-vfio.xml 
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>    
  <source>
        <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
      </source>
    </hostdev>

5. hot-plug the pci to guest, but failed.
[root@xuzhangtest2 xuzhang]# virsh attach-device a hostdev-vfio.xml 
error: Failed to attach device from hostdev-vfio.xml
error: internal error: unable to execute QEMU command 'device_add': Device initialization failed.

6. check the audit log info
# ausearch -m avc
----
time->Tue Jan 21 14:46:43 2014
type=SYSCALL msg=audit(1390286803.004:1233): arch=c000003e syscall=2 success=no exit=-13 a0=7fffad2c3480 a1=80002 a2=0 a3=0 items=0 ppid=1 pid=26105 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c489,c878 key=(null)
type=AVC msg=audit(1390286803.004:1233): avc:  denied  { read write } for  pid=26105 comm="qemu-kvm" name="2" dev="devtmpfs" ino=67428 scontext=system_u:system_r:svirt_t:s0:c489,c878 tcontext=system_u:object_r:device_t:s0 tclass=chr_file


7. set the selinux to permissive mode.
[root@xuzhangtest2 xuzhang]# setenforce 0
[root@xuzhangtest2 xuzhang]# getenforce 
Permissive

8. restart the guest.
[root@xuzhangtest2 xuzhang]# virsh destroy a
Domain a destroyed
[root@xuzhangtest2 xuzhang]# virsh start a
Domain a started

9. hot-plug the pci to guest in the permissive mode, it will be successful.
[root@xuzhangtest2 xuzhang]# virsh attach-device a hostdev-vfio.xml 
Device attached successfully

Addtional info:
Please see attached for the sealert analyses log





[Conclusion] Hot-plug pci to guest will be failed whether specify the vfio driver in xml or not. The error message is different, seems the root reason is not same, but they are all caused by selinux. See the following for the details:
1) while didn't specify the vfio driver in the pci xml, the sealert analyses result is the following one:
SELinux is preventing /usr/libexec/qemu-kvm from write access on the file /sys/devices/pci0000:00/0000:00:19.0/config

2) while specify the vfio driver in the pci xml, the sealert analyses result is the following one:
SELinux is preventing /usr/libexec/qemu-kvm from 'read, write' accesses on the chr_file 2.

Created attachment 852993 [details]
sealert analyses result

Could you please re-test it with full auditing

# echo "-w /etc/shadow -p w" >> /etc/audit/audit.rules
# systemctl restart auditd.service

As far as I know, access to the device's "config" file is only needed for legacy kvm device assignment, and if we're disabling that, it's likely that we want to disallow access to config with selinux as well - Alex?

Also, with the most recent kernel and qemu-kvm, we shouldn't be trying to do legacy kvm device assignment even if <driver name='vfio'/> isn't specified - see Bug 1044595.

Created attachment 853824 [details]
scenario_1_avc_audit_log

Created attachment 853825 [details]
scenario_2_avc_audit_log

Created attachment 853826 [details]
scenario_2_test_result

hi, Miroslav and Laine.

After add rules to the audit service, I restart the audited serivce and run the 2 scenarios again with the latest build:

package version:
kernel-3.10.0-76.el7.x86_64
qemu-kvm-rhev-1.5.3-39.el7.x86_64
libvirt-1.1.1-18.el7.x86_64
selinux-policy-3.12.1-120.el7.noarch
audit-2.3.3-2.el7.x86_64

Scenario 1:
Do not specify the <driver name='vfio'/> in xml, details step see the description of this bug.
The details avc audit log see comment 14.

Scenario 2:
Specify the <driver name='vfio'/> in xml, details steps see comment 10.
The details avc audit log see comment 15.
As for the scenario 2, the test result is a little different with before now. The hot-plug will be failed sometimes, not failed every time. You can see the comment 16 for the details test result.

hi Alex, 

Laine ping you in comment 13, would you please help to check that comment? Thanks.

I wanted to see

name="/dev/vfio/2"

The problem is we don't have a default label for this one.

(In reply to Laine Stump from comment #13)
> As far as I know, access to the device's "config" file is only needed for
> legacy kvm device assignment, and if we're disabling that, it's likely that
> we want to disallow access to config with selinux as well - Alex?
> 
> Also, with the most recent kernel and qemu-kvm, we shouldn't be trying to do
> legacy kvm device assignment even if <driver name='vfio'/> isn't specified -
> see Bug 1044595.

vfio does not use pci sysfs (/sys/bus/pci/devices/xxxx:xx:xx.x/config).  If qemu-kvm is trying to access that, it's likely started with pci-assign.  We'd need to see how libvirt launched it to know.

Created attachment 854209 [details]
message.log_20140123

Created attachment 854210 [details]
libvirtd_debug.log_20140123

(In reply to Miroslav Grepl from comment #19)
> I wanted to see
> 
> name="/dev/vfio/2"
> 
> The problem is we don't have a default label for this one.

As for scenario 2, there isn't file "/dev/vfio/2" while meet that error.

# ll /dev/vfio/2 -Z
ls: cannot access /dev/vfio/2: No such file or directory

Attached the libvirtd and message log for your reference. Please see comment 21 and comment 22.

(In reply to Alex Williamson from comment #20)
> (In reply to Laine Stump from comment #13)
> > 
> > Also, with the most recent kernel and qemu-kvm, we shouldn't be trying to do
> > legacy kvm device assignment even if <driver name='vfio'/> isn't specified -
> > see Bug 1044595.
> 
> vfio does not use pci sysfs (/sys/bus/pci/devices/xxxx:xx:xx.x/config).  If
> qemu-kvm is trying to access that, it's likely started with pci-assign. 

To quote that suit salesman - "I guarantee it". Definitely that's what libvirt is doing, and if it's doing that, it's likely because it didn't detect that vfio was available. And I had thought that the patch described in Bug 1044595 should fix that problem, and that it was now in the kernel.

> We'd need to see how libvirt launched it to know.

Actually I'm pretty sure I know how libvirt launched it (otherwise, as you say, qemu wouldn't be trying to access the device config in pci sysfs :-)

Aside from the question of whether or not Bug 1044595 is really fixed, I wonder if we should be removing or retaining the selinux rules that enable access to pci sysfs by qemu (since they're only used by a mode that is going to be disabled in RHEL7)

(In reply to Zhang Xuesong from comment #23)
> 
> As for scenario 2, there isn't file "/dev/vfio/2" while meet that error.
> 
> # ll /dev/vfio/2 -Z
> ls: cannot access /dev/vfio/2: No such file or directory

That device node is created when the device is bound to vfio-pci. By the time libvirt has logged an error and failed the operation, it has already re-attached the device to the host driver, so the /dev/vfio/* node will no longer exist.

*** Bug 1056867 has been marked as a duplicate of this bug. ***

How does it look with latest policy builds?

# rpm -qa selinux-policy\*
selinux-policy-targeted-3.12.1-124.el7.noarch
selinux-policy-minimum-3.12.1-124.el7.noarch
selinux-policy-mls-3.12.1-124.el7.noarch
selinux-policy-sandbox-3.12.1-124.el7.noarch
selinux-policy-devel-3.12.1-124.el7.noarch
selinux-policy-3.12.1-124.el7.noarch
selinux-policy-doc-3.12.1-124.el7.noarch
# modprobe vfio
# ls -l /dev/vfio/vfio 
crw-rw-rw-. 1 root root 10, 196 Feb  5 15:44 /dev/vfio/vfio
# ls -Z /dev/vfio/vfio 
crw-rw-rw-. root root system_u:object_r:device_t:s0    /dev/vfio/vfio
# matchpathcon /dev/vfio/vfio 
/dev/vfio/vfio	system_u:object_r:device_t:s0
# semanage fcontext -l | grep vfio
/dev/vfio/(vfio)?[0-9]+                            character device   system_u:object_r:vfio_device_t:s0 
#

(In reply to Miroslav Grepl from comment #27)
> How does it look with latest policy builds?

hi, Miroslav Grepl,

Sorry for the late response, since we are in Chinese Spring Festival, just come back from the vacation.

Tested build version:
libvirt-1.1.1-22.el7.x86_64
qemu-kvm-rhev-1.5.3-45.el7.x86_64
kernel-3.10.0-84.el7.x86_64
selinux-policy-3.12.1-124.el7.noarch


As for the 2 scenarios in this bug, they are all failed with same selinux AVC deny now. Please see the following audit log for reference.


#ausearch -m avc
time->Mon Feb 10 15:09:14 2014
type=PATH msg=audit(1392016154.986:657): item=0 name="/dev/vfio/vfio" inode=9509 dev=00:05 mode=020666 ouid=0 ogid=0 rdev=0a:c4 obj=system_u:object_r:device_t:s0 objtype=NORMAL
type=CWD msg=audit(1392016154.986:657):  cwd="/"
type=SYSCALL msg=audit(1392016154.986:657): arch=c000003e syscall=2 success=no exit=-13 a0=7f0eab115b16 a1=80002 a2=0 a3=0 items=1 ppid=1 pid=14852 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c306,c821 key=(null)
type=AVC msg=audit(1392016154.986:657): avc:  denied  { read write } for  pid=14852 comm="qemu-kvm" name="vfio" dev="devtmpfs" ino=9509 scontext=system_u:system_r:svirt_t:s0:c306,c821 tcontext=system_u:object_r:device_t:s0 tclass=chr_file


#sealert
SELinux is preventing /usr/libexec/qemu-kvm from 'read, write' accesses on the chr_file .

*****  Plugin device (91.4 confidence) suggests   ****************************

If you want to allow qemu-kvm to have read write access on the  chr_file
Then you need to change the label on  to a type of a similar device.
Do
# semanage fcontext -a -t SIMILAR_TYPE '$FIX_TARGET_PATH'
# restorecon -v '$FIX_TARGET_PATH'

*****  Plugin catchall (9.59 confidence) suggests   **************************

If you believe that qemu-kvm should be allowed read write access on the  chr_file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# grep qemu-kvm /var/log/audit/audit.log | audit2allow -M mypol
# semodule -i mypol.pp

Additional Information:
Source Context                system_u:system_r:svirt_t:s0:c306,c821
Target Context                system_u:object_r:device_t:s0
Target Objects                 [ chr_file ]
Source                        qemu-kvm
Source Path                   /usr/libexec/qemu-kvm
Port                          <Unknown>
Host                          xuzhangtest2
Source RPM Packages           qemu-kvm-rhev-1.5.3-45.el7.x86_64
Target RPM Packages           
Policy RPM                    selinux-policy-3.12.1-124.el7.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     xuzhangtest2
Platform                      Linux xuzhangtest2 3.10.0-84.el7.x86_64 #1 SMP Tue
                              Feb 4 16:28:19 EST 2014 x86_64 x86_64
Alert Count                   1
First Seen                    2014-02-10 15:09:14 CST
Last Seen                     2014-02-10 15:09:14 CST
Local ID                      f609934d-bf62-4af2-a0e6-c39ac3f90dde

Raw Audit Messages
type=AVC msg=audit(1392016154.986:657): avc:  denied  { read write } for  pid=14852 comm="qemu-kvm" name="vfio" dev="devtmpfs" ino=9509 scontext=system_u:system_r:svirt_t:s0:c306,c821 tcontext=system_u:object_r:device_t:s0 tclass=chr_file


type=SYSCALL msg=audit(1392016154.986:657): arch=x86_64 syscall=open success=no exit=EACCES a0=7f0eab115b16 a1=80002 a2=0 a3=0 items=1 ppid=1 pid=14852 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm=qemu-kvm exe=/usr/libexec/qemu-kvm subj=system_u:system_r:svirt_t:s0:c306,c821 key=(null)

type=CWD msg=audit(1392016154.986:657): cwd=/

type=PATH msg=audit(1392016154.986:657): item=0 name=/dev/vfio/vfio inode=9509 dev=00:05 mode=020666 ouid=0 ogid=0 rdev=0a:c4 obj=system_u:object_r:device_t:s0 objtype=NORMAL

Hash: qemu-kvm,svirt_t,device_t,chr_file,read,write

Another info for comment 29, not only pci hot-plug, the pci assignment will also meet the avc deny now. 
The pci assignment is working well before, you can see the following comments. https://bugzilla.redhat.com/show_bug.cgi?id=1044595#c14

The /dev/vfio/vfio device is labeled incorrectly by selinux-policy 3.12.1-124.el7. It should be fixed in next version.

(In reply to Milos Malik from comment #31)
> The /dev/vfio/vfio device is labeled incorrectly by selinux-policy
> 3.12.1-124.el7. It should be fixed in next version.

Yes.

Tested with selinux-policy-3.12.1-125.el7.noarch, this bug is fixed.

package version:
selinux-policy-3.12.1-125.el7.noarch
libvirt-1.1.1-23.el7.x86_64
qemu-kvm-rhev-1.5.3-45.el7.x86_64
kernel-3.10.0-86.el7.x86_64

4 scenarios:
Scenario 1:
hot-plug pci to one running guest, didn't specify the driver in xml.
# cat hostdev.xml 
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

Scenario 2:
hot-plug pci to one running guest, specify the driver in the xml
# cat hostdev-driver.xml 
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
  <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

Scenario 3:
assign one pci to one shutoff guest, didn't specify the driver in xml. Start the guest.
Scenario 4:
assign one pci to one shutoff guest, specify the driver in the xml. Start the guest.


All the 4 scenarios are passed, the vfio module will be loaded automatically.

Thank you for testing it.

This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.