RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1196185 - libvirt doesn't set permissions for VFIO endpoint
Summary: libvirt doesn't set permissions for VFIO endpoint
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.1
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Michal Privoznik
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-02-25 12:55 UTC by Martin Polednik
Modified: 2015-11-19 06:17 UTC (History)
7 users (show)

Fixed In Version: libvirt-1.2.14-1.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-11-19 06:17:34 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2202 0 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2015-11-19 08:17:58 UTC

Description Martin Polednik 2015-02-25 12:55:03 UTC 
Description of problem:

Libvirt fails to set ownership of /dev/vfio/X (where X is used iommu group) if qemu is not running under root.


Version-Release number of selected component (if applicable):

libvirt-daemon-1.2.13-1.el7.x86_64
libvirt-daemon-driver-qemu-1.2.13-1.el7.x86_64
libvirt-lock-sanlock-1.2.13-1.el7.x86_64
libvirt-daemon-driver-storage-1.2.13-1.el7.x86_64
libvirt-docs-1.2.13-1.el7.x86_64
libvirt-daemon-driver-nwfilter-1.2.13-1.el7.x86_64
libvirt-daemon-driver-secret-1.2.13-1.el7.x86_64
libvirt-daemon-config-network-1.2.13-1.el7.x86_64
libvirt-daemon-kvm-1.2.13-1.el7.x86_64
libvirt-debuginfo-1.2.13-1.el7.x86_64
libvirt-daemon-driver-network-1.2.13-1.el7.x86_64
libvirt-daemon-driver-interface-1.2.13-1.el7.x86_64
libvirt-daemon-config-nwfilter-1.2.13-1.el7.x86_64
libvirt-1.2.13-1.el7.x86_64
libvirt-login-shell-1.2.13-1.el7.x86_64
libvirt-python-1.2.8-7.el7.x86_64
libvirt-daemon-driver-nodedev-1.2.13-1.el7.x86_64
libvirt-devel-1.2.13-1.el7.x86_64
libvirt-client-1.2.13-1.el7.x86_64
libvirt-daemon-driver-lxc-1.2.13-1.el7.x86_64
libvirt-daemon-lxc-1.2.13-1.el7.x86_64
qemu-img-rhev-1.5.3-60.el7_0.10.x86_64
qemu-kvm-rhev-1.5.3-60.el7_0.10.x86_64
qemu-kvm-tools-rhev-1.5.3-60.el7_0.2.x86_64
ipxe-roms-qemu-20130517-6.gitc4bce43.el7.noarch
qemu-kvm-common-rhev-1.5.3-60.el7_0.10.x86_64



How reproducible:

Always

Steps to Reproduce:

1. Set qemu to run under non-root user,
2. create a VM with nodedev attached (it's best to use sriov nic),
3. run the VM.

Actual results:

VM doesn't start with errors
qemu-kvm: -device vfio-pci,host=02:10.2,id=hostdev0,bus=pci.0,addr=0x8: vfio: error opening /dev/vfio/23: Permission denied
qemu-kvm: -device vfio-pci,host=02:10.2,id=hostdev0,bus=pci.0,addr=0x8: vfio: failed to get group 23
qemu-kvm: -device vfio-pci,host=02:10.2,id=hostdev0,bus=pci.0,addr=0x8: Device initialization failed.
qemu-kvm: -device vfio-pci,host=02:10.2,id=hostdev0,bus=pci.0,addr=0x8: Device 'vfio-pci' could not be initialized


Expected results:

VM starts normally.


Additional info:

This can be avoided by creating udev rule that sets the ownership of /dev/vfio/X endpoints. Also, this issue exists when using managed="no" mode (where I guess it should not be a bug) but also when using managed="yes" mode.
Comment 1 Michal Privoznik 2015-03-31 13:25:08 UTC 
This is fixed upstream already by:

commit f0bd70a940de690216c538b0ab1b71c8a7d2fbb6
Author:     Laine Stump <laine>
AuthorDate: Thu Apr 25 06:37:21 2013 -0400
Commit:     Laine Stump <laine>
CommitDate: Thu Apr 25 21:28:43 2013 -0400

    security: update hostdev labelling functions for VFIO
    
    Legacy kvm style pci device assignment requires changes to the
    labelling of several sysfs files for each device, but for vfio device
    assignment, the only thing that needs to be relabelled/chowned is the
    "group" device for the group that contains the device to be assigned.

v1.0.5-rc1

However, there were some subsequent fixes.
Comment 3 Shanzhi Yu 2015-07-15 13:33:43 UTC 
I'm trying to reproduce this bug with 
libvirt-1.2.13-1.el7.x86_64
qemu-kvm-rhev-2.3.0-9.el7.x86_64

1. Set qemu to run under non-root user

# echo '
user = "test"
group = "test"' >> /etc/libvirt/qemu.conf

# systemctl restart  libvirtd.service

2. Define a guest with hostdev attached

# virsh dumpxml r7|grep hostdev -A8
    <interface type='hostdev'>
      <mac address='52:54:00:6d:90:02'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
    <serial type='pty'>

3. Try to start guest 

# virsh start r7 
Domain r7 started

# ps aux|grep qemu|grep vfio 
test      9566 63.2 13.8 1516168 1115372 ?     SLl  21:24   0:21 /usr/libexec/qemu-kvm -name r7 -S -machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 37367d9f-dd35-4425-952c-70024253a082 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/r7.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/var/lib/libvirt/images/rhel7.img,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1 -device vfio-pci,host=03:10.0,id=hostdev0,bus=pci.0,addr=0x9 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on

# ll /dev/vfio/* 
crw-------. 1 test test 245,   0 Jul 15 20:50 /dev/vfio/23
crw-rw-rw-. 1 root root  10, 196 Jul 15 16:25 /dev/vfio/vfio

So I can succeed booting up the guest with hostdev attached(sriov nic). 

Martin,

Can you show me details to reproduce this bug or correct my steps above?

Thanks
Comment 4 Shanzhi Yu 2015-07-15 13:43:19 UTC 
# virsh start r7 
Domain r7 started


# virsh dumpxml r7|grep hostdev -A 5
    <interface type='hostdev'>
      <mac address='52:54:00:6d:90:02'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target port='0'/>
--
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x10' function='0x2'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </hostdev>

# ll /dev/vfio/* 
crw-------. 1 test test 245,   0 Jul 15 20:50 /dev/vfio/23
crw-------. 1 test test 245,   1 Jul 15 21:41 /dev/vfio/24
crw-rw-rw-. 1 root root  10, 196 Jul 15 16:25 /dev/vfio/vfio

# ps aux|grep vfio
root      4355  0.0  0.0      0     0 ?        S<   16:25   0:00 [vfio-irqfd-clea]
test     10327 57.8 13.9 1532848 1121300 ?     SLl  21:41   0:22 /usr/libexec/qemu-kvm -name r7 -S -machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 37367d9f-dd35-4425-952c-70024253a082 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/r7.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/var/lib/libvirt/images/rhel7.img,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice port=5902,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1 -device vfio-pci,host=03:10.0,id=hostdev0,bus=pci.0,addr=0x9 -device vfio-pci,host=03:10.2,id=hostdev1,bus=pci.0,addr=0x3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
Comment 5 Shanzhi Yu 2015-07-16 07:10:31 UTC 
Change this bug with verify status after do some testing with latest 
libvirtlibvirt-1.2.17-2.el7.x86_64

Steps as in comment 3 and comment 4.
Comment 6 Martin Polednik 2015-08-05 11:33:58 UTC 
It seems that the is similar 1248105 - RHEV by default sets dynamic_ownership=0, which caused the endpoint not to be accessible by qemu (and we explicitly told libvirt not to do it for us). Works with dynamic_ownership=1.
Comment 8 errata-xmlrpc 2015-11-19 06:17:34 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2202.html
Note You need to log in before you can comment on or make changes to this bug.