Bug 1337290

Summary: libvirt: spice gl doesn't work with qemu:///system
Product: [Fedora] Fedora Reporter: Cole Robinson <crobinso>
Component: libvirtAssignee: Michal Privoznik <mprivozn>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 26CC: agedosier, airlied, berrange, cfergeau, clalancette, darkbasic, devurandom, dinechin, dyuan, everplays, fedora, fjin, fziglio, igeorgex, itamar, jfrieben, jtomko, kraxel, laine, lantw44, libvirt-maint, marcandre.lureau, mprivozn, paulkek, pzeppegno, pzhang, sassmann, veillard, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-05-04 17:53:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Cole Robinson 2016-05-18 17:49:18 UTC
spice GL passthrough doesn't currently work with libvirt's qemu:///system, talking to the system libvirtd instance and launching VMs as the unprivileged qemu:qemu user, vs qemu:///session which is running libvirtd and launching VMs are your local user

The issue is qemu can't open /dev/dri/render* since it lacks permissions to do so. For a login session the permissions are:

$ getfacl /dev/dri/renderD128 
getfacl: Removing leading '/' from absolute path names
# file: dev/dri/renderD128
# owner: root
# group: video
user::rw-
user:crobinso:rw-
group::rw-
mask::rw-
other::---

Even after chmod 666 or adding qemu to the 'video' group, the open() still fails but with 'Operation not permitted' instead of 'Permission denied'. selinux is permissive

$ rpm -q qemu-kvm libvirt-daemon mesa-libEGL
qemu-kvm-2.6.0-1.fc24.x86_64
libvirt-daemon-1.3.3.1-1.fc24.x86_64
mesa-libEGL-11.2.1-1.20160501.fc24.x86_64

Comment 1 Cole Robinson 2016-05-18 17:50:50 UTC
CCing some people for suggestions

Comment 2 Dave Airlie 2016-05-18 20:02:30 UTC
I think we need to change cgroup_device_acl to include the /dev/dri/render nodes
in /etc/libvirt/qemu.conf by default.

Comment 3 Ján Tomko 2016-05-19 07:35:27 UTC
Proposed upstream patch allowing "c 226:* rw":
https://www.redhat.com/archives/libvir-list/2016-May/msg01407.html

Comment 4 Ján Tomko 2016-05-19 09:00:36 UTC
Pushed as:
commit 3943bdd60c7ff1de00e73f66d907664b74a88a3f
Author:     Ján Tomko <jtomko>
CommitDate: 2016-05-19 10:52:50 +0200

    qemu_cgroup: allow access to /dev/dri for virtio-vga
    
    QEMU needs access to the /dev/dri/render* device for
    virgl to work.
    
    Allow access to all /dev/dri/* devices for domains with
    <video>
      <model type='virtio' heads='1' primary='yes'>
        <acceleration accel3d='yes'/>
      </model>
    </video>
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1337290

git describe: v1.3.4-350-g3943bdd

Comment 5 Dave Airlie 2016-05-19 10:05:00 UTC
I think we should possibly restrict it to just /dev/dri/render* for system level qemus. Accessing card nodes has a lot more rules on what you can do.

Comment 6 Ján Tomko 2016-05-19 11:32:48 UTC
Version 2:
* only allows /dev/dri/render*
* checks for <graphics type="spice"> has <gl enable="yes"/>
  instead of <acceleration accel3d='yes'/> in <model type='virtio'>

https://www.redhat.com/archives/libvir-list/2016-May/msg01435.html

Comment 7 Joachim Frieben 2016-08-06 04:09:42 UTC
(In reply to Ján Tomko from comment #6)
As of current Fedora 24, the addition of directive <gl enable='on'/> in section <graphics ..> is ignored and even removed upon start-up of the corresponding virtual machine. Current version is libvirt-1.3.3.2-1.fc24. The output of command 'dmesg' reports:

[    6.086809] [drm] Initialized drm 1.1.0 20060810
[    6.386704] [drm] pci: virtio-vga detected
[    6.396710] [drm] virgl 3d acceleration not available
[    6.397956] [drm] virtio vbuffers: 80 bufs, 192B each, 15kB total.
[    6.424352] [drm] number of scanouts: 1
[    6.424363] [drm] number of cap sets: 0
[    6.574810] virtio_gpu virtio0: fb0: virtiodrmfb frame buffer device
[    6.597074] [drm] Initialized virtio_gpu 0.0.1 0 on minor 0
 
On the contrary, under current Fedora 25, virgl 3d is then enabled correctly as long as SELinux is operating in permissive mode (bug 1364075). Here, current version is libvirt-2.1.0-1.fc25.

Comment 8 Niccolò Belli 2016-08-31 21:50:40 UTC
With the patch I get:

SPICE GL support is local-only for now and incompatible with -spice port/tls-port

Comment 9 Niccolò Belli 2016-08-31 22:10:28 UTC
The vm starts with a socket instead, but I had to manually edit the config because virt-manager didn't allow me to to either specify a socket nor enabling gl:

<graphics type='spice' autoport='yes' keymap='en-us' passwd='password'>
  <listen type='socket' socket='/tmp/spice.socket'/>
  <gl enable="yes"/>
</graphics>

I'm using libvirt-git with this patch: https://www.redhat.com/archives/libvir-list/2016-May/msg01437.html

Unfortunately virgl still don't work:

[    2.459310] [drm] Initialized drm 1.1.0 20060810
[    2.560986] [drm] pci: virtio-vga detected
[    2.562226] [drm] virgl 3d acceleration not available
[    2.562904] [drm] virtio vbuffers: 80 bufs, 192B each, 15kB total.
[    2.566460] [drm] number of scanouts: 1
[    2.566469] [drm] number of cap sets: 0
[    2.572279] virtio_gpu virtio0: fb0: virtiodrmfb frame buffer device
[    2.582905] [drm] Initialized virtio_gpu 0.0.1 0 on minor 0

Guest is Fedora 25.

Comment 10 Niccolò Belli 2016-08-31 22:43:01 UTC
My fault, forgot to reboot after updating libvirt.

Now I get:

internal error: qemu unexpectedly closed the monitor: egl: no drm render node available
qemu-system-x86_64: Failed to initialize EGL render node for SPICE GL

$ ls -l /dev/dri/renderD128
crw-rw----+ 1 root video 226, 128  1 set 00.22 /dev/dri/renderD128

So I added nobody (yeah, Archlinux runs qemu through the nobody user) to the video group and rebooted:

[    2.529792] [drm] virgl 3d acceleration enabled

Comment 11 Dennis Schridde 2016-10-09 20:44:18 UTC
I confirm for Arch Linux that the following makes the VM start with without the abovementioned error "egl: no drm render node available", but instead with VirGL support detected by the kernel:

1. Download libvirt-git PKGBUILD from AUR
2. Add this patch to prepare(): https://www.redhat.com/archives/libvir-list/2016-May/msg01437.html
3. Build and install
4. Restart libvirtd
5. Boot VM

Comment 12 Paul 2017-01-24 20:20:18 UTC
Hello,

I guess this still has not been fixed, even in the 3.0 release of libvirt since I am still getting that error message even with the patch posted in above posts.

I compiled libvirt from master with the patch applied with no luck. Is there anything I can do to solve this?

Comment 13 Paul 2017-01-26 00:26:02 UTC
Since the patch didn't work, I tried modifying cgroup_device_acl in the qemu.conf file by adding "/dev/dri/renderD128" to the list and I am no longer getting the old error message which says that no render node is available.

However, I am getting new error messages:

qemu-system-x86_64: egl: eglGetDisplay failed
qemu-system-x86_64: egl: EGL_KHR_surfaceless_context not supported
qemu-system-x86_64: Failed to initialize EGL render node for SPICE GL

Comment 14 Paul 2017-01-30 21:33:42 UTC
Ping?

Comment 15 Behrooz 2017-02-02 10:20:00 UTC
I had the same problem as Paul described: The patch did not solve the issue. However, by manually adding `/dev/dri/renderD128` to the cgroup_device_acl of `/etc/libvirt/qemu.conf` (note that it was commented), I managed to get 3D context:

> virgl 3d acceleration enabled

I am running Fedora 25 for both host and guest.

Comment 16 Cole Robinson 2017-02-09 21:40:02 UTC
There's been another attempt at this on the list from mprivozn:

http://www.redhat.com/archives/libvir-list/2017-February/msg00263.html

Comment 17 Michal Privoznik 2017-02-16 09:28:12 UTC
And yet anther one:

https://www.redhat.com/archives/libvir-list/2017-February/msg00497.html

Comment 18 Michal Privoznik 2017-02-20 10:33:32 UTC
Pushed upstream:

commit 5c74cf1f440b19c05dd9dbd2ab22c217db959fcf
Author:     Michal Privoznik <mprivozn>
AuthorDate: Thu Feb 9 17:53:53 2017 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Mon Feb 20 10:44:22 2017 +0100

    qemu: Allow @rendernode for virgl domains
    
    When enabling virgl, qemu opens /dev/dri/render*. So far, we are
    not allowing that in devices CGroup nor creating the file in
    domain's namespace and thus requiring users to set the paths in
    qemu.conf. This, however, is suboptimal as it allows access to
    ALL qemu processes even those which don't have virgl configured.
    Now that we have a way to specify render node that qemu will use
    we can be more cautious and enable just that.
    
    Signed-off-by: Michal Privoznik <mprivozn>

v3.0.0-236-g5c74cf1f4

Comment 19 Frediano Ziglio 2017-02-20 12:52:39 UTC
(In reply to Paul from comment #13)
> Since the patch didn't work, I tried modifying cgroup_device_acl in the
> qemu.conf file by adding "/dev/dri/renderD128" to the list and I am no
> longer getting the old error message which says that no render node is
> available.
> 
> However, I am getting new error messages:
> 
> qemu-system-x86_64: egl: eglGetDisplay failed
> qemu-system-x86_64: egl: EGL_KHR_surfaceless_context not supported
> qemu-system-x86_64: Failed to initialize EGL render node for SPICE GL

I think following patch should solve this issue:

http://lists.nongnu.org/archive/html/qemu-devel/2017-02/msg04223.html

Comment 20 Paul 2017-02-25 16:11:00 UTC
I am still getting the same error with the new patch (Which uses eglGetPlatformDisplayEXT instead of eglGetDisplay).

Though, I am wondering if this will work on NVIDIA drivers? Since I have noticed the usage of EGL_PLATFORM_GBM_MESA which seems Mesa specific? (This is just an assumption, I am not familiar with any of this.)

Comment 21 Paul 2017-02-25 16:16:55 UTC
Well, I've just realized that NVIDIA drivers don't support GBM.

That is unfortunate.

Comment 22 Niccolò Belli 2017-02-26 22:18:17 UTC
> That is unfortunate.

No it isn't unfortunate, it's just stupid. Allowing them to create their own standards mainlining non-gbm wayland support in Gnome is even more stupid.

Comment 23 Christophe de Dinechin 2017-04-06 09:45:59 UTC
With the current state of Fedora25, I had some success on AMD V7900 after following Paul's advice in cgroup_device_acl. It initializes OK in the guest:

[   22.531176] [drm] Initialized
[   22.938684] [drm] pci: virtio-vga detected at 0000:00:02.0
[   22.939198] [drm] virgl 3d acceleration enabled
[   22.939972] [drm] virtio vbuffers: 288 bufs, 192B each, 54kB total.
[   22.940246] [drm] number of scanouts: 1
[   22.940255] [drm] number of cap sets: 1
[   22.947790] [drm] cap set 0: id 1, max-version 1, max-size 308
[   22.969754] virtio_gpu virtio0: fb0: virtiodrmfb frame buffer device
[   22.975394] [drm] Initialized virtio_gpu 0.0.1 0 on minor 0

But when I run glxgears, I get a black window and the following message:

libGL error: MESA-LOADER: failed to retrieve device information
MESA-LOADER: failed to retrieve device information
MESA-LOADER: failed to retrieve device information
Running synchronized to the vertical refresh.  The framerate should be
approximately the same as the monitor refresh rate.
GL_RENDERER   = Gallium 0.4 on virgl
GL_VERSION    = 3.0 Mesa 13.0.4
GL_VENDOR     = Red Hat
VisualID 440, 0x1b8

Comment 24 Christophe de Dinechin 2017-04-06 09:52:38 UTC
Tried under GNOME / Xorg, in that case, the Terminal itself is not readable. I believe this is bug 1426549.

Comment 25 Cole Robinson 2017-05-04 17:53:43 UTC
I'm not going to backport the libvirt rendernode patches to f25, but they are in f26. If people want to try them on f25, grab latest libvirt and qemu from the fedora-virt-preview repo.

Closing this against F26. If anyone is still hitting issues please file a new bug

Comment 26 pzeppegno 2017-06-09 17:35:22 UTC
I still get "Failed to initialize EGL render node for SPICE GL " running fedora 26 on my fedora 26 host machine (upgraded as of today).

I would love to use 3d support and I thought fedora26 would be ready. How can I help to fix it?

Comment 27 verdre 2017-07-17 17:13:39 UTC
I'm also still getting the error, please reopen the bug.

Comment 28 Cole Robinson 2017-07-17 17:51:46 UTC
If you're using qemu:///system it could be this bug:

https://bugzilla.redhat.com/show_bug.cgi?id=1460804

If that's not it, please open a new bug, this particular report is already quite long