Bug 1628892

QEMU patches to add 'rendernode' to cmdline are in place as of:
commit 91e61947eb2be21b00091d34f5692f89cef41376
Author:     Erik Skultety <eskultet>
AuthorDate: Fri Nov 16 11:14:43 2018 +0100
Commit:     Gerd Hoffmann <kraxel>
CommitDate: Fri Nov 16 11:44:22 2018 +0100

    ui: Allow specifying 'rendernode' display option for egl-headless
    
    As libvirt can't predict which rendernode QEMU would pick, it
    won't adjust the permissions on the device, hence QEMU getting
    "Permission denied" when opening the DRI device. Therefore, enable
    'rendernode' option for egl-headless display type.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1648236
    
    Signed-off-by: Erik Skultety <eskultet>
    Message-id: 27f4617f19aa1072114f10f1aa9dd199735ef982.1542362949.git.eskultet
    Signed-off-by: Gerd Hoffmann <kraxel>

However, there doesn't seem to be possible to introspect the corresponding data structure within QAPI schema, so any libvirt patches to fix this will depend on resolving that issue first.

Upstream fix:
commit 3163de7d0e7ae92b9f3e06479c8cd46e43ac8058
Refs: v4.10.0-17-g3163de7d0e
Author:     Erik Skultety <eskultet>
AuthorDate: Thu Nov 15 11:38:00 2018 +0100
Commit:     Erik Skultety <eskultet>
CommitDate: Mon Dec 3 14:58:31 2018 +0100

    qemu: command: gfx: egl-headless: Add 'rendernode' option to the cmdline

    Depending on whether QEMU actually supports the option, we can put the
    'rendernode' on the '-display egl-headless' cmdline.

    https://bugzilla.redhat.com/show_bug.cgi?id=1628892

    Signed-off-by: Erik Skultety <eskultet>
    Reviewed-by: Ján Tomko <jtomko>

The upstream fix above introduced another bug where libvirt is trying to pick a default DRI renderer for SPICE even if the XML doesn't have OpenGL turned on using <gl enable='yes'/>. This is a problem on a headless server without DRI because it fails to start a domain which previously started just fine.

The following scenario needs to be verified:
1. prepare a VM with the following in the XML:
  <graphics type='spice'>
    <listen type='none'/>
  </graphics>
2. rename all /dev/dri/renderDX devices (skip if your host doesn't have DRI)
3. start the VM using virsh

Expected result: the domain starts successfully

The issue mentioned in comment 4 is fixed upstream by:

commit 1215195fd882efac47c07c16bfff0ad9a33c45a3
Refs: v4.10.0-43-g1215195fd8
Author:     Erik Skultety <eskultet>
AuthorDate: Thu Dec 6 16:12:14 2018 +0100
Commit:     Erik Skultety <eskultet>
CommitDate: Tue Dec 11 15:15:17 2018 +0100

    domain: conf: graphics: Fix picking DRI renderer automatically for SPICE

    Commit 255e0732 introduced a few graphics-related helpers. The problem
    is that virDomainGraphicsNeedsAutoRenderNode returns true if it gets
    NULL as a response from virDomainGraphicsNeedsAutoRenderNode. That's
    okay for egl-headless because that one always needs a DRM render node,
    the same is not true for SPICE though, and unless the XML specifies
    <gl enable='yes'> for SPICE, there's no need for any renderer.

    Signed-off-by: Erik Skultety <eskultet>
    Reviewed-by: Ján Tomko <jtomko>

Hi, Erik,

It needs to add DRI renderer device in the "cgroup_device_acl" in qemu.conf now. Does libvirt need to add  cgroup acl for DRI renderer automatically?

(In reply to yafu from comment #6)
> Hi, Erik,
> 
> It needs to add DRI renderer device in the "cgroup_device_acl" in qemu.conf
> now. Does libvirt need to add  cgroup acl for DRI renderer automatically?

I'm not sure I understand the question, libvirt already adds the DRI render node to the cgroup automatically. What issues are you experiencing?

(In reply to Erik Skultety from comment #7)
> (In reply to yafu from comment #6)
> > Hi, Erik,
> > 
> > It needs to add DRI renderer device in the "cgroup_device_acl" in qemu.conf
> > now. Does libvirt need to add  cgroup acl for DRI renderer automatically?
> 
> I'm not sure I understand the question, libvirt already adds the DRI render
> node to the cgroup automatically. What issues are you experiencing?

For now, we need to add DRI renderer device in the 'cgroup device acl' in qemu.conf as follows:
1.Add '/dev/dri/renderD128' in cgroup_device_acl in qemu.conf and restart libvirtd service:
#cat /etc/libvirt/qemu.conf
cgroup_device_acl = [
    "/dev/null", "/dev/full", "/dev/zero",
    "/dev/random", "/dev/urandom",
    "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
    "/dev/rtc","/dev/hpet", "/dev/sev",
    "/dev/dri/renderD128"
]

#systemctl restart libvirtd

otherwise, the guest with egl-headless display will fail to start since permission denied.

(In reply to yafu from comment #8)
> (In reply to Erik Skultety from comment #7)
> > (In reply to yafu from comment #6)
> > > Hi, Erik,
> > > 
> > > It needs to add DRI renderer device in the "cgroup_device_acl" in qemu.conf
> > > now. Does libvirt need to add  cgroup acl for DRI renderer automatically?
> > 
> > I'm not sure I understand the question, libvirt already adds the DRI render
> > node to the cgroup automatically. What issues are you experiencing?
> 
> For now, we need to add DRI renderer device in the 'cgroup device acl' in
> qemu.conf as follows:
> 1.Add '/dev/dri/renderD128' in cgroup_device_acl in qemu.conf and restart
> libvirtd service:
> #cat /etc/libvirt/qemu.conf
> cgroup_device_acl = [
>     "/dev/null", "/dev/full", "/dev/zero",
>     "/dev/random", "/dev/urandom",
>     "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
>     "/dev/rtc","/dev/hpet", "/dev/sev",
>     "/dev/dri/renderD128"
> ]
> 
> #systemctl restart libvirtd
> 
> otherwise, the guest with egl-headless display will fail to start since
> permission denied.

You need libvirt 5.0.0 and qemu 3.1.0

Reproduced with:
libvirt-4.4.0-9.el7.x86_64
qemu-kvm-rhev-2.12.0-14.el7.x86_64
selinux-policy-3.13.1-224.el7.noarch
kernel-3.10.0-940.el7.x86_64

Verified with:
libvirt-4.5.0-15.el7.x86_64
qemu-kvm-rhev-2.12.0-26.el7.x86_64
selinux-policy-3.13.1-245.el7.noarch
kernel-3.10.0-1034.el7.x86_64

Test steps:
Scenario 1: Start guest with egl-headless display
1.Create mdev device;

2.Define a guest with egl-headless display:
<graphics type='egl-headless'>
      <gl rendernode='/dev/dri/renderD128'/>
    </graphics>
    <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='on'>
      <source>
        <address uuid='d5b66366-625c-4632-ab15-6ef3a54666a8'/>
      </source>
    </hostdev>

4.Start the guest:
#virsh start q35
Domain q35 started

5.Check the qemu cmd line:
#ps aux | grep -i "egl-headless"
...-display egl-headless,rendernode=/dev/dri/renderD128 ... -device vfio-pci,id=hostdev0,sysfsdev=/sys/bus/mdev/devices/cba683dd-c572-475d-bad4-62c499da6360,display=on,bus=pcie.0,addr=0x3 -
...

Scenario 2: Test scenario in https://bugzilla.redhat.com/show_bug.cgi?id=1628892#c4:
1. Prepare a VM with the following in the XML:
  <graphics type='spice'>
    <listen type='none'/>
  </graphics>

2. Rename all /dev/dri/renderDX devices (skip if your host doesn't have DRI)

3. Start the guest:
#virsh start q35
Domain q35 started

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:2294