Bug 1937707

This works as root with access_drivers = [ "polkit" ]:

# busctl call org.libvirt /org/libvirt/QEMU org.libvirt.Connect ListDomains u 0

As user that works with neither the default access_drivers nor with polkit. However, cockpit-machines usually connects to qemu:///system as root (superuser: try), so calling it as root should reflect this more.

So this needs more in-depth debugging.

Copy&pasteable setup in a fresh VM, with a tiny pre-defined VM:

  sed -i '/access_drivers/ s/^#//' /etc/libvirt/libvirtd.conf; systemctl restart libvirtd
  virt-install --memory 50 --pxe --virt-type qemu --os-variant alpinelinux3.8 --disk none --wait 0 --name test1

I tested this with current cockpit-machines from upstream master (mostly 243.1) on RHEL 8.4, 8.5, RHEL 9.0, and Fedora 34. In all these cases the Machines page lists the existing VMs, and I can create a new VM (Alpine with PXE boot). Deleting a VM fails with "'QEMU' denied access" on all three OSes.

I also see a lot of these in the journal, but they don't break the UI:

  libvirtd[1909]: access denied: 'QEMU' denied access
  libvirtd[1909]: authentication failed: access denied by policy

Then I downgraded to the respective distro versions: 238.2-1.el8 (RHEL 8.4), 242-1.el8 (RHEL 8.5), 243.1-2.el9, and 243-1.fc34. The latter two are basically the same as current upstream master, the RHEL 8.4/5 versions are older (still from cockpit.git), but I see exactly the same behaviour there (not quite surprisingly, as API wise things didn't change since 242).

Doing VM deletion via the API directly reproduces the issue:

# virsh list --all
 Id   Name    State
-----------------------
 1    test1   running
 2    test2   running

# busctl call org.libvirt /org/libvirt/QEMU org.libvirt.Connect ListDomains u 0
ao 2 "/org/libvirt/QEMU/domain/_b5a8b543_ed63_453c_8b58_8318eb63f0d0" "/org/libvirt/QEMU/domain/_69007a04_04ba_4970_80ac_5dc30f6d6250"

# busctl call org.libvirt /org/libvirt/QEMU/domain/_69007a04_04ba_4970_80ac_5dc30f6d6250 org.libvirt.Domain Destroy u 0
Call failed: access denied: 'QEMU' denied access

@Juan, @Yunming, so I cannot reproduce the rejection on creating the VM, but supposedly it's the same root cause.

Reassigning to libvirtd as there is now a CLI reproducer entirely outside of cockpit. This possibly belongs to libvirt-dbus, please reassign if appropriate.

(In reply to Martin Pitt from comment #5)
> Copy&pasteable setup in a fresh VM, with a tiny pre-defined VM:
> 
>   sed -i '/access_drivers/ s/^#//' /etc/libvirt/libvirtd.conf; systemctl
> restart libvirtd
>   virt-install --memory 50 --pxe --virt-type qemu --os-variant
> alpinelinux3.8 --disk none --wait 0 --name test1
> 
> I tested this with current cockpit-machines from upstream master (mostly
> 243.1) on RHEL 8.4, 8.5, RHEL 9.0, and Fedora 34. In all these cases the
> Machines page lists the existing VMs, and I can create a new VM (Alpine with
> PXE boot). Deleting a VM fails with "'QEMU' denied access" on all three OSes.
> 
> I also see a lot of these in the journal, but they don't break the UI:
> 
>   libvirtd[1909]: access denied: 'QEMU' denied access
>   libvirtd[1909]: authentication failed: access denied by policy
> 
> Then I downgraded to the respective distro versions: 238.2-1.el8 (RHEL 8.4),
> 242-1.el8 (RHEL 8.5), 243.1-2.el9, and 243-1.fc34. The latter two are
> basically the same as current upstream master, the RHEL 8.4/5 versions are
> older (still from cockpit.git), but I see exactly the same behaviour there
> (not quite surprisingly, as API wise things didn't change since 242).
> 
> Doing VM deletion via the API directly reproduces the issue:
> 
> # virsh list --all
>  Id   Name    State
> -----------------------
>  1    test1   running
>  2    test2   running
> 
> # busctl call org.libvirt /org/libvirt/QEMU org.libvirt.Connect ListDomains
> u 0
> ao 2 "/org/libvirt/QEMU/domain/_b5a8b543_ed63_453c_8b58_8318eb63f0d0"
> "/org/libvirt/QEMU/domain/_69007a04_04ba_4970_80ac_5dc30f6d6250"
> 
> # busctl call org.libvirt
> /org/libvirt/QEMU/domain/_69007a04_04ba_4970_80ac_5dc30f6d6250
> org.libvirt.Domain Destroy u 0
> Call failed: access denied: 'QEMU' denied access
> 
> @Juan, @Yunming, so I cannot reproduce the rejection on creating the VM, but
> supposedly it's the same root cause.
> 
> Reassigning to libvirtd as there is now a CLI reproducer entirely outside of
> cockpit. This possibly belongs to libvirt-dbus, please reassign if
> appropriate.


Hi Martin,

I can hit the error with your steps, and could see the error log in libvirtd.log:
2021-05-25 02:22:53.962+0000: 39451: debug : virAccessDriverPolkitCheck:140 : Check action 'org.libvirt.api.domain.stop' for process '39445' time 401995 uid 977
2021-05-25 02:22:53.962+0000: 39451: info : virPolkitCheckAuth:84 : Checking PID 39445 running as 977
2021-05-25 02:22:53.982+0000: 39451: debug : virPolkitCheckAuth:131 : is auth 0  is challenge 0
2021-05-25 02:22:53.982+0000: 39451: error : virPolkitCheckAuth:145 : authentication failed: access denied by policy
2021-05-25 02:22:53.982+0000: 39451: info : virObjectUnref:381 : OBJECT_UNREF: obj=0x55bdc3509df0
2021-05-25 02:22:53.982+0000: 39451: error : virDomainDestroyFlagsEnsureACL:2987 : access denied: 'QEMU' denied access

And from the error log we can see cockpit delete(destroy) guest with user libvirtdbus, which uid is 977. For the non-root user, we need to add polkit rule for api action. So i added it as follows:
#cat /etc/polkit-1/rules.d/100-libvirt-acl.rules
polkit.addRule(function(action, subject) {
if (action.id == "org.libvirt.api.domain.stop" &&
subject.user == "libvirtdbus") {
if (action.lookup("connect_driver") == 'QEMU') {
return polkit.Result.YES;
} else {
return polkit.Result.NO;
}
}
});

Then destroy guest again, the guest destroy successfully:
# busctl call org.libvirt /org/libvirt/QEMU/domain/_c506dacd_1e60_408f_b983_f09b425ec058 org.libvirt.Domain Destroy u 0
# echo $?
0

So i think it's not a bug, it works as expected.

So this happens because from libvirtd's point of view (the polkit target API) it's libvirt-dbus which is doing the calls, and that is always the libvirtdbus system user which the polkit rules apply to. This is not really useful -- the actual caller that needs to be checked is the one invoking libvirt-dbus' method (which then "forwards" it to libvirtd).

libvirt-dbus ships /usr/share/polkit-1/rules.d/libvirt-dbus.rules which allows anyone calling libvirt-dbus to execute "org.libvirt.unix.manage" operations, but apparently that only allows listing and creation, but does not include "org.libvirt.api.domain.stop"? libvirt-dbus itself does not do any polkit checks against its callers, it just seems to allow calls from root.

So I guess with the current architecture of libvirt-dbus being a separate process/service to libvirtd this is unfixable. To do this cleanly, libvirtd itself would need to expose the D-Bus interface. Unless there is a way to "forward" a polkit API check -- that may be possible with a custom polkit agent?


So Juan, the summary is: libvirt polkit mode and cockpit-machines (via libvirt-dbus) are incompatible. If you want to hand out access to non-root/wheel users, there is currently no way to do that with cockpit-machines.

A bandaid fix for libvirt-dbus is then to actually do exactly what yafu proposed above: Add the other operations to /usr/share/polkit-1/rules.d/libvirt-dbus.rules , not just "org.libvirt.unix.manage". As libvirt-dbus access is restricted to root, there is no point in restricting what libvirt-dbus can do. With that, cockpit-machines/libvirt-dbus would at least work for actual root/wheel users in polkit mode. That doesn't require any architectural changes.

Thanks for the explanation, and based on that, would it make sense to expand the supplied polkit policy in /usr/share/polkit-1/rules.d/libvirt-dbus.rules to allow all actions?

Something like this:

polkit.addRule(function(action, subject) {
    if ((action.id == "org.libvirt.unix.manage" ||
         action.id.indexOf("org.libvirt.api.connect.") == 0 ||
         action.id.indexOf("org.libvirt.api.domain.") == 0 ||
         action.id.indexOf("org.libvirt.api.interface.") == 0 ||
         action.id.indexOf("org.libvirt.api.network.") == 0 ||
         action.id.indexOf("org.libvirt.api.node-device.") == 0 ||
         action.id.indexOf("org.libvirt.api.nwfilter.") == 0 ||
         action.id.indexOf("org.libvirt.api.secret.") == 0 ||
         action.id.indexOf("org.libvirt.api.storage-pool.") == 0 ||
         action.id.indexOf("org.libvirt.api.storage-vol.") == 0 ) &&
        subject.user == "libvirtdbus") {
        return polkit.Result.YES;
    }
});

Juan: In principle yes; I'd even simplify it too

    if (subject.user == "libvirtdbus" &&
        (action.id == "org.libvirt.unix.manage" || action.id.indexOf("org.libvirt.api.") == 0))

or even

    if (subject.user == "libvirtdbus" && action.id.indexOf("org.libvirt.") == 0)

as conceptually libvirt-dbus should be able to call *all* libvirt APIs?

Martin, I've tested your last example allowing all "org.libvirt." and works like a charm. Now root and the members of the libvirt group can manage the VMs from Cockpit.

Can we update the supplied policy?

As Martin mentioned in one of the comments using libvirt polkit policy together with libvirt-dbus isn't compatible. With the current state without any changes for system connection it works like this:

   polkit disabled:

      - root user: both virsh and libvirt-dbus works with all APIs

      - non-root user: using virsh requires authentication (if authenticated all APIs) and libvirt-dbus access denied

      - non-root user with libvirt group: both virsh and libvirt-dbus works with all APIs

   polkit enabled:

      - root user: virsh works for all APIs but libvirt-dbus only read-only APIs

      - non-root user: using virsh requires authentication (if authenticated APIs allowed for the authenticated user) and libvirt-dbus access denied

      - non-root user with libvirt group: both virsh and libvirt-dbus only read-only APIs

Now if we consider the situation with polkit enabled:

   - any change to polkit rules for specific user (except for libvirtdbus) will be available only for virsh usage

   - any change to polkit rules for libvirtdbus user will affect both root user and any user with libvirt group added


With the change proposed by Comment 9 or Comment 10 we would be able to use all APIs with root user but also with all users that are part of libvirt group which could break expectation for system administrators enabling polkit for libvirt. Users with libvirt group would not be able to use virsh to make changes to VMs but would be able to use libvirt-dbus to make changes to VMs which ti me sounds like security hole.


I don't think we should change the default policy rules shipped by libvirt-dbus, instead we need to properly document all limitations and offer some examples how to control access to VMs with libvirt-dbus and cockpit-machines.

I am changing the component to 'Documentation' per comment 14. Thanks.

Overall, looks good to me.