Bug 1044561 - guest fails to start with 'permission denied' accessing auto qemu-ga socket in /var/lib
Summary: guest fails to start with 'permission denied' accessing auto qemu-ga socket i...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: libvirt
Version: 21
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Libvirt Maintainers
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-12-18 14:53 UTC by Josh Boyer
Modified: 2020-09-23 07:55 UTC (History)
18 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-06-05 23:01:07 UTC
Type: Bug


Attachments (Terms of Use)
selinux denial (3.00 KB, text/plain)
2013-12-18 14:53 UTC, Josh Boyer
no flags Details
/var/log/libvirt/qemu/fedora20.log (18.38 KB, text/x-log)
2013-12-18 14:56 UTC, Josh Boyer
no flags Details
/etc/libvirt/qemu.conf (13.71 KB, text/plain)
2013-12-18 14:57 UTC, Josh Boyer
no flags Details
/etc/libvirt/libvirtd.conf (13.20 KB, text/plain)
2013-12-18 14:58 UTC, Josh Boyer
no flags Details
/etc/libvirt/libvirt.conf (518 bytes, text/plain)
2013-12-18 14:58 UTC, Josh Boyer
no flags Details
/etc/libvirt/qemu-lockd.conf (2.12 KB, text/plain)
2013-12-18 14:59 UTC, Josh Boyer
no flags Details
/etc/libvirt/virtlockd.conf (2.16 KB, text/plain)
2013-12-18 14:59 UTC, Josh Boyer
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1146886 0 unspecified CLOSED [DAC] permission denied with auto generated channel socket path when set user as root in qemu.conf 2021-02-22 00:41:40 UTC

Internal Links: 1146886

Description Josh Boyer 2013-12-18 14:53:55 UTC
Created attachment 838380 [details]
selinux denial

Description of problem:

On an F20 system I'm trying to use virt-manager to create a new F20 guest.  When I click "Finish", I'm presented with the following error:

Unable to complete install: 'internal error: process exited while 
             connecting to monitor: qemu-system-x86_64: -chardev 
socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/fedora20.org.qemu.guest_agent.0,server,nowait: Failed to bind socket: Permission denied
chardev: opening backend "socket" failed

and an SELinux AVC (attached).  When I set SELinux to permissive mode, things work fine.

Version-Release number of selected component (if applicable):

libvirt-1.1.3.2-1.fc20.x86_64
selinux-policy-3.12.1-106.fc20.noarch
qemu-1.6.1-2.fc20.x86_64

How reproducible:

Always thus far.

Steps to Reproduce:
1. Open virt-manager
2. Create a new network install guest.  Put the guest's disk in ~/images/
3. Click finish

Actual results:

Errors above

Expected results:

No errors

Additional info:

I discussed this briefly with Cole and Dan:

09:29 < jwb> dwalsh, crobinso: is it normal for F20 selinux to be denying 
             qemu-system-x86_64 things?
09:29 < jwb> dwalsh, crobinso: http://fpaste.org/62884/13873769/
09:30 < jwb> all i'm trying to do is use virt-manager to do a network install 
             of an f20 guest
09:31 < crobinso> jwb: no, there shouldn't be errors in regular usage. I don't 
                  know particularly what that issue is though, so bugzilla away 
                  :)
09:32 < dwalsh> jwb This looks like a root process is not allowed to read a 
                file based on permissions.
09:33 < dwalsh> If you add +r to your image, does the problem go away.
09:33 < jwb> dwalsh, crobinso: virt-manager is complaining about not being able 
             to connect to a socket or something
09:33 < dwalsh> dac_overide/dac_read_search means the priv process can read a 
                object based on ownership/permission.
09:34 < dwalsh> So a file labeled dwalsh:dwalsh with 640, would not be allowed 
                to a root process.
09:34 < jwb> Unable to complete install: 'internal error: process exited while 
             connecting to monitor: qemu-system-x86_64: -chardev 
socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/fedora20.org.qemu.guest_agent.0,server,nowait: Failed to bind socket: Permission denied
09:34 < jwb> chardev: opening backend "socket" failed
09:36 < dwalsh> jwb, Work in permissive mode?
09:36 < jwb> let me try
09:37 < jwb> dwalsh, yep
09:38 < crobinso> libvirt should be auto creating that socket and setting the 
                  user/group to the same as the what qemu is run as, so not 
                  sure what's going on there. libvirt bug maybe, but it doesn't 
                  reproduce here.
09:38 < dwalsh> And AVC is about DAC?
09:39 < jwb> dwalsh, apparently.  i click finish, i get that error above, i get 
             setroubleshoot popping up the DAC thing
09:39 < jwb> in permissive mode it works
09:41 < dwalsh> So 
=/var/lib/libvirt/qemu/channel/target/fedora20.org.qemu.guest_agent.0,server,nowait is not being set with the correct permissions.
09:42 < jwb> dwalsh, virt-manager bug then?
09:43 < dwalsh> crobinso, Would know better then me. virt-manager or libvirt.  
                I would guess.  We don't want to give qemu programs this access.
09:43 < ciupicri> tyll: I think I've finished the rename from python-trml2pdf 
                  to python-trml2pdf12, but I'm not sure
09:44 < crobinso> jwb: start with libvirt
09:44 < crobinso> jwb: also, show ls -lZ /var/lib/libvirt/qemu/channel/target
09:45 < crobinso> jwb: /var/log/libvirt/qemu/$vmname.log   and 
                  /etc/libvirt/qemu.conf
09:45 < jwb> ok

[root@vader qemu]# ls -lZ /var/lib/libvirt/qemu/channel/target
srwxr-xr-x. root root system_u:object_r:qemu_var_run_t:s0 fedora20.org.qemu.guest_agent.0
[root@vader qemu]#

Comment 1 Josh Boyer 2013-12-18 14:56:20 UTC
Created attachment 838381 [details]
/var/log/libvirt/qemu/fedora20.log

Comment 2 Josh Boyer 2013-12-18 14:57:40 UTC
Created attachment 838383 [details]
/etc/libvirt/qemu.conf

Comment 3 Josh Boyer 2013-12-18 14:58:09 UTC
Created attachment 838384 [details]
/etc/libvirt/libvirtd.conf

Comment 4 Josh Boyer 2013-12-18 14:58:54 UTC
Created attachment 838386 [details]
/etc/libvirt/libvirt.conf

Comment 5 Josh Boyer 2013-12-18 14:59:25 UTC
Created attachment 838387 [details]
/etc/libvirt/qemu-lockd.conf

Comment 6 Josh Boyer 2013-12-18 14:59:55 UTC
Created attachment 838389 [details]
/etc/libvirt/virtlockd.conf

Comment 7 Cole Robinson 2014-01-17 15:17:31 UTC
I hit this too when trying doing some upstream libvirt dev, trying to start a pre-existing VM, and selinux permissive. I didn't dig into the code, but here's what I think is happening:

Libvirt auto allocates a socket for us (virt-manager/virt-install) in /var/lib/libvirt/qemu/channel/target. But when the guest shuts down, the socket isn't removed. It's left with whatever user/group libvirt ran as: if using fedora packages, that's qemu:qemu, if using libvirt.git, that's likely root:root.

If you switch back and forth between the two versions, trying to start a previously defined VM with the auto socket bit, or create a new VM with a reused name, it will try to reuse the existing socket path. But if the user/group don't match, EVEN if running the emulator as root (prob due to dropping of capabilities which effectively removes root privilege as well), the emulator can't access the existing socket, startup fails.

In this case Josh, maybe you tried to create a new VM that reused the name of a pre-existing VM, the left over socket had old selinux labeling.

Workaround is to: shutdown all VMs, sudo rm -rf /var/lib/libvirt/qemu/channel/target/*

Comment 8 John Ferlan 2014-02-08 13:39:45 UTC
This did not work for me - seems to be fairly easy to reproduce.  I reserved a beaker system with f20 installed, installed qemu-kvm, libvirt, etc. and was able to create a vm w/ virt-install. This was then used for a virt-test run...
I usually clone autotest and virt-test, then use the './run -t libvirt --bootstrap' and being running tests (you could probably at least see a guest create using just --install --remove and not selecting any tests).

I then cloned, built, and installed both libvirt.git and libvirt-python.git. The latter is required to even run virt-install. After doing that, running virt-test failed to create the guest with the error message in the log:


* Command:
    /usr/bin/virt-install --hvm --accelerate --name 'virt-tests-vm1'
    --ram=1024 --vcpu=2 --import --vnc --os-variant fedora19 --serial pty
    --disk path=/home/virt-
    test/shared/data/images/jeos-19-64.qcow2,bus=virtio,size=10,format=qcow2
    --network=bridge=virbr0,model=virtio,mac=52:54:00:94:95:96 --noautoconsole
Exit status: 1
Duration: 1.12764596939

stdout:

Starting install...
stderr:
ERROR    internal error: process exited while connecting to monitor: qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/virt-tests-vm1.org.qemu.guest_agent.0,server,nowait: Failed to bind socket: Permission denied
chardev: opening backend "socket" failed


I tried the "rm -rf /var/lib/libvirt/qemu/channel/target/*", but that did not help.

Comment 9 Cole Robinson 2014-02-08 14:38:41 UTC
Why is this comment private?

(In reply to John Ferlan from comment #8)
> This did not work for me - seems to be fairly easy to reproduce.  I reserved
> a beaker system with f20 installed, installed qemu-kvm, libvirt, etc. and
> was able to create a vm w/ virt-install. This was then used for a virt-test
> run...
> I usually clone autotest and virt-test, then use the './run -t libvirt
> --bootstrap' and being running tests (you could probably at least see a
> guest create using just --install --remove and not selecting any tests).
> 
> I then cloned, built, and installed both libvirt.git and libvirt-python.git.
> The latter is required to even run virt-install. After doing that, running
> virt-test failed to create the guest with the error message in the log:
> 
> 
> * Command:
>     /usr/bin/virt-install --hvm --accelerate --name 'virt-tests-vm1'
>     --ram=1024 --vcpu=2 --import --vnc --os-variant fedora19 --serial pty
>     --disk path=/home/virt-
>     test/shared/data/images/jeos-19-64.qcow2,bus=virtio,size=10,format=qcow2
>     --network=bridge=virbr0,model=virtio,mac=52:54:00:94:95:96
> --noautoconsole
> Exit status: 1
> Duration: 1.12764596939
> 
> stdout:
> 
> Starting install...
> stderr:
> ERROR    internal error: process exited while connecting to monitor:
> qemu-system-x86_64: -chardev
> socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/virt-tests-
> vm1.org.qemu.guest_agent.0,server,nowait: Failed to bind socket: Permission
> denied
> chardev: opening backend "socket" failed
> 
> 
> I tried the "rm -rf /var/lib/libvirt/qemu/channel/target/*", but that did
> not help.

Hmm. Does virt-test run virt-install as root or as regular user? Maybe libvirt is trying to stick the socket in /var/lib when using qemu:///session?

Comment 10 John Ferlan 2014-02-08 15:58:21 UTC
I'm running virt-test as root on the beaker system


As pointed out before in email - if I modify the virt-install slightly, then things work, but IIRC that has to do with whether libvirt automatically creates the channel or not...

diff --git a/virt-install b/virt-install
index b003e95..b0bf392 100755
--- a/virt-install
+++ b/virt-install
@@ -493,12 +493,6 @@ def build_guest_instance(conn, options):
     convert_old_disks(options)
     cli.convert_old_features(options)
 
-    # Install options
-    guest.installer.extraargs = options.extra
-    guest.installer.initrd_injections = options.initrd_injections
-    cli.set_os_variant(guest, options.distro_type, options.distro_variant)
-    guest.os.init = options.init
-
     # Guest configuration
     cli.get_uuid(guest, options.uuid)
     cli.get_vcpus(guest, options.vcpus, options.check_cpu)
@@ -537,6 +531,12 @@ def build_guest_instance(conn, options):
     guest.add_default_usb_controller()
     guest.add_default_channels()
 
+    # Install options
+    guest.installer.extraargs = options.extra
+    guest.installer.initrd_injections = options.initrd_injections
+    cli.set_os_variant(guest, options.distro_type, options.distro_variant)
+    guest.os.init = options.init
+
     # Do this after setting up all optional parameters, so we report error
     # about those first.
     need_storage, need_install = validate_required_options(options, guest)

Comment 11 Cole Robinson 2014-02-09 13:46:12 UTC
Right, that change virt-install will prevent the qemu-ga channel from being added to the guest (in a roundabout way). Still, the issue has something to do with libvirt's handling of the autogenerated socket paths. You could also manually work around it with --channel none, but that's probably not easy to inject into virt-test

Comment 12 John Ferlan 2014-03-10 13:23:32 UTC
I found a different way to "work around" this without editing the virt-install on f20 - change the protections of the directories too not just the files:

          chown -R root:root /var/lib/libvirt/qemu/channel

reading back through the comments - I forget why I made things private when I did - perhaps just habit... 

I've undone the settings...

Comment 13 Timothée Ravier 2014-09-02 13:17:52 UTC
https://bbs.archlinux.org/viewtopic.php?id=177483 appears to be related.

Should virt-install chown the /var/lib/libvirt/qemu/channel and /var/lib/libvirt/qemu/dump directories according to the user/group settings in /etc/libvirt/qemu.conf before launching qemu?

Comment 14 Panormitis Petrou 2014-09-28 14:14:44 UTC
I can confirm this issue too.
I executed "chown -R root:root /var/lib/libvirt/qemu/channel" to resolve it.

Comment 15 Cole Robinson 2015-04-24 00:44:12 UTC
Should be fixed by https://www.redhat.com/archives/libvir-list/2015-April/msg01212.html

Comment 16 Martin Kletzander 2015-04-24 07:06:39 UTC
(In reply to Cole Robinson from comment #15)
This will fix it only for the default user.  As Dan said, we also need to have a special folder for each user that's going to have a domain running under it.

Comment 17 Cole Robinson 2015-04-24 13:51:58 UTC
(In reply to Martin Kletzander from comment #16)
> (In reply to Cole Robinson from comment #15)
> This will fix it only for the default user.  As Dan said, we also need to
> have a special folder for each user that's going to have a domain running
> under it.

How do we end up with different VMs running as different users? Do you mean

- start a VM with qemu.conf qemu:qemu
- change qemu.conf to root:root
- restart libvirtd
- start a new VM

I think the patches cover that case. The first VM already has an fd for the socket, changing the directory permissions when libvirtd restarts shouldn't affect the running VM.

Or maybe there's some way to configure user/group per VM that I'm not aware of...

Comment 18 Cole Robinson 2015-04-24 13:54:04 UTC
Hmm I see in bug 1146886 that there _is_ a way to specify a manual user:group per VM... that's pretty crazy :)

However I still consider this reported issue fixed since it solves the more likely problem of changing global qemu.conf user:group

Comment 19 Cole Robinson 2015-04-27 23:42:16 UTC
F20 is pretty late in life, so moving this to f21

Comment 20 Fedora Update System 2015-04-28 16:54:42 UTC
libvirt-1.2.9.3-1.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/libvirt-1.2.9.3-1.fc21

Comment 21 Fedora Update System 2015-04-29 13:05:48 UTC
Package libvirt-1.2.9.3-1.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libvirt-1.2.9.3-1.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-7150/libvirt-1.2.9.3-1.fc21
then log in and leave karma (feedback).


Note You need to log in before you can comment on or make changes to this bug.