Bug 1044561

Created attachment 838381 [details]
/var/log/libvirt/qemu/fedora20.log

Created attachment 838383 [details]
/etc/libvirt/qemu.conf

Created attachment 838384 [details]
/etc/libvirt/libvirtd.conf

Created attachment 838386 [details]
/etc/libvirt/libvirt.conf

Created attachment 838387 [details]
/etc/libvirt/qemu-lockd.conf

Created attachment 838389 [details]
/etc/libvirt/virtlockd.conf

I hit this too when trying doing some upstream libvirt dev, trying to start a pre-existing VM, and selinux permissive. I didn't dig into the code, but here's what I think is happening:

Libvirt auto allocates a socket for us (virt-manager/virt-install) in /var/lib/libvirt/qemu/channel/target. But when the guest shuts down, the socket isn't removed. It's left with whatever user/group libvirt ran as: if using fedora packages, that's qemu:qemu, if using libvirt.git, that's likely root:root.

If you switch back and forth between the two versions, trying to start a previously defined VM with the auto socket bit, or create a new VM with a reused name, it will try to reuse the existing socket path. But if the user/group don't match, EVEN if running the emulator as root (prob due to dropping of capabilities which effectively removes root privilege as well), the emulator can't access the existing socket, startup fails.

In this case Josh, maybe you tried to create a new VM that reused the name of a pre-existing VM, the left over socket had old selinux labeling.

Workaround is to: shutdown all VMs, sudo rm -rf /var/lib/libvirt/qemu/channel/target/*

This did not work for me - seems to be fairly easy to reproduce.  I reserved a beaker system with f20 installed, installed qemu-kvm, libvirt, etc. and was able to create a vm w/ virt-install. This was then used for a virt-test run...
I usually clone autotest and virt-test, then use the './run -t libvirt --bootstrap' and being running tests (you could probably at least see a guest create using just --install --remove and not selecting any tests).

I then cloned, built, and installed both libvirt.git and libvirt-python.git. The latter is required to even run virt-install. After doing that, running virt-test failed to create the guest with the error message in the log:


* Command:
    /usr/bin/virt-install --hvm --accelerate --name 'virt-tests-vm1'
    --ram=1024 --vcpu=2 --import --vnc --os-variant fedora19 --serial pty
    --disk path=/home/virt-
    test/shared/data/images/jeos-19-64.qcow2,bus=virtio,size=10,format=qcow2
    --network=bridge=virbr0,model=virtio,mac=52:54:00:94:95:96 --noautoconsole
Exit status: 1
Duration: 1.12764596939

stdout:

Starting install...
stderr:
ERROR    internal error: process exited while connecting to monitor: qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/virt-tests-vm1.org.qemu.guest_agent.0,server,nowait: Failed to bind socket: Permission denied
chardev: opening backend "socket" failed


I tried the "rm -rf /var/lib/libvirt/qemu/channel/target/*", but that did not help.

Why is this comment private?

(In reply to John Ferlan from comment #8)
> This did not work for me - seems to be fairly easy to reproduce.  I reserved
> a beaker system with f20 installed, installed qemu-kvm, libvirt, etc. and
> was able to create a vm w/ virt-install. This was then used for a virt-test
> run...
> I usually clone autotest and virt-test, then use the './run -t libvirt
> --bootstrap' and being running tests (you could probably at least see a
> guest create using just --install --remove and not selecting any tests).
> 
> I then cloned, built, and installed both libvirt.git and libvirt-python.git.
> The latter is required to even run virt-install. After doing that, running
> virt-test failed to create the guest with the error message in the log:
> 
> 
> * Command:
>     /usr/bin/virt-install --hvm --accelerate --name 'virt-tests-vm1'
>     --ram=1024 --vcpu=2 --import --vnc --os-variant fedora19 --serial pty
>     --disk path=/home/virt-
>     test/shared/data/images/jeos-19-64.qcow2,bus=virtio,size=10,format=qcow2
>     --network=bridge=virbr0,model=virtio,mac=52:54:00:94:95:96
> --noautoconsole
> Exit status: 1
> Duration: 1.12764596939
> 
> stdout:
> 
> Starting install...
> stderr:
> ERROR    internal error: process exited while connecting to monitor:
> qemu-system-x86_64: -chardev
> socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/virt-tests-
> vm1.org.qemu.guest_agent.0,server,nowait: Failed to bind socket: Permission
> denied
> chardev: opening backend "socket" failed
> 
> 
> I tried the "rm -rf /var/lib/libvirt/qemu/channel/target/*", but that did
> not help.

Hmm. Does virt-test run virt-install as root or as regular user? Maybe libvirt is trying to stick the socket in /var/lib when using qemu:///session?

I'm running virt-test as root on the beaker system


As pointed out before in email - if I modify the virt-install slightly, then things work, but IIRC that has to do with whether libvirt automatically creates the channel or not...

diff --git a/virt-install b/virt-install
index b003e95..b0bf392 100755
--- a/virt-install
+++ b/virt-install
@@ -493,12 +493,6 @@ def build_guest_instance(conn, options):
     convert_old_disks(options)
     cli.convert_old_features(options)
 
-    # Install options
-    guest.installer.extraargs = options.extra
-    guest.installer.initrd_injections = options.initrd_injections
-    cli.set_os_variant(guest, options.distro_type, options.distro_variant)
-    guest.os.init = options.init
-
     # Guest configuration
     cli.get_uuid(guest, options.uuid)
     cli.get_vcpus(guest, options.vcpus, options.check_cpu)
@@ -537,6 +531,12 @@ def build_guest_instance(conn, options):
     guest.add_default_usb_controller()
     guest.add_default_channels()
 
+    # Install options
+    guest.installer.extraargs = options.extra
+    guest.installer.initrd_injections = options.initrd_injections
+    cli.set_os_variant(guest, options.distro_type, options.distro_variant)
+    guest.os.init = options.init
+
     # Do this after setting up all optional parameters, so we report error
     # about those first.
     need_storage, need_install = validate_required_options(options, guest)

Right, that change virt-install will prevent the qemu-ga channel from being added to the guest (in a roundabout way). Still, the issue has something to do with libvirt's handling of the autogenerated socket paths. You could also manually work around it with --channel none, but that's probably not easy to inject into virt-test

I found a different way to "work around" this without editing the virt-install on f20 - change the protections of the directories too not just the files:

          chown -R root:root /var/lib/libvirt/qemu/channel

reading back through the comments - I forget why I made things private when I did - perhaps just habit... 

I've undone the settings...

https://bbs.archlinux.org/viewtopic.php?id=177483 appears to be related.

Should virt-install chown the /var/lib/libvirt/qemu/channel and /var/lib/libvirt/qemu/dump directories according to the user/group settings in /etc/libvirt/qemu.conf before launching qemu?

I can confirm this issue too.
I executed "chown -R root:root /var/lib/libvirt/qemu/channel" to resolve it.

Should be fixed by https://www.redhat.com/archives/libvir-list/2015-April/msg01212.html

(In reply to Cole Robinson from comment #15)
This will fix it only for the default user.  As Dan said, we also need to have a special folder for each user that's going to have a domain running under it.

(In reply to Martin Kletzander from comment #16)
> (In reply to Cole Robinson from comment #15)
> This will fix it only for the default user.  As Dan said, we also need to
> have a special folder for each user that's going to have a domain running
> under it.

How do we end up with different VMs running as different users? Do you mean

- start a VM with qemu.conf qemu:qemu
- change qemu.conf to root:root
- restart libvirtd
- start a new VM

I think the patches cover that case. The first VM already has an fd for the socket, changing the directory permissions when libvirtd restarts shouldn't affect the running VM.

Or maybe there's some way to configure user/group per VM that I'm not aware of...

Hmm I see in bug 1146886 that there _is_ a way to specify a manual user:group per VM... that's pretty crazy :)

However I still consider this reported issue fixed since it solves the more likely problem of changing global qemu.conf user:group

F20 is pretty late in life, so moving this to f21

libvirt-1.2.9.3-1.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/libvirt-1.2.9.3-1.fc21

Package libvirt-1.2.9.3-1.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libvirt-1.2.9.3-1.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-7150/libvirt-1.2.9.3-1.fc21
then log in and leave karma (feedback).