Bug 1008328 - Failed to run vm with libvirt internal error
Failed to run vm with libvirt internal error
Product: ovirt-host-deploy
Classification: oVirt
Component: Plugins.VDSM (Show other bugs)
Unspecified Unspecified
unspecified Severity unspecified (vote)
: ---
: 1.1.0
Assigned To: Michal Skrivanek
Tareq Alayan
: Triaged
Depends On: 964359 1006511
Blocks: 1019461
  Show dependency treegraph
Reported: 2013-09-16 03:34 EDT by Artyom
Modified: 2015-09-20 16:03 EDT (History)
15 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2013-11-19 04:11:42 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
pm-rhel: blocker+
michal.skrivanek: devel_ack+

Attachments (Terms of Use)
Vdsm and libvirtd logs (250.00 KB, application/gzip)
2013-09-16 03:34 EDT, Artyom
no flags Details
host deploy and certificate (30.00 KB, application/gzip)
2013-09-17 04:05 EDT, Artyom
no flags Details

External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 19316 None None None Never

  None (edit)
Description Artyom 2013-09-16 03:34:25 EDT
Created attachment 798133 [details]
Vdsm and libvirtd logs

Description of problem:
Failed to run vm with libvirt internal error:
libvirtError: internal error process exited while connecting to monitor: ((null):21350): Spice-Warning **: reds.c:3236:reds_init_ssl: Could not load certificates from /etc/pki/vdsm/libvirt-spice/server-cert.pem

Version-Release number of selected component (if applicable):
rhevm - is14, rhel6.5(2.6.32-358.el6.x86_64), rhevm-spice-client-x64-cab-3.3-4.el6_4.noarch
host - rhel6.5(2.6.32-358.18.1.el6.x86_64), vdsm-4.12.0-127.gitedb88bf.el6ev.x86_64, libvirt-0.10.2-23.el6.x86_64, qemu-kvm-rhev-

How reproducible:

Steps to Reproduce:
1. rhel6.5 on rhevm and also on node
2. create new vm and try to run on node

Actual results:
Run failed with internal error

Expected results:
Vm run normally

Additional info:
With VNC as console vm run normally
SELinux disabled
Comment 1 Michal Skrivanek 2013-09-17 03:30:38 EDT
it indeed fails in QEMU startup, looks like the certificate was not generated/deployed? Alon?
Comment 2 Michal Skrivanek 2013-09-17 03:31:57 EDT
also I suppose we need install logs.
Comment 3 Alon Bar-Lev 2013-09-17 03:37:45 EDT
Please attach:

1. host: /etc/pki/vdsm/libvirt-spice/server-cert.pem
2. engine: relevant /var/log/ovirt-engine/host-deploy/*.log

Comment 4 Artyom 2013-09-17 04:05:14 EDT
Created attachment 798658 [details]
host deploy and certificate
Comment 5 Alon Bar-Lev 2013-09-17 04:24:51 EDT
# ls -lad /etc/pki/vdsm/libvirt-spice/
drwxr-x---. 2 root kvm 20480 Sep 15 16:44 /etc/pki/vdsm/libvirt-spice/

notice: owned by root and not vdsm.
notice: not readable by world.

Something else created that directory before host-deploy, can you please determine what?

And question... why is this directory not owned by the vdsm spec...
Comment 6 Alon Bar-Lev 2013-09-17 04:26:11 EDT
Hi Dan,

Any reason why /etc/pki/vdsm/libvirt-spice not owned by the vdsm spec?
%dir %{_sysconfdir}/pki/%{vdsm_name}
%dir %{_sysconfdir}/pki/%{vdsm_name}/keys
%dir %{_sysconfdir}/pki/%{vdsm_name}/certs
%config(noreplace) %{_sysconfdir}/pki/%{vdsm_name}/keys/libvirt_password
Comment 7 Alon Bar-Lev 2013-09-17 04:31:50 EDT

./vdsm_reg/deployUtil.py messes with that directory!

So host was installed using rhevm-3.1 will will have this issue, as rpm of upgraded vdsm will not enforce permissions and mode.

        SPICEPKIPATH = os.path.join(ts, 'libvirt-spice')

        if not os.path.exists(SPICEPKIPATH):

        os.chown(SPICEPKIPATH, 0, nGID)
        os.chmod(SPICEPKIPATH, 0750)
Comment 8 Michal Skrivanek 2013-09-17 04:41:40 EDT
good catch. flagging for 3.3 as this needs to be resolved for the release...
Comment 9 Alon Bar-Lev 2013-09-17 05:10:01 EDT

I do not understand why it actually did not work.

# ls -lad /etc/pki/vdsm/libvirt-spice/ /etc/pki/vdsm/libvirt-spice/*.pem
drwxr-x---. 2 root kvm  20480 Sep 15 16:44 /etc/pki/vdsm/libvirt-spice/
-rw-r--r--  1 root root  1472 Sep 15 16:44 /etc/pki/vdsm/libvirt-spice/ca-cert.pem
-rw-r--r--  1 root root  1594 Sep 15 16:44 /etc/pki/vdsm/libvirt-spice/server-cert.pem
-r--r-----  1 vdsm kvm   1675 Sep 15 16:44 /etc/pki/vdsm/libvirt-spice/server-key.pem

Unless qemu does not run under kvm any more... it should be been able to access the file.

Comment 10 Alon Bar-Lev 2013-09-17 05:14:14 EDT
# su - qemu -s /bin/sh
-sh-4.1$ cat /etc/pki/vdsm/libvirt-spice/server-cert.pem

so problem is different.

as far as I can see the CA and server certificates are OK, and is the one that was issued at deploy.
Comment 11 Michal Skrivanek 2013-09-17 05:48:18 EDT
hmm can we get qemu log? or yt from cmdline?
this is what libvirt issued:
2013-09-16 06:48:13.345+0000: 8691: debug : virCommandRunAsync:2253 : About to run LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_
DRV=spice /usr/libexec/qemu-kvm -name vm_1 -S -M rhel6.5.0 -cpu Conroe -enable-kvm -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid cf2f4209-f283-455
c-8163-d0aec5903a26 -smbios 'type=1,manufacturer=Red Hat,product=RHEV Hypervisor,version=6Server-,serial=802B8DA9-3B43-B601-19FE-00145EDD0555,uuid=cf2f4209-f2
83-455c-8163-d0aec5903a26' -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/vm_1.monitor,server,nowait -mon chardev=charmonitor,id=moni
tor,mode=control -rtc base=2013-09-16T06:48:13,driftfix=slew -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0
,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-drive,bu
s=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/var/run/vdsm/storage/a39cc532-84ea-45b0-9beb-d69f40c84cc0/cdb48ddc-eb04-44b1-b409-ceaf699c3db1/82eaf02a-4b21
-42a4-b9e8-4ef3743a631e,if=none,id=drive-virtio-disk0,format=raw,serial=cdb48ddc-eb04-44b1-b409-ceaf699c3db1,cache=none,werror=stop,rerror=stop,aio=threads -device virti
o-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device virtio-net-pci,netde
v=hostnet0,id=net0,mac=00:1a:4a:34:fe:e7,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/cf2f4209-f283-455c-8163-d0aec5903a26.com.
redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchann
el1,path=/var/lib/libvirt/qemu/channels/cf2f4209-f283-455c-8163-d0aec5903a26.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,charde
v=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchanne
l2,id=channel2,name=com.redhat.spice.0 -spice port=5902,tls-port=5903,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs
,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -k en-us -vga qxl -global qxl-vga.ram_size=6
7108864 -global qxl-vga.vram_size=33554432 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7
Comment 12 Michal Skrivanek 2013-09-17 05:49:15 EDT
I mean to try the same command from cmdline on that server
Comment 13 Alon Bar-Lev 2013-09-17 05:53:58 EDT
Not that this actually fixes anything as far as I see, but better reaching consistent state and drop the legacy.

commit f0d30c31f936005eaaa44af21e8d22b20a9a74f7
Author: Alon Bar-Lev <alonbl@redhat.com>
Date:   Tue Sep 17 11:56:51 2013 +0300

    vdsm: pki: fix permission of spice directory
    old vdsm-bootstrap implementations touched
    spice pki directory explicitly, so we need to revert
    to something sane.
    Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1008328
    Change-Id: Ib47feea4d9beace8acc38cfc0e4cd18a46c22654
    Signed-off-by: Alon Bar-Lev <alonbl@redhat.com>
Comment 14 Dan Kenigsberg 2013-09-17 05:54:23 EDT
I do not recall any reason not to include the /etc/pki/vdsm/libvirt-spice directory in vdsm.rpm (its content, though, must not be listed there. we do not want to clear vdsm's key/cert on a temporary vdsm.rpm removal).

However, this report is most likely an instance of https://access.redhat.com/security/cve/CVE-2013-4291 .

Please upgrade to libvirt-0.10.2-24 and try again.
Comment 15 Michal Skrivanek 2013-09-17 05:58:59 EDT
bug 1006394 seems to be the same issue
Comment 16 Michal Skrivanek 2013-11-19 04:11:42 EST
closing as per comment #14

Note You need to log in before you can comment on or make changes to this bug.