Bug 1067683

Summary: Allinone install fails to start VM's due to missing spice path
Product: Red Hat Enterprise Virtualization Manager Reporter: Dave Kline <dkline>
Component: vdsmAssignee: Yedidyah Bar David <didi>
Status: CLOSED DUPLICATE QA Contact: Aharon Canan <acanan>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.3.0CC: abaron, acathrow, alonbl, bazulay, danken, didi, dkline, gklein, iheim, lpeer, pstehlik, Rhev-m-bugs, sbonazzo, ybronhei, yeylon
Target Milestone: ---Keywords: Triaged
Target Release: 3.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: integration
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-03-10 06:34:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
qemu.conf
none
vdsm.conf none

Description Dave Kline 2014-02-20 20:54:50 UTC 
Description of problem:
After performing an all-in-one install, RHEV will not start virtual machines due to a missing path.  The exact error message RHEV-M gives is:

VM r7 is down. Exit message: internal error process exited while
connecting to monitor: ((null):25215): Spice-Warning **:
reds.c:3255:reds_init_ssl: Could not load certificates from
/etc/pki/libvirt-spice/server-cert.pem failed to initialize spice
server.

The 'libvirt-spice' directory does not exist, however creating a symlink works around the issue:

[root@work2 pki]# pwd 
/etc/pki
[root@work2 pki]# ls -tlrh
total 48K
drwx------. 2 root  root  4.0K Aug 15  2013 rsyslog
drwxr-xr-x. 2 root  root  4.0K Jan 31 14:40 rpm-gpg
drwxr-xr-x. 4 root  root  4.0K Jan 31 15:07 ca-trust
drwxr-xr-x. 2 root  root  4.0K Jan 31 15:07 java
drwxr-xr-x. 5 root  root  4.0K Jan 31 15:07 tls
drwxr-xr-x. 2 root  root  4.0K Jan 31 15:45 product
drwxr-xr-x. 2 root  root  4.0K Feb  3 16:25 entitlement
drwxr-xr-x. 5 vdsm  kvm   4.0K Feb 13 09:01 vdsm
drwxr-xr-x. 6 ovirt ovirt 4.0K Feb 19 16:04 ovirt-engine
drwxr-xr-x. 3 root  root  4.0K Feb 19 16:04 libvirt
drwxr-xr-x. 6 root  root  4.0K Feb 19 16:04 CA
lrwxrwxrwx. 1 root  root    19 Feb 20 11:42 libvirt-spice -> vdsm/libvirt-spice/
drwxr-xr-x. 2 root  root  4.0K Feb 20 13:40 nssdb

Version-Release number of selected component (if applicable):

3.3.0-0.46.el6ev

Additional info:

The all-in-one install is shipped by Red Hat, however longer term it appears a single-node hosted install would be a more elegant path.  That said, the convenience and simplicity of the all-in-one install is valuable.
Comment 1 Fabian Deutsch 2014-02-21 08:22:28 UTC 
Itamar,

it seems as if there is now component for the All-In-One image. If we ship it we should have a component, shouldn't we?
Comment 2 Itamar Heim 2014-02-21 08:27:51 UTC 
the rpm is part of setup package iirc.
Comment 3 Alon Bar-Lev 2014-02-24 18:13:53 UTC 
This is strange!
vdsm should use /etc/pki/vdsm/libvirt-spice/server-cert.pem

Dan, any thought why it looks at the wrong place?
Comment 4 Yaniv Bronhaim 2014-02-25 06:38:04 UTC 
its declared in /etc/libvirt/qemu.conf (spice_tls_x509_cert_dir="/etc/pki/vdsm/libvirt-spice")

Dave, can you check if the value exists there after fresh allinone installation ? it also part of the libvirt_configure when we set qemu.conf values.
Comment 5 Dan Kenigsberg 2014-02-25 13:34:46 UTC 
Dave, please attach your complete /etc/libvirt/qemu.conf and /etc/vdsm/vdsm.conf.

Have you disabled ssl at one time?
Comment 6 Dave Kline 2014-02-25 22:18:54 UTC 
Created attachment 867656 [details]
qemu.conf
Comment 7 Dave Kline 2014-02-25 22:19:30 UTC 
Created attachment 867657 [details]
vdsm.conf
Comment 8 Dave Kline 2014-02-25 22:24:12 UTC 
Yaniv:
On a fresh install, the spice_tls_x509_cert_dir variable matches your output.

Dan:
Attached.  I have not disabled SSL.

Some additional information.  I did a fresh install (allinone / rhevm-setup) on a new host and couldn't replicate the problem.  On the host exibiting the problem, I'd initially tried an unsuccessful self-hosted install via 'hosted-engine --deply' before trying allinone/rhevm-setup.
Comment 9 Sandro Bonazzola 2014-02-28 09:46:13 UTC 
Didi, I think you've seen something similar this week. Can you take a look?
Comment 10 Yedidyah Bar David 2014-03-09 13:07:09 UTC 
(In reply to Dave Kline from comment #8)
> Yaniv:
> On a fresh install, the spice_tls_x509_cert_dir variable matches your output.
> 
> Dan:
> Attached.  I have not disabled SSL.
> 
> Some additional information.  I did a fresh install (allinone / rhevm-setup)
> on a new host and couldn't replicate the problem.  On the host exibiting the
> problem, I'd initially tried an unsuccessful self-hosted install via
> 'hosted-engine --deply' before trying allinone/rhevm-setup.

(In reply to Sandro Bonazzola from comment #9)
> Didi, I think you've seen something similar this week. Can you take a look?

Sorry, don't remember the exact details. I am pretty certain that the failed hosted-engine deploy left the system in a "bad" state (as described above) which is not solved by merely re-configuring system components for vdsm (libvirt, spice etc) - because vdsm-tool checks and thinks it already did the required configuration and does not do the required changes.

The real bug is that hosted-engine/all-in-one/vdsm do not provide cleanup scripts, nor manage to clean up previous attempts during setup.

I currently suggest to close this as duplicate of bug #1034634 .

Dave - can you please verify that running a recent (changed merged upstream at end of last week) hosted-engine --deploy, then killing it in the middle (e.g. when it asks if the OS was successfully installed on the VM reply 'Cancel' or whatever it's called), then trying engine-setup all-in-one does work?
Comment 11 Dave Kline 2014-03-10 00:30:59 UTC 
Happy to try again with the latest bits.  I'll need a few days due to travel and obligations however.
Comment 12 Yedidyah Bar David 2014-03-10 06:34:06 UTC 
(In reply to Dave Kline from comment #11)
> Happy to try again with the latest bits.  I'll need a few days due to travel
> and obligations however.

Very well. I am currently closing this bug. Please reopen if still relevant. Thanks!

*** This bug has been marked as a duplicate of bug 1034634 ***