Bug 684764 - VDSM: Can't run VM. due to Could not load certificates from /etc/pki/libvirt-spice/server-cert.pem
Summary: VDSM: Can't run VM. due to Could not load certificates from /etc/pki/libvirt...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: vdsm
Version: 6.1
Hardware: x86_64
OS: Linux
medium
low
Target Milestone: rc
: ---
Assignee: Yotam Oron
QA Contact: Jakub Libosvar
URL:
Whiteboard:
: 706039 709696 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-03-14 13:01 UTC by Avi Tal
Modified: 2016-04-22 04:58 UTC (History)
13 users (show)

Fixed In Version: vdsm-4.9-76.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-12-06 07:09:08 UTC
Target Upstream Version:


Attachments (Terms of Use)
vdsm log (5.98 MB, application/octet-stream)
2011-03-14 13:04 UTC, Avi Tal
no flags Details
vds_bootstrap log (23.10 KB, application/octet-stream)
2011-06-06 10:07 UTC, Jakub Libosvar
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2011:1782 0 normal SHIPPED_LIVE new packages: vdsm 2011-12-06 11:55:51 UTC

Description Avi Tal 2011-03-14 13:01:15 UTC
Description of problem:
Can't create VM. due to Could not load certificates from /etc/pki/libvirt-spice/server-cert.pem

Steps to Reproduce:
1. create VM
2.
3.
  

Vdsm.log:
[root@navy-vds1 ~]# less /var/log/vdsm/vdsm.log
Thread-6550::ERROR::2011-03-14 14:23:36,330::vm::643::vm.Vm::(_startUnderlyingVm) vmId=`d7138c82-b646-4673-a9eb-bc9eca8c6b8e`::Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 613, in _startUnderlyingVm
    self._run()
  File "/usr/share/vdsm/libvirtvm.py", line 782, in _run
    self._connection.createXML(domxml, flags),
  File "/usr/share/vdsm/libvirtconnection.py", line 72, in wrapper
    raise e
libvirtError: internal error process exited while connecting to monitor: do_spice_init: starting 0.7.2
reds_init_ssl: Could not load certificates from /etc/pki/libvirt-spice/server-cert.pem


Thread-6550::DEBUG::2011-03-14 14:23:36,333::vm::1776::vm.Vm::(setDownStatus) vmId=`d7138c82-b646-4673-a9eb-bc9eca8c6b8e`::Changed state to Down: internal error process exited wh
ile connecting to monitor: do_spice_init: starting 0.7.2
reds_init_ssl: Could not load certificates from /etc/pki/libvirt-spice/server-cert.pem

Comment 1 Avi Tal 2011-03-14 13:03:04 UTC
libvirtd.conf:
listen_addr="0" # by vdsm
unix_sock_group="kvm" # by vdsm
unix_sock_rw_perms="0770" # by vdsm
auth_unix_rw="sasl" # by vdsm
save_image_format="lzop" # by vdsm
log_outputs="1:file:/var/log/libvirtd.log" # by vdsm
log_filters="1:util 3:json 1:libvirt 1:qemu 1:remote" # by vdsm
auth_tcp="none" # by vdsm
listen_tcp=1 # by vdsm
listen_tls=0 # by vdsm


missing the following values:
ca_file="/etc/pki/vdsm/certs/cacert.pem"
cert_file="/etc/pki/vdsm/certs/vdsmcert.pem"
key_file="/etc/pki/vdsm/keys/vdsmkey.pem"

Comment 2 Avi Tal 2011-03-14 13:04:35 UTC
Created attachment 484171 [details]
vdsm log

Comment 3 Dan Kenigsberg 2011-03-27 16:45:53 UTC
But do the files at /etc/pki/libvirt-spice/server-cert.pem and /etc/pki/vdsm/certs/cacert.pem etc actually exist?

I think this can happen if you manually changed vdsm to use ssl without generating the keys.

I guess it makes sense for vdsm to check their configuration and existence on service vdsmd start.

Comment 4 Yotam Oron 2011-04-11 10:54:37 UTC
This looks like https://bugzilla.redhat.com/show_bug.cgi?id=683905, is it reproducing since the commit that fixed 683905 ?

Comment 5 Avi Tal 2011-04-17 07:21:39 UTC
/etc/pki/libvirt-spice/server-cert.pem and /etc/pki/vdsm/certs/cacert.pem files does exist!
I did not change any default vdsm settings

This bug is very hard to reproduce because there is no pre-requirements

Comment 6 Avi Tal 2011-04-17 08:06:10 UTC
After talking to Yotam i would like to clear my bug description.
The bug is:
vdsm does not fully configure libvirtd.conf the following lines are missing:
ca_file="/etc/pki/vdsm/certs/cacert.pem"
cert_file="/etc/pki/vdsm/certs/vdsmcert.pem"
key_file="/etc/pki/vdsm/keys/vdsmkey.pem"

Manually adding these line makes vdsm works great!

Comment 7 Yotam Oron 2011-04-18 07:16:30 UTC
From reading the code, I assume that this can happen if the vdsm is started before the installation process is over.
The vdsm will not re-insert values into libvirtd.conf if it already did so, so subsequent installs won't change the (bad) values there.
I think this is the scenario that reproduces it:
- Install RHEL6 from scratch
- Install VDSM
- Start the vdsm
- Stop the vdsm
- Start installation

You should see the bug reproducing.
Opened a preintegration ticket.

Comment 8 Yotam Oron 2011-05-23 07:28:20 UTC
*** Bug 706039 has been marked as a duplicate of this bug. ***

Comment 11 Jakub Libosvar 2011-06-06 10:01:03 UTC
Reproducible in vdsm-71

Problem is when configuration script is run before host is installed into rhev-m environment, libvirtd.conf is edited and due to certificates in /etc/pki/vdsm/ don't exist, following lines are appended
auth_tcp="none" # by vdsm
listen_tcp=1 # by vdsm
listen_tls=0 # by vdsm

After host is installed into rhev-m env., configuration script is not run again therefore it can't append paths to certs.

Comment 12 Jakub Libosvar 2011-06-06 10:02:56 UTC
Steps to reproduce:
1) Remove host from rhev-m setup
2) yum -y remove libvirt vdsm vdsm-cli
3) rm -rf /etc/pki/vdsm
4) yum -y install vdsm vdsm-cli
5) service vdsmd start
6) Install host to rhev-m

Comment 13 Jakub Libosvar 2011-06-06 10:07:51 UTC
Created attachment 503189 [details]
vds_bootstrap log

Attaching vds_bootstrap log

Isn't bug 709696 dup of this one?

Comment 14 Dan Kenigsberg 2011-06-12 13:47:33 UTC
*** Bug 709696 has been marked as a duplicate of this bug. ***

Comment 15 Dan Kenigsberg 2011-06-12 14:19:44 UTC
we must not forget to call 'reconfigure' at the end of RHEL installation.

http://gerrit.usersys.redhat.com/574

Comment 16 Dan Kenigsberg 2011-07-17 11:47:57 UTC
Bug report changed to ON_QA status by me since Errata System refused to do so.
A QE request has been submitted for advisory RHEA-2011:11186-01
http://errata.devel.redhat.com/errata/show/11186

Comment 17 Tomas Dosek 2011-07-18 07:51:19 UTC
Verified - vdsm-4.9-81.el6 - above described scenarios no longer reproduce on vdsm-4.9-81.el6.

Comment 18 errata-xmlrpc 2011-12-06 07:09:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2011-1782.html


Note You need to log in before you can comment on or make changes to this bug.