Bug 501111

Summary: pki authentication not working with libvirtd
Product: [Fedora] Fedora Reporter: Jeff Layton <jlayton>
Component: libvirtAssignee: Daniel Veillard <veillard>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: low    
Version: 10CC: berrange, clalance, crobinso, itamar, jdvorak, markmc, steved, veillard, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-11-13 08:09:31 EST Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Description Flags
server side log
client side log
/etc/hosts from server
client side log (connecting to fqdn) none

Description Jeff Layton 2009-05-16 10:16:57 EDT
I just tried setting up a virtualization host with TLS "auth". I've set up a CA, verified that the certificates are good.

I've configured libvirt to listen on a socket and pointed it at the right certificate and key files. When I try to connect from my client (an F11 machine) using either virt-manager or virsh, it fails. The error from virt-manager is this:

Unable to open connection to hypervisor URI 'qemu+tls://salusa/system':
<class 'libvirt.libvirtError'> unable to connect to 'salusa': Invalid argument

...on the server side, I see this error in /var/log/messages:

2009-05-16T10:00:32.638817-04:00 salusa libvirtd: remoteCheckCertificate: the client certificate is not trusted.
2009-05-16T10:00:32.638904-04:00 salusa libvirtd: remoteCheckCertificate: failed to verify client's certificate

I've tried setting up tls_allowed_dn_list on the server to explicitly allow this cert, but that didn't make any difference. I've also put both client and server in SELinux permissive mode, but that also didn't make any difference. Finally, I tried running libvirtd in the foreground with --verbose set, but that didn't make it log anything more helpful.

Attaching my libvirtd.conf file from the server. The cert and key files are all in their default locations.
Comment 1 Jeff Layton 2009-05-16 11:17:48 EDT
For the record, I went ahead and used the certtool util to verify the certs:


$ certtool --verify-chain --infile /etc/pki/libvirt/clientcert.pem --infile /etc/pki/CA/cacert.pem  | grep Verif
	Verification output: Verified.


$ certtool --verify-chain --infile /etc/pki/libvirt/servercert.pem --infile /etc/pki/CA/cacert.pem  | grep Verif
	Verification output: Verified.
Comment 2 Daniel Berrange 2009-05-18 09:01:20 EDT
You don't appear to have attached anything as per comment #0.

Can you provide

 * /etc/libvirt/libvirtd.conf
 * /etc/hosts
 * Output from 'hostname' command
 * Full unedited output from  
    LIBVIRT_DEBUG=1  virsh -c qemu+tls://salusa/system

 * Full log from libvirtd, after settting log_level=1 in libvirtd.conf, and then running

   LIBVIRT_DEBUG=1 /usr/sbin/libvirtd

and stop the log after a connection attempt

Also can you also verify that running 'host salusa' returns an IP address, and that then running 'host $IP' returns 'salusa', and that 'salusa' matches the fully qualified  hostname configured for the machine as per the 'hostname' command.
Comment 3 Jeff Layton 2009-05-18 10:08:14 EDT
Created attachment 344440 [details]
server side log
Comment 4 Jeff Layton 2009-05-18 10:09:04 EDT
Created attachment 344441 [details]
client side log
Comment 5 Jeff Layton 2009-05-18 10:12:27 EDT
It turns out that my libvirtd.conf is actually blank once you strip out comments, so I won't bother attaching it.

Server-side hostname:

# hostname

# host salusa
salusa.poochiereds.net has address

# host domain name pointer salusa.poochiereds.net.

Similar results for the client. Certificates being used specify the FQDN's of the client and server.
Comment 6 Jeff Layton 2009-05-18 10:13:12 EDT
Created attachment 344442 [details]
/etc/hosts from server
Comment 7 Jeff Layton 2009-05-18 10:15:35 EDT
Created attachment 344443 [details]

Actually, I did make a few changes to make it so that I didn't need root privs to run virt-manager locally. Here's what I'm using now. I don't think those changes make much difference in cert handling though.
Comment 8 Daniel Berrange 2009-05-18 10:18:52 EDT
Ahh, in your initial comment you indicate you're using the URI:


Since your hostname is salusa.poochiereds.net, you should be using that in the URI, not the short form. Can you try connecting using

  'virsh -c qemu+tls://salusa.poochiereds.net/system'
Comment 9 Jeff Layton 2009-05-18 10:33:14 EDT
Created attachment 344444 [details]
client side log (connecting to fqdn)

Sorry, should have made it clear that I tried that already. The results are the same, here's the client-side log from the fqdn connect attempt.
Comment 10 Daniel Berrange 2009-05-18 13:09:06 EDT
Hmmm, are the clocks on the client and server hosts in sync with each other ?
Comment 11 Mark McLoughlin 2009-05-25 10:15:54 EDT
Comment 12 Jeff Layton 2009-05-25 10:20:14 EDT
Sorry, missed the query before. Yes, clocks are synched between client and server -- the server in this case is an ntp server as well and the client syncs to it.
Comment 13 Jan Dvorak 2009-08-14 10:54:46 EDT
I have the exactly same problem and after some investigation it seems like a bug somewhere between gnutls and libvirt. Google suggests that TLS might be broken in this case.
Comment 14 Daniel Berrange 2009-11-13 06:17:19 EST
We've recently come across an issue that looks very similar to your bug. Did you by chance generate your CA certificate using OpenSSL ?

Can you run

  # certtool -i --infile <CA-CERT-FILE>

And paste in the output from the 'Extensions' block.

What we expect to see is:

		Basic Constraints (critical):
			Certificate Authority (CA): TRUE
		Key Usage (critical):
			Certificate signing.
		Subject Key Identifier (not critical):

if the extensions field is empty then that would be the cause of this problem.
Comment 15 Jeff Layton 2009-11-13 08:03:49 EST
Haven't played with this in a while so I don't know whether it's still a problem or not...

Yes, I did generate my cacert with openssl. Here's the extensions block in the output:

		Basic Constraints (not critical):
			Certificate Authority (CA): FALSE
		Unknown extension 2.16.840.1.113730.1.13 (not critical):
			ASCII: ..OpenSSL Generated Certificate
			Hexdump: 161d4f70656e53534c2047656e657261746564204365727469666963617465
		Subject Key Identifier (not critical):
		Authority Key Identifier (not critical):
Comment 16 Daniel Berrange 2009-11-13 08:09:31 EST
Ok, this is the problem:

   Certificate Authority (CA): FALSE

your CA certificate is claiming that it is not to be used as a CA certificate. GNUTLS enforces this when checking the certificate chains on a connection, hence why its saying

  "client certificate is not trusted."

even though certtool says the cert is valid.

So if you re-create your CA  cert ensuring is has  "Certificate Authority (CA): TRUE", then it will work with libvirt.

We have another BZ open about giving a better error message diagnostic for this scenario: