Created attachment 385230 [details] Patch to display appropriate error message when libvirtd client fails to connect to libvirtd server using TLS Customer had a typo in client's "Distinguished Name" specified in "tls_allowed_dn_list" in /etc/libvirt/libvirtd.conf on the libvirtd server/hypervisor.Because of the error in defining the "tls_allowed_dn_list" field, the libvirtd client was being refused a connection to libvirtd on the remote host. The following error message showed up in /var/log/messages file on the libvirtd server. Dec 3 14:53:09 sf4150host01 libvirtd: 14:53:09.859: error : remoteCheckCertificate: client's Distinguished Name is not on the list of allowed clients (tls_allowed_dn_list). Use 'openssl x509 -in clientcert.pem -text' to view the Distinguished Name field in the client certificate, or run this daemon with --verbose option. As per the above message,he executed "openssl x509 -in clientcert.pem -text" command and got output in below format. "C=AU, O=Sydney Medical School, L=Sydney, ST=NSW, CN=sf4150host00.med.usyd.edu.au" He tried using this string in "libvirtd.conf" and he was not able to get rid of the authentication failure issue . Then he used "certtool -i --infile /etc/pki/libvirt/clientcert.pem" command to retrieve "DN" string from the certificate and output was , "C=AU,O=Sydney Medical School,L=Sydney,ST=NSW,CN=sf4150host00.med.usyd.edu.au" When compared to "openssl" output we can see that there is a "space" character difference among these .Using the DN string retrieved using "certtool" command he was able to get it working. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: I've reproduced this using a KVM setup. 1)Set up TLS certificates for verifying the identity of libvirtd server and client.Refer to http://libvirt.org/remote.html for more information on setting up a Certificate Authority(CA) and TLS certificates. 2)Setup of libvirtd server: 2.1)Enable an access control list of client certificate Distinguished Names (DNs) which can connect to the TLS port on this server.In order to reproduce the customer reported error,make sure that there are spaces after the comma in the fields. vi /etc/libvirt/libvirtd.conf # By default, no DN's are checked tls_allowed_dn_list = ["C=US, O=Red Hat, L=Raleigh, ST=Raleigh, CN=nachandr.rdu.redhat.com"] 2.2)Enable logging.Refer to http://libvirt.org/logging.html for more information on logging in libvirt. Note that I've used config variables in configuration files to control logging behavior.Environment variables can also be used to control logging behavior. # Logging controls # # Logging level: 0 none, 4 errors, 3 warnings, 2 informations, 1 debug # basically 1 will log everything possible log_level = 1 2.3)After making the above changes,start the libvirtd server in listening mode by running it with the '--listen' and '--verbose' options or by editing /etc/sysconfig/libvirtd. vi /etc/sysconfig/libvirtd LIBVIRTD_ARGS="--listen --verbose" The verbose option is to get more informative log messages. Starting libvirtd with the listen option ensures that it listens for secure TLS connections on the public TCP/IP port.Also note that listen_tls[default value= 1] in /etc/libvirt/libvirtd.conf is turned on by default. 2.4)Start libvirtd server /etc/init.d/libvirtd start 3)Run the following command on the libvirtd client to connect to the remote libvirtd server: virsh -c qemu+tls://dhcp243-202.rdu.redhat.com/system list --all Actual results: libvirtd client fails to connect to the libvirtd server and the following misleading error message is seen in /var/log/messages on libvirtd server: Dec 3 14:53:09 sf4150host01 libvirtd: 14:53:09.859: error : remoteCheckCertificate: client's Distinguished Name is not on the list of allowed clients (tls_allowed_dn_list). Use 'openssl x509 -in clientcert.pem -text' to view the Distinguished Name field in the client certificate, or run this daemon with --verbose option. Expected results: A more appropriate error message like the following should be displayed: "remoteCheckCertificate: client's Distinguished Name is not on the list of allowed clients (tls_allowed_dn_list).Libvirt supports a Certificate Authority(CA)set up by GnuTLS certtool.Run 'certtool -i --infile /etc/pki/libvirt/clientcert.pem' to view the Distinguished Name field in the client certificate, or run this daemon with --verbose option." Additional info: When there are no spaces after the comma in the fields in the DN specified in tls_allowed_dn_list,libvirtd client can successfully connect the the libvirtd server. vi /etc/libvirt/libvirtd.conf # By default, no DN's are checked tls_allowed_dn_list = ["C=US,O=Red Hat,L=Raleigh,ST=Raleigh, CN=nachandr.rdu.redhat.com"]
Created attachment 447511 [details] Updated patch, against present upstream git source. This bug bit me too, when I was working through understanding and documenting TLS recently. Also had to go through the process of figuring out it was caused by spaces, so thanks Nandini. :) With your patch, it the right kind of thing, but it's hard coding a full path to the default client cert file, which might be different on the client's computer. I've taken the spirit/intention behind your patch though, and used it to generate a similar patch against the present libvirt daemon. This has been submitted upstream here: http://www.redhat.com/archives/libvir-list/2010-September/msg00252.html
Also checked the qemu-kvm git tree upstream, but they don't appear to have "openssl" mentioned anywhere. So, I think this is limited to just libvirtd, and the submitted patch will fix the problem.
This patch has been accepted and applied upstream. Still needs to be submitted to the RHEL 5.x tree.
This patch has been submitted for RHEL inclusion in 5.6.
Fix built in libvirt-0.8.2-5.el5
Verified with Passed in below environment on 4 arches: -RHEL5.6-Server-x86_64-KVM -RHEL5.6-Server-x86_64-Xen -RHEL5.6-Client-i386-Xen -RHEL5.6-Server-ia64-Xen kernel-xen-2.6.18-228.el5 xen-3.0.3-117.el5 kvm-qemu-img-83-205.el5 kernel-2.6.18-228.el5 libvirt-0.8.2-8.el5 steps: Make sure package "gnutls-utils" is installed. 1. Create a private key for CA # certtool --generate-privkey > cakey.pem 2. self-sign it by creating a file with the signature details called ca.info containing: cn=Redhat QA,CN=10.66.70.128 ca cert_signing_key # certtool --generate-self-signed --load-privkey cakey.pem --template ca.info --outfile cacert.pem then delete ca.info # rm ca.info -f 3. copy certificate to right position # cp cacert.pem cakey.pem /etc/pki/CA 4. issuing server certificates 1> Make a private key for the server: # certtool --generate-privkey > serverkey.pem 2> and sign that key with the CA's private key by first creating a template file called server.info: organization = redhat-china cn = 10.66.70.128 tls_www_server encryption_key signing_key 3> sign # certtool --generate-certificate --load-privkey serverkey.pem \ --load-ca-certificate cacert.pem --load-ca-privkey cakey.pem \ --template server.info --outfile servercert.pem 5. copy server certificates to right position # mkdir -p /etc/pki/libvirt/private # cp serverkey.pem /etc/pki/libvirt/private # cp servercert.pem /etc/pki/libvirt 6. issuing client certificates 1>. Make a private key # certtool --generate-privkey > clientkey.pem 2> Create client.info and sign the certificate client.info: country = China state = Beijign locality = Beijing organization = Red Hat cn = client1 tls_www_client encryption_key signing_key 3> sign # certtool --generate-certificate --load-privkey clientkey.pem \ --load-ca-certificate cacert.pem --load-ca-privkey cakey.pem \ --template client.info --outfile clientcert.pem 7. copy client certificates to right position # cp clientkey.pem /etc/pki/libvirt/private # cp clientcert.pem /etc/pki/libvirt 8. enable "listen_tls" in "libvirtd.conf" 9. Setup of libvirt server: vi /etc/libvirt/libvirt.conf # By default, no DN's are checked tls_allowed_dn_list = ["C=China, O=Red Hat, L=Beijing, ST=Beijign, CN=client1"] 10. Enable logging. vi /etc/libvirt/libvirt.conf # Logging level: 0 none, 4 errors, 3 warnings, 2 informations, 1 debug # basically 1 will log everything possible log_level = 1 11. stop libvirtd # service libvirtd stop 12. start libvirtd with option "--listen" # libvirtd -d --listen 13. #virsh -c qemu+tls://10.66.70.128/system list --all expect result: on /var/log/message it has: "remoteCheckCertificate: client's Distinguished Name is not on the list of allowed clients (tls_allowed_dn_list).Run 'certtool -i --infile /etc/pki/libvirt/clientcert.pem' to view the Distinguished Name field in the client certificate, or run this daemon with --verbose option."
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHEA-2011-0060.html