Description of problem: I've created client and server certificates signed by CA certificate. If I use them for authenticate with aviary server, client cannot connect to aviary. Version-Release number of selected component (if applicable): condor-wallaby-tools-4.1-5.el6.noarch condor-classads-7.6.4-0.6.el6.i686 condor-wallaby-base-db-1.16-1.el6.noarch condor-7.6.4-0.6.el6.i686 condor-aviary-7.6.4-0.6.el6.i686 condor-debuginfo-7.6.4-0.6.el6.i686 condor-qmf-7.6.4-0.6.el6.i686 python-condorutils-1.5-4.el6.noarch condor-wallaby-client-4.1-5.el6.noarch python-2.6.6-20.el6.i686 How reproducible: 100% Steps to Reproduce: 1. create CA, server and client certificates and private keys by certutil 2. export all certificates and private keys from nss database to pem format 3. set certificates in aviary settings 4. use testing client to connect to aviary Actual results: It is not possible to connect to aviary with proper certificates and do all supported operations there. Expected results: It will be possible to connect to aviary with proper certificates and do all supported operations there. Additional info: creates certificates by certutil: function mrg_gen_ssl_certs() { MRG_CA_DIR="$1" #CA authority if [ "0${MRG_CA_DIR}" = "0" ]; then MRG_CA_DIR="$(readlink -f $(dirname $0))/CA_db" fi if [ "0${MRG_CA_PASSWORD}" = "0" ]; then MRG_CA_PASSWORD="imstrong" fi local MRG_NOISE_FILE=$(mktemp) dd count=1 bs=4096 if=/dev/urandom of=${MRG_NOISE_FILE} MRG_CA_PW_FILE="${MRG_CA_DIR}/passwordfile" MRG_CA_NICK=CAnick #SERVER related variables MRG_SRV_PASSWORD=${MRG_CA_PASSWORD} MRG_SRV_DIR=${MRG_CA_DIR} MRG_SRV_PW_FILE=${MRG_CA_PW_FILE} MRG_SRV_NICK=serv_$(uname -n) MRG_SRV_CN=$(uname -n) if [ -z "${MRG_SRV_CN}" ]; then echo "ERROR: FQDN not obtained" return 1 fi #CLIENT related variables MRG_CLI_PASSWORD=${MRG_CA_PASSWORD} MRG_CLI_DIR=${MRG_CA_DIR} MRG_CLI_PW_FILE=${MRG_CA_PW_FILE} MRG_CLI_NICK=client_$(uname -n) MRG_CLI_CN=guest if [ -n "${MRG_CA_DIR}" ]; then mkdir ${MRG_CA_DIR} echo ${MRG_CA_PASSWORD} > ${MRG_CA_PW_FILE} fi # Initialise CERT DB certutil -N -d ${MRG_CA_DIR} -f ${MRG_CA_PW_FILE} || return 2 # Generate a new public and private key pair within a key database certutil -G -d ${MRG_CA_DIR} -f ${MRG_CA_PW_FILE} -z ${MRG_NOISE_FILE} || return 3 #Create the self-signed Root CA certificate, specifying the subject name for the certificate. echo -e "y\n0\nn\n" | certutil -S -d ${MRG_CA_DIR} -n "${MRG_CA_NICK}" -s "CN=CAcert"\ -t "CT,," -x -m 1000 -v 120 -f ${MRG_CA_PW_FILE} -z ${MRG_NOISE_FILE} || return 4 #Create Server cert certutil -S -n "${MRG_SRV_NICK}" -s "CN=${MRG_SRV_CN}" -c "${MRG_CA_NICK}" -t "u,u,u"\ -m 1001 -v 120 -d ${MRG_CA_DIR} -f ${MRG_CA_PW_FILE} -z ${MRG_NOISE_FILE} || return 5 #Create Client cert certutil -S -n "${MRG_CLI_NICK}" -s "CN=${MRG_CLI_CN}" -c "${MRG_CA_NICK}" -t "u,u,u"\ -m 1002 -v 120 -d ${MRG_CA_DIR} -f ${MRG_CA_PW_FILE} -z ${MRG_NOISE_FILE} || return 6 return 0 } Export certificates from database: pk12util -d . -k passwordfile -o ca.p12 -n "NSS Certificate DB:CAnick" and same for serv.p12 and client.p12 Change format: openssl pkcs12 -in client.p12 -out client.pem -nodes and same for serv.pem and ca.pem CERTS VERIFICATION: [root@ ssl]# openssl verify -CAfile ./ca.pem serv.pem client.pem serv.pem: OK client.pem: OK [root@ ssl]# openssl verify -purpose sslclient -CAfile ./ca.pem serv.pem client.pem serv.pem: OK client.pem: OK [root@ ssl]# openssl verify -purpose sslserver -CAfile ./ca.pem serv.pem client.pem serv.pem: OK client.pem: OK [root@ ssl]# openssl verify -purpose any -CAfile ./ca.pem serv.pem client.pem serv.pem: OK client.pem: OK MODIFIED MODULE so it is possible to use ca_cert_file: try: import ssl except ImportError: pass else: class HTTPSConnection(httplib.HTTPConnection): "This class allows communication via SSL." default_port = httplib.HTTPS_PORT def __init__(self, host, port=None, key_file=None, cert_file=None, strict=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT, ca_cert_file=None): httplib.HTTPConnection.__init__(self, host, port, strict, timeout) self.key_file = key_file self.cert_file = cert_file self.ca_cert_file = ca_cert_file def connect(self): "Connect to a host on a given (SSL) port." sock = socket.create_connection((self.host, self.port), self.timeout) if self._tunnel_host: self.sock = sock self._tunnel() self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file, ca_certs=self.ca_cert_file) class HTTPSClientAuthHandler(u2.HTTPSHandler): def __init__(self, key, cert, ca_cert): u2.HTTPSHandler.__init__(self) self.key = key self.cert = cert self.ca_cert = ca_cert def https_open(self, req): #Rather than pass in a reference to a connection class, we pass in # a reference to a function which, for all intents and purposes, # will behave as a constructor return self.do_open(self.getConnection, req) def getConnection(self, host, timeout=300): return HTTPSConnection(host, key_file=self.key, cert_file=self.cert, ca_cert_file=self.ca_cert) class HTTPSClientCertTransport(HttpTransport): def __init__(self, key, cert, ca_cert, *args, **kwargs): HttpTransport.__init__(self, *args, **kwargs) self.key = key self.cert = cert self.ca_cert = ca_cert def u2open(self, u2request): """ Open a connection. @param u2request: A urllib2 request. @type u2request: urllib2.Request. @return: The opened file-like urllib2 object. @rtype: fp """ tm = self.options.timeout url = u2.build_opener(HTTPSClientAuthHandler(self.key, self.cert, self.ca_cert)) if self.u2ver() < 2.6: socket.setdefaulttimeout(tm) return url.open(u2request) else: return url.open(u2request, timeout=tm) client code: from suds import * from suds.client import Client from sys import exit, argv import time, pwd, os import logging import optparse from aviary.https import * # aviary.https is above uid = pwd.getpwuid(os.getuid())[0] if not uid: uid = "condor" # change these for other default locations and ports wsdl = 'file:/var/lib/condor/aviary/services/job/aviary-job.wsdl' key = '/etc/pki/tls/certs/client.key' cert = '/etc/pki/tls/certs/client.crt' cacert = '/etc/pki/tls/certs/client.crt' parser = optparse.OptionParser(description='Submit a job remotely via SOAP.') parser.add_option('-v','--verbose', action="store_true",default=False, help='enable SOAP logging') parser.add_option('-u','--url', action="store", nargs='?', dest='url', default='https://_host_:9090/services/job/submitJob', help='http or https URL prefix to be added to cmd') parser.add_option('-k','--key', action="store", nargs='?', dest='key', help='client SSL key file', default='/tmp/ssl/client.cert') parser.add_option('-c','--cert', action="store", nargs='?', dest='cert', help='client SSL certificate file', default='/tmp/ssl/client.cert') parser.add_option('-a','--cacert', action="store", nargs='?', dest='cacert', help='client SSL CA certificate file', default='/tmp/ssl/ca.cert') args = parser.parse_args() if "https://" in args[0].url: client = Client(wsdl,transport = HTTPSClientCertTransport(key,cert,cacert)) else: client = Client(wsdl) # NOTE: the following form to enable attribute additions # is only supported with suds >= 0.4.1 #client = Client(job_wsdl,plugins=[OverridesPlugin()]); client.set_options(location=args[0].url) # enable to see service schema logging.basicConfig(level=logging.INFO) logging.getLogger('suds.client').setLevel(logging.DEBUG) print client # add specific requirements here req1 = client.factory.create("ns0:ResourceConstraint") req1.type = 'OS' req1.value = 'LINUX' reqs = [ req1 ] # add extra Condor-specific or custom job attributes here extra1 = client.factory.create("ns0:Attribute") extra1.name = 'RECIPE' extra1.type = 'STRING' extra1.value = '"SECRET_SAUCE"' extras = [ extra1 ] try: result = client.service.submitJob( \ # the executable command '/bin/sleep', \ # some arguments for the command '120', \ # the submitter name uid, \ # initial working directory wwhere job will execute '/tmp', \ # an arbitrary string identifying the target submission group 'python_test_submit', \ # special resource requirements reqs, \ # additional attributes extras ) except Exception, e: print "invocation failed at: ", args[0].url print e exit(1) if result.status.code != "OK": print result.status.code,"; ", result.status.text exit(1) print result aviary log: [Thu Oct 13 09:45:25 2011] [debug] class_loader.c(131) /usr/lib/libwsf_cpp_msg_recv.so.0 shared lib loaded successfully [Thu Oct 13 09:45:25 2011] [debug] class_loader.c(131) /usr/lib/libwsf_cpp_msg_recv.so.0 shared lib loaded successfully [Thu Oct 13 09:45:25 2011] [debug] class_loader.c(131) /usr/lib/libwsf_cpp_msg_recv.so.0 shared lib loaded successfully [Thu Oct 13 09:45:25 2011] [debug] svc_builder.c(295) DLL path is : /var/lib/condor/aviary/services/query/libaviary_query_axis.so [Thu Oct 13 09:45:25 2011] [debug] class_loader.c(131) /usr/lib/libwsf_cpp_msg_recv.so.0 shared lib loaded successfully [Thu Oct 13 09:45:25 2011] [debug] class_loader.c(131) /usr/lib/libwsf_cpp_msg_recv.so.0 shared lib loaded successfully [Thu Oct 13 09:45:25 2011] [debug] class_loader.c(131) /usr/lib/libwsf_cpp_msg_recv.so.0 shared lib loaded successfully [Thu Oct 13 09:45:25 2011] [debug] class_loader.c(131) /usr/lib/libwsf_cpp_msg_recv.so.0 shared lib loaded successfully [Thu Oct 13 09:45:25 2011] [debug] class_loader.c(131) /usr/lib/libwsf_cpp_msg_recv.so.0 shared lib loaded successfully [Thu Oct 13 09:45:25 2011] [debug] dep_engine.c(1029) No modules configured [Thu Oct 13 09:46:32 2011] [error] /builddir/build/BUILD/condor-7.6.3/src/condor_contrib/aviary/src/axis2_ssl_utils.c(195) [ssl] SSL_accept failed = 0 client log: DEBUG:suds.client:sending to (https://_host_:9090/services/job/submitJob) message: <?xml version="1.0" encoding="UTF-8"?> <SOAP-ENV:Envelope xmlns:ns0="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns1="http://job.aviary.grid.redhat.com" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"> <SOAP-ENV:Header/> <ns0:Body> <ns1:SubmitJob> <cmd>/bin/sleep</cmd> <args>120</args> <owner>root</owner> <iwd>/tmp</iwd> <submission_name>python_test_submit</submission_name> <requirements> <type>OS</type> <value>LINUX</value> </requirements> <extra> <name>RECIPE</name> <type>STRING</type> <value>"SECRET_SAUCE"</value> </extra> </ns1:SubmitJob> </ns0:Body> </SOAP-ENV:Envelope> DEBUG:suds.client:headers = {'SOAPAction': '"http://grid.redhat.com/aviary-job/submit"', 'Content-Type': 'text/xml; charset=utf-8'} invocation failed at: https://host:9090/services/job/submitJob <urlopen error [Errno 336265218] _ssl.c:339: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib> Example of format(pem) of server certificate: Bag Attributes friendlyName: serv_host_ localKeyID: E2 2A 96 FA 37 F5 EA 59 1B 24 C8 E8 17 36 0C 6C 14 13 86 16 Key Attributes: <No Attributes> -----BEGIN PRIVATE KEY----- MIICdgIBADANBgkqhkiG9w0BAQEFAASCAmAwggJcAgEAAoGBAKblNcQtOHJ3f6dz ... vp7f6W8boo7s1A== -----END PRIVATE KEY----- Bag Attributes friendlyName: serv_host_ localKeyID: E2 2A 96 FA 37 F5 EA 59 1B 24 C8 E8 17 36 0C 6C 14 13 86 16 subject=/CN=_host_ issuer=/CN=CAcert -----BEGIN CERTIFICATE----- MIIBsTCCARqgAwIBAgICA+kwDQYJKoZIhvcNAQEFBQAwETEPMA0GA1UEAxMGQ0Fj ... be8t8q4= -----END CERTIFICATE-----
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: Cause: Invoking an SSL connection using Aviary with just a CA file. Consequence: SSL handshake fails since the local issuer of the client can't be resolved. Fix: Corrected a code path to ensure that both a CA file and a CA dir are tried in the OpenSSL code within Aviary servers. Result: Client can authenticate and establish secure connection over SSL when CA is set as a file.
Tested on RHEL 5.7/6.1 x x86_64/i386 with condor-aviary-7.6.5-0.2 and it works. -->VERIFIED Configuration of certificates is above. I've used stunnel to create authenticated ssl connection to aviary and I've tested this with official examples from condor-aviary package.