The python suds stack has some limitations with respect to server certificate verification as noted here: http://stackoverflow.com/questions/6167148/drop-in-replacement-for-urllib2-urlopen-that-does-cert-verification We should investigate and make sure that is not a gating problem or that another python library such as soappy is a better alternative.
Update, I've found an example posted in which the developer rolled a server certificate check via a new handler class passed to urllib2.build_opener. We could probably do something similar, seeing as we're already using build_opener in the ssl interface for aviary. Folks also use pycurl to do this, haven't looked there yet but pycurl is bundled in el5, el6, and Fedora. That is all....
Note, also looks like m2crypto is available on el[5|6] and Fedora. Could be something here.
Found a cool article explaining how to use the Python ssl module (standard in 2.6, has been ported back to 2.3.5+ but not officially) to extend urllib2 to do server certificate validation. This gets us very, very close to a solution since SUDS is running on top of urllib2. Pete's https.py already does things similar to the code in the article. However, we have the issue of the ssl module not being available in Python 2.4 on RHEL 5 (we have 2.6 on RHEL 6) The python-ssl package has even been built in brew for el5, but for internal use only. http://www.muchtooscrawled.com/2010/03/https-certificate-verification-in-python-with-urllib2/
Resolved in revision 4957 for cumin. For RHEL 6, there are no additional dependencies. The solution uses the Python 2.6 standard language module 'ssl'. As noted above, on RHEL 5 the ssl module is not available natively (Python 2.4). If we can get the ssl module supported on RHEL 5, we will have server certificate validation on that platform. In general, cumin will fall back on client only validation if the import of the ssl module fails. An entry in the cumin log files indicates what type of ssl is in use.
Update on this, looking into the possiblity of using M2Crypto in the class derived from httplib.HTTPConnection which is the key to this solution. Using M2Crypto here to do client and server cert validation would remove the dependency on the Python ssl module, which would allow full functionality on RHEL 5 without supporting the backport of ssl. OpenSSL and M2Cryptor are already both supported on RHEL 5.
Note, we have a solution working with M2Crypto on el5. Need to test for memory leaks, re previous posts/BZs concerning leaks in M2Crypto.
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: Cause Integration with Aviary called for secure communication over SSL with the Aviary server. The existing out of the box solution provided only for client certificate validation. Consequence Server certificate validation was not supported. Change Cumin has been extended to use either the Python ssl module or M2Crypto to supply server certificate validation. Result Communication with Aviary over SSL will perform both client and server certificate validation.
Technical note updated. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1,3 +1,9 @@ +Note, in hindsight this Tech Note can probably be combined with that for BZ733677 which will describe the integration of Aviary. + +The only really valuable content in the note below is that Cumin will do server certificate authentication over ssl for Aviary. + +-- + Cause Integration with Aviary called for secure communication over SSL with the Aviary server. The existing out of the box solution provided only for client certificate validation.
There will be logging entries in web.log (info level) that indicate what type of communication Cumin will use with Aviary, including which technology (Python ssl module or M2Crypto) is used and whether server certificate validation is in use. If Python ssl is available, it will be used first. If not, M2Crypto will be tried. finally, if neither is available client certificate validation only will be used. (In reply to comment #6) > Note, we have a solution working with M2Crypto on el5. Need to test for memory > leaks, re previous posts/BZs concerning leaks in M2Crypto.
Technical note updated. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1,17 +1,14 @@ -Note, in hindsight this Tech Note can probably be combined with that for BZ733677 which will describe the integration of Aviary. - -The only really valuable content in the note below is that Cumin will do server certificate authentication over ssl for Aviary. - +Added more content. This Tech Note may still be a candidate for combining with content for BZ733677 (forthcoming) -- Cause - Integration with Aviary called for secure communication over SSL with the Aviary server. The existing out of the box solution provided only for client certificate validation. + Integration with Aviary called for secure communication over SSL with the Aviary server. Native httplib classes in Python provided only for client certificate validation. Consequence - Server certificate validation was not supported. + Server certificate validation was not supported (although client certificate validation was). Change - Cumin has been extended to use either the Python ssl module or M2Crypto to supply server certificate validation. + Cumin has been extended to use either the Python ssl module or M2Crypto to supply server certificate validation. Cumin will prefer Python ssl over M2Crypto if it is available, otherwise it will try M2Crypto. If neither is available, server certificate validation will not be used. Cumin-web will write log entries beginning with "AviaryOperations:" that will indicate what type of communication is being used for Aviary. Result - Communication with Aviary over SSL will perform both client and server certificate validation.+ Communication with Aviary over SSL will perform client certificate validation and will also perform server certificate validation when possible, informing the user of the mode of communication via log entries.
General steps for test/verification 1) Set up condor-aviary and cumin for communication over ssl. For condor-aviary, ssl is off by default. It can be turned on in the /etc/condor/config.d/61aviary.config file. For cumin, set aviary-job-servers, aviary-query-servers, aviary-key, and aviary-cert. (This should be like the setup done for verification of BZ733677 I am guessing). 2) Verify that everything is working by doing a submission, drilling into submissions for job summaries, etc. 3) To turn on server certificate validation in cumin, set the aviary-root-cert parameter in cumin.conf to the path of a file containing a CA cert. This can be a certificate chain from an authentic certificate authority if "real" certificates are being used, but if self-signed certificates are being used this can simply be a copy of the server certificate (this is what I did for development, I never had "real" certificates) 4) Restart cumin and check /var/log/cumin/web.log for a log entry that says "AviaryOperations: using client and server certificate validation for ssl connections, technology is ...". On RHEL 5, the tecnhology will be M2Crypto. On RHEL 6, the technology is Python ssl. 5) Try aviary operations, like drilling into a submission for a job list, drilling into job details, submitting jobs, etc. It should work. 6) If a self-signed server certificate was generated, and the common name in the certificate does not match the hostname of the server, domain verification can be tested, too. Set "aviary-domain-verify: True" in the cumin.conf file and restart cumin. Aviary operations like those in #5 above should now fail with a banner that says "Server certificate doesn't match domain; untrusted connection". If the common name in the certificate matches the hostname, the operation will succeed and you will not see this banner. 7) To make aviary operations fail server certificate validation, simply copy over the file pointed to by aviary-root-cert with another certificate. Any other certificate will do, even the client certificate (just something that guarantees it will not match the information sent by the server). Restart cumin (not sure if we really need a restart here, but just to be safe...). Aviary operations like those in #5 should now fail with a message from the certificate validation layer.
Testing on RHEL5/6, i386/x86_64 RHEL5 packages: condor-7.6.5-0.7.el5 condor-aviary-7.6.5-0.7.el5 cumin-0.1.5098-2.el5 python-suds-0.4.1-2.el5 RHEL6 packages: condor-7.6.5-0.7.el6 condor-aviary-7.6.5-0.7.el6 cumin-0.1.5098-2.el6 python-suds-0.4.1-3.el6 [client validation only] ======================== 1) Set up condor-aviary and cumin for communication over ssl. For condor-aviary, ssl is off by default. It can be turned on in the /etc/condor/config.d/61aviary.config file. For cumin, set aviary-job-servers, aviary-query-servers, aviary-key, and aviary-cert. (This should be like the setup done for verification of BZ733677 I am guessing). 2) Verify that everything is working by doing a submission, drilling into submissions for job summaries, etc. - cumin.conf: log-level: debug aviary-job-servers: https://localhost:9090 aviary-query-servers: https://localhost:9091 aviary-key: /tmp/ssl/client.pem aviary-cert: /tmp/ssl/client.pem #aviary-root-cert: /tmp/ssl/ca.pem aviary-suds-logs: True - web.log: INFO AviaryOperations: no root certificate file specified, using client validation only for ssl connections. INFO Enabled Aviary interface for job submission and control. INFO Enabled Aviary interface for query operations. - submitjob: Submission: "zzz" Cmd: "/bin/sleep 360" Requirements: 'Memory >= 32 && OpSys == "LINUX" && Arch =="X86_64"' Working Directory: "/tmp" - after job is submitted - look at details of this job - verify log files - no qmf methods used # grep -i method /var/log/cumin/web.log* ecode=1 - grep zzz /var/log/cumin/suds.client.log <submission_name>zzz</submission_name> <SOAP-ENV:Envelope xmlns:SOAP-ENV="...> <name>zzz</name> <SOAP-ENV:Envelope xmlns:SOAP-ENV="...>
[client and server certificate validation] ========================================== 3) To turn on server certificate validation in cumin, set the aviary-root-cert parameter in cumin.conf to the path of a file containing a CA cert. This can be a certificate chain from an authentic certificate authority if "real" certificates are being used, but if self-signed certificates are being used this can simply be a copy of the server certificate (this is what I did for development, I never had "real" certificates) 4) Restart cumin and check /var/log/cumin/web.log for a log entry that says "AviaryOperations: using client and server certificate validation for ssl connections, technology is ...". On RHEL 5, the tecnhology will be M2Crypto. On RHEL 6, the technology is Python ssl. 5) Try aviary operations, like drilling into a submission for a job list, drilling into job details, submitting jobs, etc. It should work. Bug 733677, comment 17
comment 12 > 6) If a self-signed server certificate was generated,... Can you provide example for generating proper self-signed certificates and how should relevant cumin.conf part look like?
Technical note updated. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1,14 +1 @@ -Added more content. This Tech Note may still be a candidate for combining with content for BZ733677 (forthcoming) +Native httplib classes in Python provided support only for client certificate validation; server certificate validation was not supported. Consequently, the Cumin web console could not communicate securely over SSL with Aviary servers. With this update, Cumin has been enhanced to use either the Python SSL module or the M2Crypto SSL toolkit to supply server certificate validation. The cumin-web utility now writes log entries beginning with "AviaryOperations:", which indicates what type of communication is being used for Aviary.--- - -Cause - Integration with Aviary called for secure communication over SSL with the Aviary server. Native httplib classes in Python provided only for client certificate validation. - -Consequence - Server certificate validation was not supported (although client certificate validation was). - -Change - Cumin has been extended to use either the Python ssl module or M2Crypto to supply server certificate validation. Cumin will prefer Python ssl over M2Crypto if it is available, otherwise it will try M2Crypto. If neither is available, server certificate validation will not be used. Cumin-web will write log entries beginning with "AviaryOperations:" that will indicate what type of communication is being used for Aviary. - -Result - Communication with Aviary over SSL will perform client certificate validation and will also perform server certificate validation when possible, informing the user of the mode of communication via log entries.
(In reply to comment #15) > comment 12 > > 6) If a self-signed server certificate was generated,... > > Can you provide example for generating proper self-signed certificates and how > should relevant cumin.conf part look like? Here is the procedure I used to test on my VM, maybe not what you would in a "production" environment but valid I think. Note, I just used the server.crt itself as the root-cert file for cumin (simple) but you can also append it to the ca-bundle (shown below). Overview: Create a client key and cert, put them somewhere cumin can use them. Copy the client cert to /etc/pki/tls/certs and run a script to link it to the ca bundle. Generate a server.key and server.crt in /etc/pki/tls/certs where aviary will look for them. Configure cumin.conf to use the client key, client cert, and a copy of the server cert as the aviary-root-cert. Depending on what you define for common name, you may have to turn off domain verification. Put this bash script in /etc/pki/tls/certs/certlink.sh first... #!/bin/sh # # usage: certlink.sh filename [filename ...] for CERTFILE in $*; do # make sure file exists and is a valid cert test -f "$CERTFILE" || continue HASH=$(openssl x509 -noout -hash -in "$CERTFILE") test -n "$HASH" || continue # use lowest available iterator for symlink for ITER in 0 1 2 3 4 5 6 7 8 9; do test -f "${HASH}.${ITER}" && continue ln -s "$CERTFILE" "${HASH}.${ITER}" test -L "${HASH}.${ITER}" && break done done # openssl genrsa -out tmckay.key 2048 # openssl req -new -x509 -key tmckay.key -out tmckay.crt # cp tmckay.key tmckay.crt /etc/cumin/ # cp tmckay.crt /etc/pki/tls/certs # rm tmckay.crt tmckay.key # cd /etc/pki/tls/certs # ./certlink.sh tmckay.crt # openssl genrsa -out server.key 2048 # openssl req -new -x509 -key server.key -out server.crt To put the server certificate in the ca-bundle # openssl x509 -in server.crt -text >> ca-bundle.crt Set the following in cumin.conf aviary-job-servers: https://localhost:9090 aviary-query-servers: https://localhost:9091 aviary-key: /etc/cumin/tmckay.key aviary-cert: /etc/cumin/tmckay.crt aviary-root-cert: /etc/pki/tls/certs/server.crt aviary-domain-verify: False (aviary-root-cert could be /etc/pki/tls/certs/ca-bundle.crt if you did the import of the server.crt) That's it!
Left out the part where you delete the server private key, or store it away somewhere safe :)
This example doesn't work in my case - Bug 754988
What I found playing around... Apparently, there can be trouble when the client cert and the server cert have the same hash (which is calculated from the subject name in the certificate) and the server cert is appended to the ca-bundle.crt file. I'm guessing the server looks in the bundle first when verifying the client cert, finds a cert in the bundle with a matching hash but different content, and throws an error. This is empirical... When I tested this originally, I was using certs with different subject names and therefore different hashes. So, do not append anything to the ca-bundle.crt file, and set cumin to use the server.crt directly as the root cert. aviary-root-cert: /etc/pki/tls/certs/server.crt This should work.
Alternatively, if you want to point cumin at a ca-bundle.crt file and you have this problem with matching hashes, you can do this: 1) leave /etc/pki/tls/certs/ca-bundle.crt unchanged 2) copy /etc/pki/tls/certs/ca-bundle.crt to /etc/cumin 3) append the server.crt to /etc/cumin/ca-bundle.crt 4) set cumin's aviary-root-cert to /etc/cumin/ca-bundle.crt Note, this is only a problem when using self-signed certificates with matching hashes on the same machine.
[Self-signed certificate] ========================= 6) If a self-signed server certificate was generated, and the common name in the certificate does not match the hostname of the server, domain verification can be tested, too. Set "aviary-domain-verify: True" in the cumin.conf file and restart cumin. Aviary operations like those in #5 above should now fail with a banner that says "Server certificate doesn't match domain; untrusted connection". If the common name in the certificate matches the hostname, the operation will succeed and you will not see this banner. ~]# vi /etc/pki/tls/certs/certlink.sh ~]# openssl genrsa -out sgraf.key 2048 ~]# openssl req -new -x509 -key sgraf.key -out sgraf.crt ~]# cp sgraf.key sgraf.crt /etc/cumin/ ~]# cp sgraf.crt /etc/pki/tls/certs ~]# cd /etc/pki/tls/certs /etc/pki/tls/certs]# openssl genrsa -out server.key 2048 /etc/pki/tls/certs]# openssl req -new -x509 -key server.key -out server.crt /etc/pki/tls/certs]# sh certlink.sh sgraf.crt server.crt ca-bundle.crt condor-aviary: SCHEDD.AVIARY_SSL = True SCHEDD.AVIARY_SSL_SERVER_CERT = /etc/pki/tls/certs/server.crt SCHEDD.AVIARY_SSL_SERVER_KEY = /etc/pki/tls/certs/server.key SCHEDD.AVIARY_SSL_CA_DIR = /etc/pki/tls/certs SCHEDD.AVIARY_SSL_CA_FILE = /etc/pki/tls/certs/ca-bundle.crt QUERY_SERVER.AVIARY_SSL = True QUERY_SERVER.AVIARY_SSL_SERVER_CERT = /etc/pki/tls/certs/server.crt QUERY_SERVER.AVIARY_SSL_SERVER_KEY = /etc/pki/tls/certs/server.key QUERY_SERVER.AVIARY_SSL_CA_DIR = /etc/pki/tls/certs QUERY_SERVER.AVIARY_SSL_CA_FILE = /etc/pki/tls/certs/ca-bundle.crt cumin.conf: aviary-job-servers: https://localhost:9090 aviary-query-servers: https://localhost:9091 aviary-key: /etc/cumin/sgraf.key aviary-cert: /etc/cumin/sgraf.crt aviary-root-cert: /etc/pki/tls/certs/server.crt aviary-domain-verify: True RHEL5 web.log 6155 2011-11-20 15:20:38,030 INFO AviaryOperations: using client and server certificate validation for ssl connections, solution is M2Crypto 6155 2011-11-20 15:20:38,031 INFO AviaryOperations: verify server domain against certificate during validation (True) 6155 2011-11-20 15:20:38,033 INFO Enabled Aviary interface for job submission and control. 6155 2011-11-20 15:20:38,034 INFO Enabled Aviary interface for query operations. RHEL6 web.log 15922 2011-11-20 15:20:36,211 INFO AviaryOperations: using client and server certificate validation for ssl connections, solution is Python ssl 15922 2011-11-20 15:20:36,211 INFO AviaryOperations: verify server domain against certificate during validation (True) 15922 2011-11-20 15:20:36,215 INFO Enabled Aviary interface for job submission and control. 15922 2011-11-20 15:20:36,215 INFO Enabled Aviary interface for query operations. - submitjob: Submission: "zzz" Cmd: "/bin/sleep 360" Requirements: 'Memory >= 32 && OpSys == "LINUX" && Arch =="X86_64"' Working Directory: "/tmp" - after job is submitted - look at details of this job Regenerate server certificate (put bad hostname): ================================================= ~]# cd /etc/pki/tls/certs /etc/pki/tls/certs]# rm -f server.key server.crt /etc/pki/tls/certs]# openssl genrsa -out server.key 2048 /etc/pki/tls/certs]# openssl req -new -x509 -key server.key -out server.crt /etc/pki/tls/certs]# sh certlink.sh server.crt ca-bundle.crt - submitjob: Submission: "zzz" Cmd: "/bin/sleep 360" Requirements: 'Memory >= 32 && OpSys == "LINUX" && Arch =="X86_64"' Working Directory: "/tmp" Result: Submit job 'zzz': Failed (Server certificate doesn't match domain; untrusted connection) Try aviary-domain-verify: False =============================== Let's have server certificate with bad hostname and set in cumin.conf: aviary-domain-verify: False - submitjob: Submission: "zzz" Cmd: "/bin/sleep 360" Requirements: 'Memory >= 32 && OpSys == "LINUX" && Arch =="X86_64"' Working Directory: "/tmp" - after job is submitted - look at details of this job
7) To make aviary operations fail server certificate validation, simply copy over the file pointed to by aviary-root-cert with another certificate. Any other certificate will do, even the client certificate (just something that guarantees it will not match the information sent by the server). Restart cumin (not sure if we really need a restart here, but just to be safe...). Aviary operations like those in #5 should now fail with a message from the certificate validation layer. Use setup from Comment 22 In cumin.conf change from: aviary-root-cert: /etc/pki/tls/certs/server.crt to aviary-root-cert: /etc/cumin/sgraf.crt (client cert) Try to submit job: RHEL5: Submit job 'zzz': Failed (certificate verify failed) RHEL6: Submit job 'zzz': Failed (Trouble reaching host [hostname], [Errno 1] _ssl.c:490: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed)
Verified on RHEL5/6 i386/x86_64 RHEL5 packages: condor-7.6.5-0.7.el5 condor-aviary-7.6.5-0.7.el5 cumin-0.1.5098-2.el5 python-suds-0.4.1-2.el5 RHEL6 packages: condor-7.6.5-0.7.el6 condor-aviary-7.6.5-0.7.el6 cumin-0.1.5098-2.el6 python-suds-0.4.1-3.el6
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2012-0045.html