Bug 733447 - cumin deps for integration for Aviary with SSL
Summary: cumin deps for integration for Aviary with SSL
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: cumin
Version: Development
Hardware: Unspecified
OS: Unspecified
medium
unspecified
Target Milestone: 2.1
: ---
Assignee: Trevor McKay
QA Contact: Stanislav Graf
URL:
Whiteboard:
Depends On: 694612
Blocks: 743350
TreeView+ depends on / blocked
 
Reported: 2011-08-25 18:04 UTC by Pete MacKinnon
Modified: 2012-02-08 10:45 UTC (History)
6 users (show)

Fixed In Version: cumin-0.1.5068-1
Doc Type: Bug Fix
Doc Text:
Native httplib classes in Python provided support only for client certificate validation; server certificate validation was not supported. Consequently, the Cumin web console could not communicate securely over SSL with Aviary servers. With this update, Cumin has been enhanced to use either the Python SSL module or the M2Crypto SSL toolkit to supply server certificate validation. The cumin-web utility now writes log entries beginning with "AviaryOperations:", which indicates what type of communication is being used for Aviary.
Clone Of:
Environment:
Last Closed: 2012-01-23 17:28:38 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2012:0045 normal SHIPPED_LIVE Red Hat Enterprise MRG Grid 2.1 bug fix and enhancement update 2012-01-23 22:22:58 UTC
Red Hat Bugzilla 754988 None None None Never

Internal Links: 754988 790863

Description Pete MacKinnon 2011-08-25 18:04:10 UTC
The python suds stack has some limitations with respect to server certificate verification as noted here:
http://stackoverflow.com/questions/6167148/drop-in-replacement-for-urllib2-urlopen-that-does-cert-verification

We should investigate and make sure that is not a gating problem or that another python library such as soappy is a better alternative.

Comment 1 Trevor McKay 2011-09-01 18:56:31 UTC
Update, I've found an example posted in which the developer rolled a server certificate check via a new handler class passed to urllib2.build_opener.  We could probably do something similar, seeing as we're already using build_opener in the ssl interface for aviary.

Folks also use pycurl to do this, haven't looked there yet but pycurl is bundled in el5, el6, and Fedora.  That is all....

Comment 2 Trevor McKay 2011-09-01 19:10:02 UTC
Note, also looks like m2crypto is available on el[5|6] and Fedora.  Could be something here.

Comment 3 Trevor McKay 2011-09-02 19:20:22 UTC
Found a cool article explaining how to use the Python ssl module (standard in 2.6, has been ported back to 2.3.5+ but not officially) to extend urllib2 to do server certificate validation.

This gets us very, very close to a solution since SUDS is running on top of urllib2.  Pete's https.py already does things similar to the code in the article.  However, we have the issue of the ssl module not being available in Python 2.4 on RHEL 5 (we have 2.6 on RHEL 6)

The python-ssl package has even been built in brew for el5, but for internal use only.

http://www.muchtooscrawled.com/2010/03/https-certificate-verification-in-python-with-urllib2/

Comment 4 Trevor McKay 2011-09-16 14:29:46 UTC
Resolved in revision 4957 for cumin.

For RHEL 6, there are no additional dependencies.  The solution uses the Python 2.6 standard language module 'ssl'.

As noted above, on RHEL 5 the ssl module is not available natively (Python 2.4).  If we can get the ssl module supported on RHEL 5, we will have server certificate validation on that platform.

In general, cumin will fall back on client only validation if the import of the ssl module fails.  An entry in the cumin log files indicates what type of ssl is in use.

Comment 5 Trevor McKay 2011-09-22 16:13:21 UTC
Update on this, looking into the possiblity of using M2Crypto in the class derived from httplib.HTTPConnection which is the key to this solution.  

Using M2Crypto here to do client and server cert validation would remove the dependency on the Python ssl module, which would allow full functionality on RHEL 5 without supporting the backport of ssl.  OpenSSL and M2Cryptor are already both supported on RHEL 5.

Comment 6 Trevor McKay 2011-10-04 21:12:34 UTC
Note, we have a solution working with M2Crypto on el5.  Need to test for memory leaks, re previous posts/BZs concerning leaks in M2Crypto.

Comment 7 Trevor McKay 2011-10-04 21:12:34 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause
    Integration with Aviary called for secure communication over SSL with the Aviary server.  The existing out of the box solution provided only for client certificate validation.

Consequence
    Server certificate validation was not supported.

Change
    Cumin has been extended to use either the Python ssl module or M2Crypto to supply server certificate validation.
  
Result
    Communication with Aviary over SSL will perform both client and server certificate validation.

Comment 8 Trevor McKay 2011-10-04 22:36:39 UTC
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,3 +1,9 @@
+Note, in hindsight this Tech Note can probably be combined with that for BZ733677 which will describe the integration of Aviary.  
+
+The only really valuable content in the note below is that Cumin will do server certificate authentication over ssl for Aviary.
+
+-- 
+
 Cause
     Integration with Aviary called for secure communication over SSL with the Aviary server.  The existing out of the box solution provided only for client certificate validation.

Comment 10 Trevor McKay 2011-10-05 13:08:55 UTC
There will be logging entries in web.log (info level) that indicate what type of communication Cumin will use with Aviary, including which technology (Python ssl module or M2Crypto) is used and whether server certificate validation is in use.

If Python ssl is available, it will be used first.  If not, M2Crypto will be tried.  finally, if neither is available client certificate validation only will be used.

(In reply to comment #6)
> Note, we have a solution working with M2Crypto on el5.  Need to test for memory
> leaks, re previous posts/BZs concerning leaks in M2Crypto.

Comment 11 Trevor McKay 2011-10-05 13:08:55 UTC
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,17 +1,14 @@
-Note, in hindsight this Tech Note can probably be combined with that for BZ733677 which will describe the integration of Aviary.  
-
-The only really valuable content in the note below is that Cumin will do server certificate authentication over ssl for Aviary.
-
+Added more content.  This Tech Note may still be a candidate for combining with content for BZ733677 (forthcoming)
 -- 
 
 Cause
-    Integration with Aviary called for secure communication over SSL with the Aviary server.  The existing out of the box solution provided only for client certificate validation.
+    Integration with Aviary called for secure communication over SSL with the Aviary server.  Native httplib classes in Python provided only for client certificate validation.
 
 Consequence
-    Server certificate validation was not supported.
+    Server certificate validation was not supported (although client certificate validation was).
 
 Change
-    Cumin has been extended to use either the Python ssl module or M2Crypto to supply server certificate validation.
+    Cumin has been extended to use either the Python ssl module or M2Crypto to supply server certificate validation.  Cumin will prefer Python ssl over M2Crypto if it is available, otherwise it will try M2Crypto.  If neither is available, server certificate validation will not be used.  Cumin-web will write log entries beginning with "AviaryOperations:" that will indicate what type of communication is being used for Aviary.  
   
 Result
-    Communication with Aviary over SSL will perform both client and server certificate validation.+    Communication with Aviary over SSL will perform client certificate validation and will also perform server certificate validation when possible, informing the user of the mode of communication via log entries.

Comment 12 Trevor McKay 2011-11-10 19:34:06 UTC
General steps for test/verification

1) Set up condor-aviary and cumin for communication over ssl.  For condor-aviary, ssl is off by default.  It can be turned on in the /etc/condor/config.d/61aviary.config file.  For cumin, set aviary-job-servers, aviary-query-servers, aviary-key, and aviary-cert.  (This should be like the setup done for verification of BZ733677 I am guessing).

2) Verify that everything is working by doing a submission, drilling into submissions for job summaries, etc.

3) To turn on server certificate validation in cumin, set the aviary-root-cert parameter in cumin.conf to the path of a file containing a CA cert.  This can be a certificate chain from an authentic certificate authority if "real" certificates are being used, but if self-signed certificates are being used this can simply be a copy of the server certificate (this is what I did for development, I never had "real" certificates)

4) Restart cumin and check /var/log/cumin/web.log for a log entry that says "AviaryOperations: using client and server certificate validation for ssl connections, technology is ...".  On RHEL 5, the tecnhology will be M2Crypto.  On RHEL 6, the technology is Python ssl.

5) Try aviary operations, like drilling into a submission for a job list, drilling into job details, submitting jobs, etc.  It should work.

6) If a self-signed server certificate was generated, and the common name in the certificate does not match the hostname of the server, domain verification can be tested, too.  Set "aviary-domain-verify: True" in the cumin.conf file and restart cumin.  Aviary operations like those in #5 above should now fail with a banner that says "Server certificate doesn't match domain; untrusted connection".  If the common name in the certificate matches the hostname, the operation will succeed and you will not see this banner.

7) To make aviary operations fail server certificate validation, simply copy over the file pointed to by aviary-root-cert with another certificate.  Any other certificate will do, even the client certificate (just something that guarantees it will not match the information sent by the server).  Restart cumin (not sure if we really need a restart here, but just to be safe...).  Aviary operations like those in #5 should now fail with a message from the certificate validation layer.

Comment 13 Stanislav Graf 2011-11-15 12:12:22 UTC
Testing on RHEL5/6, i386/x86_64

RHEL5 packages:
condor-7.6.5-0.7.el5
condor-aviary-7.6.5-0.7.el5
cumin-0.1.5098-2.el5
python-suds-0.4.1-2.el5

RHEL6 packages:
condor-7.6.5-0.7.el6
condor-aviary-7.6.5-0.7.el6
cumin-0.1.5098-2.el6
python-suds-0.4.1-3.el6

[client validation only]
========================
1) Set up condor-aviary and cumin for communication over ssl.  For
condor-aviary, ssl is off by default.  It can be turned on in the
/etc/condor/config.d/61aviary.config file.  For cumin, set aviary-job-servers,
aviary-query-servers, aviary-key, and aviary-cert.  (This should be like the
setup done for verification of BZ733677 I am guessing).
2) Verify that everything is working by doing a submission, drilling into
submissions for job summaries, etc.

- cumin.conf:
log-level: debug
aviary-job-servers: https://localhost:9090
aviary-query-servers: https://localhost:9091
aviary-key: /tmp/ssl/client.pem
aviary-cert: /tmp/ssl/client.pem
#aviary-root-cert: /tmp/ssl/ca.pem
aviary-suds-logs: True

- web.log:
INFO AviaryOperations: no root certificate file specified, using client validation only for ssl connections.
INFO Enabled Aviary interface for job submission and control.
INFO Enabled Aviary interface for query operations.

- submitjob:
Submission: "zzz"
Cmd: "/bin/sleep 360"
Requirements: 'Memory >= 32 && OpSys == "LINUX" && Arch =="X86_64"'
Working Directory: "/tmp"
- after job is submitted - look at details of this job

- verify log files
- no qmf methods used
# grep -i method /var/log/cumin/web.log*
ecode=1

- grep zzz /var/log/cumin/suds.client.log 
         <submission_name>zzz</submission_name> <SOAP-ENV:Envelope
xmlns:SOAP-ENV="...>
         <name>zzz</name> <SOAP-ENV:Envelope xmlns:SOAP-ENV="...>

Comment 14 Stanislav Graf 2011-11-15 12:12:45 UTC
[client and server certificate validation]
==========================================
3) To turn on server certificate validation in cumin, set the aviary-root-cert
parameter in cumin.conf to the path of a file containing a CA cert.  This can
be a certificate chain from an authentic certificate authority if "real"
certificates are being used, but if self-signed certificates are being used
this can simply be a copy of the server certificate (this is what I did for
development, I never had "real" certificates)
4) Restart cumin and check /var/log/cumin/web.log for a log entry that says
"AviaryOperations: using client and server certificate validation for ssl
connections, technology is ...".  On RHEL 5, the tecnhology will be M2Crypto. 
On RHEL 6, the technology is Python ssl.
5) Try aviary operations, like drilling into a submission for a job list,
drilling into job details, submitting jobs, etc.  It should work.

Bug 733677, comment 17

Comment 15 Stanislav Graf 2011-11-16 15:22:20 UTC
comment 12
> 6) If a self-signed server certificate was generated,...

Can you provide example for generating proper self-signed certificates and how should relevant cumin.conf part look like?

Comment 16 Tomas Capek 2011-11-16 15:37:34 UTC
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,14 +1 @@
-Added more content.  This Tech Note may still be a candidate for combining with content for BZ733677 (forthcoming)
+Native httplib classes in Python provided support only for client certificate validation; server certificate validation was not supported. Consequently, the Cumin web console could not communicate securely over SSL with Aviary servers. With this update, Cumin has been enhanced to use either the Python SSL module or the M2Crypto SSL toolkit to supply server certificate validation. The cumin-web utility now writes log entries beginning with "AviaryOperations:", which indicates what type of communication is being used for Aviary.--- 
-
-Cause
-    Integration with Aviary called for secure communication over SSL with the Aviary server.  Native httplib classes in Python provided only for client certificate validation.
-
-Consequence
-    Server certificate validation was not supported (although client certificate validation was).
-
-Change
-    Cumin has been extended to use either the Python ssl module or M2Crypto to supply server certificate validation.  Cumin will prefer Python ssl over M2Crypto if it is available, otherwise it will try M2Crypto.  If neither is available, server certificate validation will not be used.  Cumin-web will write log entries beginning with "AviaryOperations:" that will indicate what type of communication is being used for Aviary.  
-  
-Result
-    Communication with Aviary over SSL will perform client certificate validation and will also perform server certificate validation when possible, informing the user of the mode of communication via log entries.

Comment 17 Trevor McKay 2011-11-16 23:10:19 UTC
(In reply to comment #15)
> comment 12
> > 6) If a self-signed server certificate was generated,...
> 
> Can you provide example for generating proper self-signed certificates and how
> should relevant cumin.conf part look like?

Here is the procedure I used to test on my VM, maybe not what you would in a "production" environment but valid I think.  Note, I just used the server.crt itself as the root-cert file for cumin (simple) but you can also append it to the ca-bundle (shown below).  

Overview:
Create a client key and cert, put them somewhere cumin can use them.  Copy the client cert to /etc/pki/tls/certs and run a script to link it to the ca bundle.  Generate a server.key and server.crt in /etc/pki/tls/certs where aviary will look for them.  Configure cumin.conf to use the client key, client cert, and a copy of the server cert as the aviary-root-cert.  Depending on what you define for common name, you may have to turn off domain verification.

Put this bash script in /etc/pki/tls/certs/certlink.sh first...

#!/bin/sh
#
# usage: certlink.sh filename [filename ...]

for CERTFILE in $*; do
  # make sure file exists and is a valid cert
  test -f "$CERTFILE" || continue
  HASH=$(openssl x509 -noout -hash -in "$CERTFILE")
  test -n "$HASH" || continue

  # use lowest available iterator for symlink
  for ITER in 0 1 2 3 4 5 6 7 8 9; do
    test -f "${HASH}.${ITER}" && continue
    ln -s "$CERTFILE" "${HASH}.${ITER}"
    test -L "${HASH}.${ITER}" && break
  done
done

# openssl genrsa -out tmckay.key 2048
# openssl req -new -x509 -key tmckay.key -out tmckay.crt
# cp tmckay.key tmckay.crt /etc/cumin/
# cp tmckay.crt /etc/pki/tls/certs
# rm tmckay.crt tmckay.key
# cd /etc/pki/tls/certs
# ./certlink.sh tmckay.crt
# openssl genrsa -out server.key 2048
# openssl req -new -x509 -key server.key -out server.crt

To put the server certificate in the ca-bundle

# openssl x509 -in server.crt -text >> ca-bundle.crt

Set the following in cumin.conf

aviary-job-servers: https://localhost:9090
aviary-query-servers: https://localhost:9091
aviary-key: /etc/cumin/tmckay.key
aviary-cert: /etc/cumin/tmckay.crt
aviary-root-cert: /etc/pki/tls/certs/server.crt
aviary-domain-verify: False

(aviary-root-cert could be /etc/pki/tls/certs/ca-bundle.crt if you did the import of the server.crt)

That's it!

Comment 18 Trevor McKay 2011-11-16 23:12:32 UTC
Left out the part where you delete the server private key, or store it away somewhere safe :)

Comment 19 Stanislav Graf 2011-11-18 13:55:55 UTC
This example doesn't work in my case - Bug 754988

Comment 20 Trevor McKay 2011-11-18 18:32:29 UTC
What I found playing around...

Apparently, there can be trouble when the client cert and the server cert have the same hash (which is calculated from the subject name in the certificate) and the server cert is appended to the ca-bundle.crt file.  I'm guessing the server looks in the bundle first when verifying the client cert, finds a cert in the bundle with a matching hash but different content, and throws an error.  This is empirical...  When I tested this originally, I was using certs with different subject names and therefore different hashes.

So, do not append anything to the ca-bundle.crt file, and set cumin to use the server.crt directly as the root cert.

aviary-root-cert: /etc/pki/tls/certs/server.crt

This should work.

Comment 21 Trevor McKay 2011-11-18 19:18:38 UTC
Alternatively,

  if you want to point cumin at a ca-bundle.crt file and you have this problem with matching hashes, you can do this:

1) leave /etc/pki/tls/certs/ca-bundle.crt unchanged
2) copy /etc/pki/tls/certs/ca-bundle.crt to /etc/cumin
3) append the server.crt to /etc/cumin/ca-bundle.crt
4) set cumin's aviary-root-cert to /etc/cumin/ca-bundle.crt

Note, this is only a problem when using self-signed certificates with matching hashes on the same machine.

Comment 22 Stanislav Graf 2011-11-20 15:01:52 UTC
[Self-signed certificate]
=========================

6) If a self-signed server certificate was generated, and the common name in
the certificate does not match the hostname of the server, domain verification
can be tested, too.  Set "aviary-domain-verify: True" in the cumin.conf file
and restart cumin.  Aviary operations like those in #5 above should now fail
with a banner that says "Server certificate doesn't match domain; untrusted
connection".  If the common name in the certificate matches the hostname, the
operation will succeed and you will not see this banner.

~]# vi /etc/pki/tls/certs/certlink.sh
~]# openssl genrsa -out sgraf.key 2048
~]# openssl req -new -x509 -key sgraf.key -out sgraf.crt
~]# cp sgraf.key sgraf.crt /etc/cumin/
~]# cp sgraf.crt /etc/pki/tls/certs
~]# cd /etc/pki/tls/certs
/etc/pki/tls/certs]# openssl genrsa -out server.key 2048
/etc/pki/tls/certs]# openssl req -new -x509 -key server.key -out server.crt
/etc/pki/tls/certs]# sh certlink.sh sgraf.crt server.crt ca-bundle.crt

condor-aviary:
SCHEDD.AVIARY_SSL = True
SCHEDD.AVIARY_SSL_SERVER_CERT = /etc/pki/tls/certs/server.crt
SCHEDD.AVIARY_SSL_SERVER_KEY = /etc/pki/tls/certs/server.key
SCHEDD.AVIARY_SSL_CA_DIR = /etc/pki/tls/certs
SCHEDD.AVIARY_SSL_CA_FILE = /etc/pki/tls/certs/ca-bundle.crt
QUERY_SERVER.AVIARY_SSL = True
QUERY_SERVER.AVIARY_SSL_SERVER_CERT = /etc/pki/tls/certs/server.crt
QUERY_SERVER.AVIARY_SSL_SERVER_KEY = /etc/pki/tls/certs/server.key
QUERY_SERVER.AVIARY_SSL_CA_DIR = /etc/pki/tls/certs
QUERY_SERVER.AVIARY_SSL_CA_FILE = /etc/pki/tls/certs/ca-bundle.crt

cumin.conf:
aviary-job-servers: https://localhost:9090
aviary-query-servers: https://localhost:9091
aviary-key: /etc/cumin/sgraf.key
aviary-cert: /etc/cumin/sgraf.crt
aviary-root-cert: /etc/pki/tls/certs/server.crt
aviary-domain-verify: True


RHEL5 web.log
6155 2011-11-20 15:20:38,030 INFO AviaryOperations: using client and server certificate validation for ssl connections, solution is M2Crypto
6155 2011-11-20 15:20:38,031 INFO AviaryOperations: verify server domain against certificate during validation (True)
6155 2011-11-20 15:20:38,033 INFO Enabled Aviary interface for job submission and control.
6155 2011-11-20 15:20:38,034 INFO Enabled Aviary interface for query operations.

RHEL6 web.log
15922 2011-11-20 15:20:36,211 INFO AviaryOperations: using client and server certificate validation for ssl connections, solution is Python ssl
15922 2011-11-20 15:20:36,211 INFO AviaryOperations: verify server domain against certificate during validation (True)
15922 2011-11-20 15:20:36,215 INFO Enabled Aviary interface for job submission and control.
15922 2011-11-20 15:20:36,215 INFO Enabled Aviary interface for query operations.

- submitjob:
Submission: "zzz"
Cmd: "/bin/sleep 360"
Requirements: 'Memory >= 32 && OpSys == "LINUX" && Arch =="X86_64"'
Working Directory: "/tmp"
- after job is submitted - look at details of this job

Regenerate server certificate (put bad hostname):
=================================================

~]# cd /etc/pki/tls/certs
/etc/pki/tls/certs]# rm -f server.key server.crt
/etc/pki/tls/certs]# openssl genrsa -out server.key 2048
/etc/pki/tls/certs]# openssl req -new -x509 -key server.key -out server.crt
/etc/pki/tls/certs]# sh certlink.sh server.crt ca-bundle.crt

- submitjob:
Submission: "zzz"
Cmd: "/bin/sleep 360"
Requirements: 'Memory >= 32 && OpSys == "LINUX" && Arch =="X86_64"'
Working Directory: "/tmp"

Result:
Submit job 'zzz': Failed (Server certificate doesn't match domain; untrusted connection)

Try aviary-domain-verify: False
===============================

Let's have server certificate with bad hostname and set in cumin.conf:
aviary-domain-verify: False

- submitjob:
Submission: "zzz"
Cmd: "/bin/sleep 360"
Requirements: 'Memory >= 32 && OpSys == "LINUX" && Arch =="X86_64"'
Working Directory: "/tmp"
- after job is submitted - look at details of this job

Comment 23 Stanislav Graf 2011-11-20 15:18:36 UTC
7) To make aviary operations fail server certificate validation, simply copy
over the file pointed to by aviary-root-cert with another certificate.  Any
other certificate will do, even the client certificate (just something that
guarantees it will not match the information sent by the server).  Restart
cumin (not sure if we really need a restart here, but just to be safe...). 
Aviary operations like those in #5 should now fail with a message from the
certificate validation layer.

Use setup from Comment 22

In cumin.conf change from:
aviary-root-cert: /etc/pki/tls/certs/server.crt
to
aviary-root-cert: /etc/cumin/sgraf.crt
(client cert)

Try to submit job:
RHEL5:
Submit job 'zzz': Failed (certificate verify failed)
RHEL6:
Submit job 'zzz': Failed (Trouble reaching host [hostname], [Errno 1] _ssl.c:490: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed)

Comment 24 Stanislav Graf 2011-11-20 15:20:45 UTC
Verified on RHEL5/6 i386/x86_64

RHEL5 packages:
condor-7.6.5-0.7.el5
condor-aviary-7.6.5-0.7.el5
cumin-0.1.5098-2.el5
python-suds-0.4.1-2.el5

RHEL6 packages:
condor-7.6.5-0.7.el6
condor-aviary-7.6.5-0.7.el6
cumin-0.1.5098-2.el6
python-suds-0.4.1-3.el6

Comment 25 errata-xmlrpc 2012-01-23 17:28:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2012-0045.html


Note You need to log in before you can comment on or make changes to this bug.