Bug 733677 - Integration of aviary for job control, submission, and job/submission queries [RFE]
Integration of aviary for job control, submission, and job/submission queries...
Status: CLOSED ERRATA
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: cumin (Show other bugs)
Development
Unspecified Unspecified
high Severity medium
: 2.1
: ---
Assigned To: Trevor McKay
Stanislav Graf
: FutureFeature, TechPreview
Depends On: 751278
Blocks: 743350 755648 814386
  Show dependency treegraph
 
Reported: 2011-08-26 09:30 EDT by Trevor McKay
Modified: 2012-04-19 14:31 EDT (History)
5 users (show)

See Also:
Fixed In Version: cumin-0.1.5047-1
Doc Type: Technology Preview
Doc Text:
Technology Preview feature CuminAviary Description: This feature allows Cumin to use the Aviary web services provided in the condor-aviary package for certain functions in the user interface. If the CuminAviary feature is enabled, Cumin will use Aviary services rather than QMF method calls where possible. The CuminAviary feature is controlled through the cumin configuration file. Relevant configuration parameters with descriptive comments can be found in the default /etc/cumin/cumin.conf file by searching for a line containing "Aviary interface to condor". Aviary provides a job service and a query service; Cumin may use either, both or neither. By default, Cumin will use QMF methods rather than Aviary services. To enable use of the Aviary job service, the 'aviary-job-servers' parameter must be uncommented and set (see the comments in the configuration file). Setting this parameter will cause Cumin to use the Aviary job service for job submission, for the hold, release, and remove job control functions, and for editing of job ad attributes. To enable use of the Aviary query service, the 'aviary-query-servers' parameter must be uncommented and set (see the comments in the configuration file). Setting this parameter will cause Cumin to use the Aviary query service for retrieving job output files, retrieving job ad details, and retreiving the list of jobs in a submission. Cumin will make INFO level entries in the log file for cumin-web that indicate whether use of the job and/or query services has been enabled and what type of certificate validation will be used for servers configured for SSL (see below). These log entries will begin with "AviaryOperations:" or contain the string "Aviary" somewhere in the message. If an Aviary operation fails, the yellow task banner associated with the operation will contain error information. By default, the Aviary services in condor will not use SSL (Secure Socket Layer) for communication and no other configuration parameters need to be set for this feature. However, if the Aviary services in condor have been configured to use SSL then additional configuration parameters must be set. First, note that the scheme for Aviary servers will change from "http" to "https" for any server using SSL. Failure to specify schemes correctly in the 'aviary-job-servers' or 'aviary-query-servers' parameters will prevent the CuminAviary feature from functioning. An incorrect server address may result in a default 90 second timeout when Cumin attempts to perform an operation using that server. Second, the 'aviary-key' and 'aviary-cert' parameters must be set. These parameters give the full paths to a PEM formatted private key file and PEM formatted certificate file that Cumin will use as a client to access the Aviary services. The Aviary servers will validate Cumin's client certificate and allow access if validation succeeds. Optionally, the 'aviary-root-cert' parameter may be set. This is the full path to a PEM formatted file containing CA (certificate authority) certificates that Cumin will use to validate the server certificate. If this parameter is unset Cumin will NOT validate server certificates. Here is a note relating to the ordering of certificate chains within a file from the OpenSSL documentation: "SSL_CTX_use_certificate_chain_file() loads a certificate chain from file into ctx. The certificates must be in PEM format and must be sorted starting with the subject's certificate (actual client or server certificate), followed by intermediate CA certificates if applicable, and ending at the highest level (root) CA. There is no corresponding function working on a single SSL object." Lastly, the 'aviary-domain-verify' parameter controls whether or not Cumin checks the hostname of the server against the server certificate during validation. This parameter has no effect unless 'aviary-root-cert' is set. The default value is True; it may be useful to set this parameter to False if the server is using a self-signed certificate with a non-matching hostname. Cumin will provide server certificate validation using the Python ssl standard language module if available or M2Crypto otherwise. If neither of these components are available, server certificate validation will be disabled. Dependencies: The CuminAviary feature has a dependency on python-suds-0.4.1 or newer. This package has been added as a dependency in the cumin rpm. Feedback: Bug reports or requests for enhancement can be made through http://bugzilla.redhat.com. General questions about this feature can be handled through the email list cumin-users@lists.fedorahosted.org Full support: This feature is intended to be fully supported in an upcoming minor release. Where to find this information: Content similar to this Release Note may be found in the file /usr/share/doc/cumin-*/AVIARY-README after the software is installed. However, the Release Note should be considered more up to date and where there are any discrepancies the Release Note supersedes the readme file. Technology Preview Policy: Technology Preview features are not currently supported under Red Hat Enterprise Linux subscription services, may not be functionally complete, and are generally not suitable for production use. However, these features are included as a customer convenience and to provide the technologies with wider exposure. Customers may find these features useful in non-production environments, and can provide feedback and functionality suggestions prior to their transition to fully supported status. Erratas will be provided for high-priority security issues. During its development additional components of a Technology Preview feature may become available to the public for testing. It is the intention of Red Hat to fully support Technology Preview features in a future release.
Story Points: ---
Clone Of:
: 814386 (view as bug list)
Environment:
Last Closed: 2012-01-23 12:28:52 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
Content of AVIARY-README file in /usr/share/doc/cumin-*, should match Tech Note. (5.31 KB, application/octet-stream)
2011-10-06 14:33 EDT, Trevor McKay
no flags Details

  None (edit)
Description Trevor McKay 2011-08-26 09:30:58 EDT
Description of problem:

Use the aviary service for job control and submission functions in cumin (query capabilities will come later).  This will be presented as a tech preview feature until an automated discovery mechanism is developed for aviary that removes the need for manual configuration.
Comment 1 Trevor McKay 2011-09-13 09:16:05 EDT
Fixed in revision 4963.

Job operations and query operations may be enabled independently.

Doc on config, etc, to come.
Comment 2 Trevor McKay 2011-09-22 12:24:02 EDT
Note, this feature has been expanded to explicitly include the query capabilities rather than just the job control and submission functions.  When the BZ was created, query operations had not been finished.  The query functions show up in 3 places:

When drilling into a submission to get a list of jobs within the submission.
When drilling into a job to get the job ad
When retrieving the output, error, and log files for a job.
Comment 4 Trevor McKay 2011-10-05 11:41:36 EDT
Updated in revision 5041.

INFO level log entries in web.log will indicate whether Aviary has been abled for job control and submission and/or query operations.  Entries will also indicate the type of communication used for Aviary.  Entries begin with "AviaryOperations:".  If there are no AviaryOperations logging messages indicating that it has been enabled, all functions are  handled through QMF.  This is the default.

A development-only configuration flag "aviary-suds-logs" (default False) can be set to True in cumin.conf in the [common] or [web] section.  This will turn on debug logging in the suds module used in communication with Aviary.  The logs will be $CUMIN_HOME/logs/suds.*.log.  This can help with debugging, also they can be used to verify that a particular operation in Cumin is going through Aviary.
Comment 5 Trevor McKay 2011-10-05 14:55:09 EDT
Technology Preview relevant Tech Notes added.
Comment 6 Trevor McKay 2011-10-05 14:55:09 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Technology Preview feature CuminAviary

Description:

This feature allows Cumin to use the Aviary web services provided in the condor-aviary package for certain functions in the user interface.  If the CuminAviary feature is enabled, Cumin will use Aviary services rather than QMF method calls where possible.  

The CuminAviary feature is enabled and configured through the /etc/cumin/cumin.conf file.  Relevant configuration parameters with descriptive comments can be found in the default /etc/cumin/cumin.conf file by searching for a line containing "Aviary interface to condor".

Aviary provides a job service and a query service; Cumin may use either, both or neither.  By default, Cumin will use no Aviary services and will use QMF methods instead.

To enable use of the Aviary job service, the 'aviary-job-servers' parameter must be uncommented and set (see the comments in the configuration file for details).  Setting this parameter will cause Cumin to use the Aviary job service for job submission, for the hold, release, and remove job control functions, and for editing of job ads.  

To enable use of the Aviary query service, the 'aviary-query-servers' parameter must be uncommented and set (see the comments in the configuration file for details).  Setting this parameter will cause Cumin to use the Aviary query service for retrieving job output files, retrieving job ad details, and retreiving the list of jobs in a submission.

Cumin will make INFO level entries in the log file for cumin-web that indicate whether use of the job and/or query services has been enabled and what type of certificate validation will be used for servers configured for SSL (see below).  These log entries will begin with "AviaryOperations:".  If an Aviary operation fails, the yellow task banner associated with the operation will contain error information.

By default, the Aviary services in condor will not use SSL (Secure Socket Layer) for communication and no other Cumin configuration parameters need to be set for the CuminAviary feature.  However, if the Aviary services in condor have been configured to use SSL then additional Cumin configuration parameters must be set.

First, note that the scheme for Aviary servers will change from "http" to "https" for any server using SSL.  Failure to specify schemes correctly in the 'aviary-job-servers' or 'aviary-query-servers' parameters will prevent CuminAviary from functioning.

Second, the 'aviary-key' and 'aviary-cert' parameters must be set.  These parameters give the full paths to a PEM formated private key file and PEM formatted certificate file that Cumin will use as a client to access the Aviary services.  The Aviary servers will validate Cumin's client certificate and allow access if validation succeeds.

Optionally, the 'aviary-root-cert' parameter may be set.  This is the full path to a PEM formatted file containing CA (certificate authority) certificates that Cumin will use to validate the server certificate.  If this parameter is unset Cumin will NOT validate server certificates.

Lastly, the 'aviary-domain-verify' parameter controls whether or not Cumin checks the hostname of the server against the server certificate during validation.  This parameter has no effect unless 'aviary-root-cert' is set.  The default value is True; it may be useful to set this parameter to False if the server is using a self-signed certificate with a non-matching hostname.

Feedback: bug reports or requests for enhancement can be made through http://bugzilla.redhat.com.  General questions about this feature can be handled through cumin-users@lists.fedorahosted.org

Full support:  This feature is intended to be fully supported in an upcoming minor release.

Where to find this information:  The content given here may be found in the Release Notes accompanying the software or in the file /usr/share/cumin/AVIARY-README after the software is installed.

Technology Preview Policy:
Technology Preview features are not currently supported under Red Hat Enterprise Linux subscription services, may not be functionally complete, and are generally not suitable for production use. However, these features are included as a customer convenience and to provide the technologies with wider exposure.

Customers may find these features useful in non-production environments, and can provide feedback and functionality suggestions prior to their transition to fully supported status. Erratas will be provided for high-priority security issues.

During its development additional components of a Technology Preview feature may become available to the public for testing. It is the intention of Red Hat to fully support Technology Preview features in a future release.
Comment 7 Trevor McKay 2011-10-05 15:26:09 EDT
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -28,7 +28,7 @@
 
 Full support:  This feature is intended to be fully supported in an upcoming minor release.
 
-Where to find this information:  The content given here may be found in the Release Notes accompanying the software or in the file /usr/share/cumin/AVIARY-README after the software is installed.
+Where to find this information:  The content given here may be found in the Release Notes accompanying the software or in the file /usr/share/doc/cumin-*/AVIARY-README after the software is installed.
 
 Technology Preview Policy:
 Technology Preview features are not currently supported under Red Hat Enterprise Linux subscription services, may not be functionally complete, and are generally not suitable for production use. However, these features are included as a customer convenience and to provide the technologies with wider exposure.
Comment 8 Trevor McKay 2011-10-06 14:33:04 EDT
Created attachment 526772 [details]
Content of AVIARY-README file in /usr/share/doc/cumin-*, should match Tech Note.

Attached the content of the AVIARY-README file that is part of the install.  This file is formatted carriage returns at 80 characters.  Thought it might be useful to attach it here, the content should match the Tech Note.
Comment 9 Trevor McKay 2011-10-06 14:45:51 EDT
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -4,19 +4,19 @@
 
 This feature allows Cumin to use the Aviary web services provided in the condor-aviary package for certain functions in the user interface.  If the CuminAviary feature is enabled, Cumin will use Aviary services rather than QMF method calls where possible.  
 
-The CuminAviary feature is enabled and configured through the /etc/cumin/cumin.conf file.  Relevant configuration parameters with descriptive comments can be found in the default /etc/cumin/cumin.conf file by searching for a line containing "Aviary interface to condor".
+The CuminAviary feature is controlled through the cumin configuration file. Relevant configuration parameters with descriptive comments can be found in the default /etc/cumin/cumin.conf file by searching for a line containing "Aviary interface to condor".
 
-Aviary provides a job service and a query service; Cumin may use either, both or neither.  By default, Cumin will use no Aviary services and will use QMF methods instead.
+Aviary provides a job service and a query service; Cumin may use either, both or neither.  By default, Cumin will use QMF methods rather than Aviary services.
 
-To enable use of the Aviary job service, the 'aviary-job-servers' parameter must be uncommented and set (see the comments in the configuration file for details).  Setting this parameter will cause Cumin to use the Aviary job service for job submission, for the hold, release, and remove job control functions, and for editing of job ads.  
+To enable use of the Aviary job service, the 'aviary-job-servers' parameter must be uncommented and set (see the comments in the configuration file).  Setting this parameter will cause Cumin to use the Aviary job service for job submission, for the hold, release, and remove job control functions, and for editing of job ad attributes.  
 
-To enable use of the Aviary query service, the 'aviary-query-servers' parameter must be uncommented and set (see the comments in the configuration file for details).  Setting this parameter will cause Cumin to use the Aviary query service for retrieving job output files, retrieving job ad details, and retreiving the list of jobs in a submission.
+To enable use of the Aviary query service, the 'aviary-query-servers' parameter must be uncommented and set (see the comments in the configuration file).  Setting this parameter will cause Cumin to use the Aviary query service for retrieving job output files, retrieving job ad details, and retreiving the list of jobs in a submission.
 
-Cumin will make INFO level entries in the log file for cumin-web that indicate whether use of the job and/or query services has been enabled and what type of certificate validation will be used for servers configured for SSL (see below).  These log entries will begin with "AviaryOperations:".  If an Aviary operation fails, the yellow task banner associated with the operation will contain error information.
+Cumin will make INFO level entries in the log file for cumin-web that indicate whether use of the job and/or query services has been enabled and what type of certificate validation will be used for servers configured for SSL (see below).  These log entries will begin with "AviaryOperations:" or contain the string "Aviary" somewhere in the message. If an Aviary operation fails, the yellow task banner associated with the operation will contain error information.
 
-By default, the Aviary services in condor will not use SSL (Secure Socket Layer) for communication and no other Cumin configuration parameters need to be set for the CuminAviary feature.  However, if the Aviary services in condor have been configured to use SSL then additional Cumin configuration parameters must be set.
+By default, the Aviary services in condor will not use SSL (Secure Socket Layer) for communication and no other configuration parameters need to be set for this feature.  However, if the Aviary services in condor have been configured to use SSL then additional configuration parameters must be set.
 
-First, note that the scheme for Aviary servers will change from "http" to "https" for any server using SSL.  Failure to specify schemes correctly in the 'aviary-job-servers' or 'aviary-query-servers' parameters will prevent CuminAviary from functioning.
+First, note that the scheme for Aviary servers will change from "http" to "https" for any server using SSL.  Failure to specify schemes correctly in the 'aviary-job-servers' or 'aviary-query-servers' parameters will prevent the CuminAviary feature from functioning.
 
 Second, the 'aviary-key' and 'aviary-cert' parameters must be set.  These parameters give the full paths to a PEM formated private key file and PEM formatted certificate file that Cumin will use as a client to access the Aviary services.  The Aviary servers will validate Cumin's client certificate and allow access if validation succeeds.
 
@@ -24,13 +24,28 @@
 
 Lastly, the 'aviary-domain-verify' parameter controls whether or not Cumin checks the hostname of the server against the server certificate during validation.  This parameter has no effect unless 'aviary-root-cert' is set.  The default value is True; it may be useful to set this parameter to False if the server is using a self-signed certificate with a non-matching hostname.
 
-Feedback: bug reports or requests for enhancement can be made through http://bugzilla.redhat.com.  General questions about this feature can be handled through cumin-users@lists.fedorahosted.org
+Cumin will provide server certificate validation using the Python ssl standard language module if available or M2Crypto otherwise.  If neither of these components are available, server certificate validation will be disabled.
 
-Full support:  This feature is intended to be fully supported in an upcoming minor release.
+Dependencies:
 
-Where to find this information:  The content given here may be found in the Release Notes accompanying the software or in the file /usr/share/doc/cumin-*/AVIARY-README after the software is installed.
+The CuminAviary feature has a dependency on 
+python-suds-0.4.1 or newer.  To date, this dependency is not enforced by the Cumin rpm.  On a system without python-suds installed, Cumin will install and run but the Aviary interface will be disabled.  If the CuminAviary feature is turned on in cumin.conf, an entry will be made in the log for cumin-web noting that Aviary has been disabled because of failed imports and Cumin will continue.
 
+Feedback: 
+
+Bug reports or requests for enhancement can be made through http://bugzilla.redhat.com.  General questions about this feature can be handled through the email list 
+cumin-users@lists.fedorahosted.org
+
+Full support:
+
+This feature is intended to be fully supported in an upcoming minor release.
+
+Where to find this information:  
+
+The content given here may be found in the Release Notes accompanying the software or in the file /usr/share/doc/cumin-*/AVIARY-README after the software is installed.
+
 Technology Preview Policy:
+
 Technology Preview features are not currently supported under Red Hat Enterprise Linux subscription services, may not be functionally complete, and are generally not suitable for production use. However, these features are included as a customer convenience and to provide the technologies with wider exposure.
 
 Customers may find these features useful in non-production environments, and can provide feedback and functionality suggestions prior to their transition to fully supported status. Erratas will be provided for high-priority security issues.
Comment 10 Trevor McKay 2011-10-27 12:08:12 EDT
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -16,7 +16,7 @@
 
 By default, the Aviary services in condor will not use SSL (Secure Socket Layer) for communication and no other configuration parameters need to be set for this feature.  However, if the Aviary services in condor have been configured to use SSL then additional configuration parameters must be set.
 
-First, note that the scheme for Aviary servers will change from "http" to "https" for any server using SSL.  Failure to specify schemes correctly in the 'aviary-job-servers' or 'aviary-query-servers' parameters will prevent the CuminAviary feature from functioning.
+First, note that the scheme for Aviary servers will change from "http" to "https" for any server using SSL.  Failure to specify schemes correctly in the 'aviary-job-servers' or 'aviary-query-servers' parameters will prevent the CuminAviary feature from functioning.  An incorrect server address will result in a default 90 second timeout when Cumin attempts to perform an operation using that server.
 
 Second, the 'aviary-key' and 'aviary-cert' parameters must be set.  These parameters give the full paths to a PEM formated private key file and PEM formatted certificate file that Cumin will use as a client to access the Aviary services.  The Aviary servers will validate Cumin's client certificate and allow access if validation succeeds.
Comment 11 Stanislav Graf 2011-11-03 08:15:28 EDT
testing non-ssl part:

Cumin versions:
cumin-0.1.5098-1.el5
cumin-0.1.5098-1.el6

[Play with cumin.conf and web.log]
- search for a line containing "Aviary interface to condor" in /etc/cumin/cumin.conf file
- 'aviary-job-servers' and 'aviary-query-servers' commented and restart cumin, web.log contains
INFO Disabled Aviary interface for job submission and control.
INFO Disabled Aviary interface for query operations.
- setup only 'aviary-job-servers: http://localhost:9090' and restart cumin
INFO Enabled Aviary interface for job submission and control.
INFO Disabled Aviary interface for query operations.
- setup only 'aviary-query-servers: http://localhost:9091' and restart cumin
INFO Disabled Aviary interface for job submission and control.
INFO Enabled Aviary interface for query operations.
- setup both job-servers and query-servers and restart cumin:
INFO Enabled Aviary interface for job submission and control.
INFO Enabled Aviary interface for query operations.
- try enable this feature on a system without python-suds:
INFO Imports failed for Aviary interface, disabling
INFO Disabled Aviary interface for job submission and control.
INFO Disabled Aviary interface for query operations.
- comment 'aviary-root-cert'
INFO AviaryOperations: no root certificate file specified, using client validation only for ssl connections.
- setup 'aviary-root-cert', comment aviary-domain-verify (leave default)
INFO AviaryOperations: using client and server certificate validation for ssl connections, solution is Python ssl
INFO AviaryOperations: verify server domain against certificate during validation (True)
- setup 'aviary-root-cert', 'aviary-domain-verify = False'
INFO AviaryOperations: using client and server certificate validation for ssl connections, solution is M2Crypto
INFO AviaryOperations: verify server domain against certificate during validation (False)

[Test without SSL - aviary]
- cumin.conf:
aviary-job-servers: http://localhost:9090
aviary-query-servers: http://localhost:9091
aviary-suds-logs: True
log-level: debug
- submitjob:
Submission: "zzz"
Cmd: "/bin/sleep 360"
Requirements: 'Memory >= 32 && OpSys == "LINUX" && Arch =="X86_64"'
Working Directory: "/tmp"
- after job is submitted - look at details of this job
- grep -i -e aviary -e method /var/log/cumin/web.log*
DEBUG AviaryOperations: suds logging on
INFO AviaryOperations: no root certificate file specified, using client validation only for ssl connections.
INFO Enabled Aviary interface for job submission and cont
rol.
INFO Enabled Aviary interface for query operations.
- grep zzz /var/log/cumin/suds.client.log 
<submission_name>zzz</submission_name> <SOAP-ENV:Envelope xmlns:SOAP-ENV=...
<name>zzz</name> <SOAP-ENV:Envelope xmlns:SOAP-ENV=.....

[Test without SSL - QMF]
- cumin.conf:
#aviary-job-servers: http://localhost:9090
#aviary-query-servers: http://localhost:9091
aviary-suds-logs: True
log-level: debug
- submitjob:
Submission: "zzz"
Cmd: "/bin/sleep 360"
Requirements: 'Memory >= 32 && OpSys == "LINUX" && Arch =="X86_64"'
Working Directory: "/tmp"
- after job is submitted - look at details of this job
- grep -i -e aviary -e method /var/log/cumin/web.log*
INFO Disabled Aviary interface for job submission and control.
INFO Disabled Aviary interface for query operations.
DEBUG Method response for request 1320321945 received from Broker connected at: localhost:5672
DEBUG Method response for request 1320321948 received from Broker connected at: localhost:5672
DEBUG Method response for request 1320321950 received from Broker connected at: localhost:5672
- grep zzz /var/log/cumin/suds.client.log 
grep: /var/log/cumin/suds.client.log: No such file or directory
Comment 12 Stanislav Graf 2011-11-04 03:02:39 EDT
[Test with SSL]

Try without CA certificate -> Bug 751278
Comment 13 Stanislav Graf 2011-11-09 08:07:01 EST
I ended with Bug 752414
Comment 14 Stanislav Graf 2011-11-10 06:48:07 EST
Cumin fresh install doesn't contain proper cumin.conf
bug 752763
Comment 15 Stanislav Graf 2011-11-10 07:25:34 EST
As was mentioned in Bug 752414 , generated certificates are verified by openssl as OK, but they need manual change/reformating to be able to use with cumin and condor-aviary - Bug 752777
Comment 16 Trevor McKay 2011-11-10 13:50:49 EST
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -16,12 +16,14 @@
 
 By default, the Aviary services in condor will not use SSL (Secure Socket Layer) for communication and no other configuration parameters need to be set for this feature.  However, if the Aviary services in condor have been configured to use SSL then additional configuration parameters must be set.
 
-First, note that the scheme for Aviary servers will change from "http" to "https" for any server using SSL.  Failure to specify schemes correctly in the 'aviary-job-servers' or 'aviary-query-servers' parameters will prevent the CuminAviary feature from functioning.  An incorrect server address will result in a default 90 second timeout when Cumin attempts to perform an operation using that server.
+First, note that the scheme for Aviary servers will change from "http" to "https" for any server using SSL.  Failure to specify schemes correctly in the 'aviary-job-servers' or 'aviary-query-servers' parameters will prevent the CuminAviary feature from functioning.  An incorrect server address may result in a default 90 second timeout when Cumin attempts to perform an operation using that server.
 
-Second, the 'aviary-key' and 'aviary-cert' parameters must be set.  These parameters give the full paths to a PEM formated private key file and PEM formatted certificate file that Cumin will use as a client to access the Aviary services.  The Aviary servers will validate Cumin's client certificate and allow access if validation succeeds.
+Second, the 'aviary-key' and 'aviary-cert' parameters must be set.  These parameters give the full paths to a PEM formatted private key file and PEM formatted certificate file that Cumin will use as a client to access the Aviary services.  The Aviary servers will validate Cumin's client certificate and allow access if validation succeeds.
 
 Optionally, the 'aviary-root-cert' parameter may be set.  This is the full path to a PEM formatted file containing CA (certificate authority) certificates that Cumin will use to validate the server certificate.  If this parameter is unset Cumin will NOT validate server certificates.
 
+Here is a note relating to the ordering of certificate chains within a file from the OpenSSL documentation: "SSL_CTX_use_certificate_chain_file() loads a certificate chain from file into ctx. The certificates must be in PEM format and must be sorted starting with the subject's certificate (actual client or server certificate), followed by intermediate CA certificates if applicable, and ending at the highest level (root) CA. There is no corresponding function working on a single SSL object."
+
 Lastly, the 'aviary-domain-verify' parameter controls whether or not Cumin checks the hostname of the server against the server certificate during validation.  This parameter has no effect unless 'aviary-root-cert' is set.  The default value is True; it may be useful to set this parameter to False if the server is using a self-signed certificate with a non-matching hostname.
 
 Cumin will provide server certificate validation using the Python ssl standard language module if available or M2Crypto otherwise.  If neither of these components are available, server certificate validation will be disabled.
@@ -29,7 +31,7 @@
 Dependencies:
 
 The CuminAviary feature has a dependency on 
-python-suds-0.4.1 or newer.  To date, this dependency is not enforced by the Cumin rpm.  On a system without python-suds installed, Cumin will install and run but the Aviary interface will be disabled.  If the CuminAviary feature is turned on in cumin.conf, an entry will be made in the log for cumin-web noting that Aviary has been disabled because of failed imports and Cumin will continue.
+python-suds-0.4.1 or newer.  This package has been added as a dependency in the cumin rpm.
 
 Feedback: 
 
@@ -42,7 +44,7 @@
 
 Where to find this information:  
 
-The content given here may be found in the Release Notes accompanying the software or in the file /usr/share/doc/cumin-*/AVIARY-README after the software is installed.
+Content similar to this Release Note may be found in the file /usr/share/doc/cumin-*/AVIARY-README after the software is installed.  However, the Release Note should be considered more up to date and where there are any discrepancies the Release Note supersedes the readme file.
 
 Technology Preview Policy:
Comment 17 Stanislav Graf 2011-11-11 04:11:06 EST
RHEL5/6 i386/x86_64

RHEL5:
condor-7.6.5-0.7.el5
condor-aviary-7.6.5-0.7.el5
cumin-0.1.5098-2.el5
python-suds-0.4.1-2.el5

RHEL6:
condor-7.6.5-0.7.el6
condor-aviary-7.6.5-0.7.el6
cumin-0.1.5098-2.el6
python-suds-0.4.1-3.el6

[Aviary SSL]
- generate certificate using guide in Bug 752414 (especially with switch
described in Bug 752414, comment 2 )
- cumin configuration
log-level: debug
aviary-job-servers: https://localhost:9090
aviary-query-servers: https://localhost:9091
aviary-key: /tmp/ssl/client.pem
aviary-cert: /tmp/ssl/client.pem
aviary-root-cert: /tmp/ssl/ca.pem
aviary-suds-logs: True
- aviary configuration:
SCHEDD.AVIARY_SSL = True
SCHEDD.AVIARY_SSL_SERVER_CERT = /tmp/ssl/serv.pem
SCHEDD.AVIARY_SSL_SERVER_KEY = /tmp/ssl/serv.pem
SCHEDD.AVIARY_SSL_CA_DIR = /tmp/ssl
SCHEDD.AVIARY_SSL_CA_FILE = /tmp/ssl/ca.pem
QUERY_SERVER.AVIARY_SSL = True
QUERY_SERVER.AVIARY_SSL_SERVER_CERT = /tmp/ssl/serv.pem
QUERY_SERVER.AVIARY_SSL_SERVER_KEY = /tmp/ssl/serv.pem
QUERY_SERVER.AVIARY_SSL_CA_DIR = /tmp/ssl
QUERY_SERVER.AVIARY_SSL_CA_FILE = /tmp/ssl/ca.pem
- submitjob:
Submission: "zzz"
Cmd: "/bin/sleep 360"
Requirements: 'Memory >= 32 && OpSys == "LINUX" && Arch =="X86_64"'
Working Directory: "/tmp"
- after job is submitted - look at details of this job

[Verify log files]
- on RHEL5 - M2Crypto is used
- on RHEL6 - Python ssl is used
this is in accordance to Bug 733447, comment 12
- no qmf methods used

- grep -i -e aviary -e method /var/log/cumin/web.log*

RHEL5
DEBUG AviaryOperations: suds logging on
INFO AviaryOperations: using client and server certificate validation for ssl
connections, solution is M2Crypto
INFO AviaryOperations: verify server domain against certificate during
validation (True)
INFO Enabled Aviary interface for job submission and control.
Enabled Aviary interface for query operations.

RHEL6
DEBUG AviaryOperations: suds logging on
INFO AviaryOperations: using client and server certificate validation for ssl
connections, solution is Python ssl
INFO AviaryOperations: verify server domain against certificate during
validation (True)
INFO Enabled Aviary interface for job submission and control.
INFO Enabled Aviary interface for query operations.

- grep zzz /var/log/cumin/suds.client.log 
         <submission_name>zzz</submission_name> <SOAP-ENV:Envelope
xmlns:SOAP-ENV="...>
         <name>zzz</name> <SOAP-ENV:Envelope xmlns:SOAP-ENV="...>
Comment 18 Stanislav Graf 2011-11-11 05:18:40 EST
cumin dependency contains python-suds on RHEL5/6
package: cumin.noarch 0.1.5098-2.el5
  ...
  dependency: qpid-qmf >= 0.10-11
  ...
  dependency: python-suds >= 0.4.1
  ...
package: cumin.noarch 0.1.5098-2.el6
  ...
  dependency: qpid-qmf >= 0.12-6
  ...
  dependency: python-suds >= 0.4.1
  ...

condor-aviary dependency contains python-suds on RHEL5/6
package: condor-aviary.i386 7.6.5-0.7.el5
  ...
  dependency: python-suds >= 0.4.1
  ...
  dependency: condor = 7.6.5-0.7.el5
  ...
package: condor-aviary.i686 7.6.5-0.7.el6
  ...
  dependency: condor = 7.6.5-0.7.el6
  ...
  dependency: python-suds >= 0.4.1
  ...
Comment 19 Stanislav Graf 2011-11-11 06:16:33 EST
[Condor SSL, Cumin non-SSL]

- enable SSL on condor-aviary

- setup cumin to not use / disable SSL
log-level: debug
aviary-job-servers: http://localhost:9090
aviary-query-servers: http://localhost:9091
#aviary-key: /tmp/ssl/client.pem
#aviary-cert: /tmp/ssl/client.pem
#aviary-root-cert: /tmp/ssl/ca.pem
aviary-suds-logs: True

- try to submit job and it failed
Cumin shows on both RHEL5 and RHEL6:
Submit job 'zzz': Failed (Trouble reaching host (hostname), timed out)
condor log ShedLog shows:
11/11/11 11:32:46 (pid:2326) axis2_ssl_utils_initialize_ssl failed
11/11/11 11:32:46 (pid:2326) Error processing request: Failed to accept connection

-> OK

[Condor non-SSL, Cumin SSL]

- setup condor to not use /disable SSL
SCHEDD.AVIARY_SSL = False
SCHEDD.AVIARY_SSL_SERVER_CERT = /tmp/ssl/serv.pem
SCHEDD.AVIARY_SSL_SERVER_KEY = /tmp/ssl/serv.pem
SCHEDD.AVIARY_SSL_CA_DIR = /tmp/ssl
SCHEDD.AVIARY_SSL_CA_FILE = /tmp/ssl/ca.pem
QUERY_SERVER.AVIARY_SSL = False
QUERY_SERVER.AVIARY_SSL_SERVER_CERT = /tmp/ssl/serv.pem
QUERY_SERVER.AVIARY_SSL_SERVER_KEY = /tmp/ssl/serv.pem
QUERY_SERVER.AVIARY_SSL_CA_DIR = /tmp/ssl
QUERY_SERVER.AVIARY_SSL_CA_FILE = /tmp/ssl/ca.pem

- enable SSL on cumin side

- try to submit job and it failed
RHEL5
Submit job 'zzz': Failed (unexpected eof)
RHEL6
Submit job 'zzz': Failed (Trouble reaching host (hostname), [Errno 8] _ssl.c:490: EOF occurred in violation of protocol)

-> OK
Comment 22 Stanislav Graf 2011-11-11 06:45:50 EST
RHEL5/6 i386/x86_64

---> VERIFIED
Comment 23 errata-xmlrpc 2012-01-23 12:28:52 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2012-0045.html

Note You need to log in before you can comment on or make changes to this bug.