Bug 812915 - Audrey agent fails to connect to config server openssl self signed cert issue
Summary: Audrey agent fails to connect to config server openssl self signed cert issue
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: CloudForms Cloud Engine
Classification: Retired
Component: aeolus-audrey-agent
Version: 1.0.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: rc
Assignee: Dan Radez
QA Contact: dgao
URL:
Whiteboard:
Depends On:
Blocks: 813319 826708
TreeView+ depends on / blocked
 
Reported: 2012-04-16 14:45 UTC by Rehana
Modified: 2012-12-04 15:03 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
The two versions of python-httplib2 (0.6x and 0.7x) were available, however Audrey Agent was not compatible with 0.7x, and as a result Cloud Engine was unable to communicate with Agent Agent. This fix updates Audrey Agent to be compatible with 0.7x by allowing self signed certification. This means that Audrey Agent can successfully communicate with Cloud Engine.
Clone Of:
: 813319 826708 (view as bug list)
Environment:
Last Closed: 2012-12-04 15:03:48 UTC

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2012:1516 0 normal SHIPPED_LIVE CloudForms Cloud Engine 1.1 update 2012-12-04 19:51:45 UTC

Description Rehana 2012-04-16 14:45:13 UTC 
Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.Login to conductor
2.Launch runtime configuration instance (tested with tpl from URL:https://github.com/aeolusproject/audrey/tree/master/examples/wordpress) on vsphere provider

  
Actual results:
Observed that the 'wordpress' application was not available after the launching the template.

Note:Observed that the instances were in running state for some time with out getting the ip, and after some time the instances got the ip.

audrey.log from Apache machine:

2012-04-16 05:44:55,902 - INFO    : audrey:1293 Invoked audrey_script_main
2012-04-16 05:44:56,074 - INFO    : audrey:1313 Failed attempt to contact config server
2012-04-16 05:45:06,218 - INFO    : audrey:1313 Failed attempt to contact config server
2012-04-16 05:45:16,449 - INFO    : audrey:1313 Failed attempt to contact config server
2012-04-16 05:45:26,591 - INFO    : audrey:1313 Failed attempt to contact config server
2012-04-16 05:45:36,732 - INFO    : audrey:1313 Failed attempt to contact config server
2012-04-16 05:45:46,942 - ERROR   : audrey:1316 Failed to connect to the Configserver
 

Expected results:
Once the instance are up the wordpress application should be available.


Additional info:
rpm -qa | grep aeolus
aeolus-conductor-0.8.7-1.el6.noarch
aeolus-configure-2.5.2-1.el6.noarch
aeolus-conductor-daemons-0.8.7-1.el6.noarch
rubygem-aeolus-image-0.3.0-12.el6.noarch
rubygem-aeolus-cli-0.3.1-1.el6.noarch
aeolus-all-0.8.7-1.el6.noarch
aeolus-conductor-doc-0.8.7-1.el6.noarch
Comment 1 Greg Blomquist 2012-04-16 15:48:38 UTC 
I added some additional logging statements to /usr/bin/audrey on the guest in question.

Current logs show:

[root ~]# cat /var/log/audrey.log 
2012-04-16 06:44:29,151 - INFO    : audrey:1295 Invoked audrey_script_main
2012-04-16 06:44:29,151 - DEBUG   : audrey:868 HTTP GET: https://cloudengine-audrey.usersys.redhat.com/version; headers=None
2012-04-16 06:44:29,418 - ERROR   : audrey:871 Error in HTTP GET: https://cloudengine-audrey.usersys.redhat.com/version; headers=None;
error=[Errno 1] _ssl.c:490: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
2012-04-16 06:44:29,418 - INFO    : audrey:1315 Failed attempt to contact config server
2012-04-16 06:44:39,429 - DEBUG   : audrey:868 HTTP GET: https://cloudengine-audrey.usersys.redhat.com/version; headers=None
2012-04-16 06:44:39,566 - ERROR   : audrey:871 Error in HTTP GET: https://cloudengine-audrey.usersys.redhat.com/version; headers=None;
error=[Errno 1] _ssl.c:490: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
2012-04-16 06:44:39,566 - INFO    : audrey:1315 Failed attempt to contact config server
2012-04-16 06:44:49,577 - DEBUG   : audrey:868 HTTP GET: https://cloudengine-audrey.usersys.redhat.com/version; headers=None
2012-04-16 06:44:49,700 - ERROR   : audrey:871 Error in HTTP GET: https://cloudengine-audrey.usersys.redhat.com/version; headers=None;
error=[Errno 1] _ssl.c:490: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
2012-04-16 06:44:49,701 - INFO    : audrey:1315 Failed attempt to contact config server
2012-04-16 06:44:59,711 - DEBUG   : audrey:868 HTTP GET: https://cloudengine-audrey.usersys.redhat.com/version; headers=None
2012-04-16 06:44:59,840 - ERROR   : audrey:871 Error in HTTP GET: https://cloudengine-audrey.usersys.redhat.com/version; headers=None;
error=[Errno 1] _ssl.c:490: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
2012-04-16 06:44:59,840 - INFO    : audrey:1315 Failed attempt to contact config server
2012-04-16 06:45:09,851 - DEBUG   : audrey:868 HTTP GET: https://cloudengine-audrey.usersys.redhat.com/version; headers=None
2012-04-16 06:45:09,979 - ERROR   : audrey:871 Error in HTTP GET: https://cloudengine-audrey.usersys.redhat.com/version; headers=None;
error=[Errno 1] _ssl.c:490: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
2012-04-16 06:45:09,980 - INFO    : audrey:1315 Failed attempt to contact config server
2012-04-16 06:45:19,990 - DEBUG   : audrey:868 HTTP GET: https://cloudengine-audrey.usersys.redhat.com/version; headers=None
2012-04-16 06:45:20,127 - ERROR   : audrey:871 Error in HTTP GET: https://cloudengine-audrey.usersys.redhat.com/version; headers=None;
error=[Errno 1] _ssl.c:490: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
2012-04-16 06:45:20,127 - ERROR   : audrey:1318 Failed to connect to the Configserver
Comment 2 Greg Blomquist 2012-04-16 17:05:24 UTC 
This problem stems from the version of python-httplib2.

The version on the RHEL-6.2 guest was 0.7.2-1.el6.  That version came from epel.

We downgraded to the standard rhel6.2 provided version of python-httplib2 0.6.0-4.el6_0

This fixes the problem, but leaves us with two other problems:

1)  epel is needed for the wordpress demo on rhel because epel is the only place to get wordpress

2)  when python-httplib2 0.7.2-1.el6 becomes the standard version in rhel, this problem will arise again in a much more permanent way
Comment 3 Greg Blomquist 2012-04-16 17:07:42 UTC 
A quick workaround could be to restrict the nvr of python-httplib2 to 0.6.0-4.el6_0 or something similar in the aeolus-audrey-agent spec file.
Comment 4 wes hayutin 2012-04-16 17:44:31 UTC 
This needs to be fixed in the 1.0.z release. The fix is for the audrey agent, customers that use audrey to install code that requires python-httplib2 will fail.
Comment 5 jrd 2012-04-17 16:58:36 UTC 
Greg, can you put together a couple sentences to go into relnotes for this, to tide us over until 1.0z?
Comment 6 Greg Blomquist 2012-04-19 12:50:08 UTC 
Updating the component to audrey-agent.  Adding relnotes to technical notes.
Comment 7 Greg Blomquist 2012-04-19 12:50:08 UTC 
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
The audrey-agent currently requires python-httplib2 v0.6.0.  CloudForms Cloud Engine channel provides python-httplib2-0.6.0-4.el6_0.  However, EPEL is providing python-httplib2-0.7.2-1.el6.  Therefore, images built with the audrey agent and that include EPEL as a repository in the image template will experience this bug.
Comment 8 Greg Blomquist 2012-04-25 20:05:39 UTC 
We're leaving this bug open to track the issue in the audrey-agent to be fixed in zstream.

The issue is that audrey-agent should handle ssl cert validation in both python-httplib 0.6.0 and python-httplib 0.7.0.

In the short term (zstream timeline), the fix will be to introduce logic into audrey-agent that turns off ssl cert validation when python-httplib 0.7.0 is loaded (and not change anything when python-httplib 0.6.0 is loaded).

In the long term (1.1?/2.0?), the fix will be to propagate the appropriate ssl cert from the config server to the launching guest with audrey-agent.  There's no specific plan in place yet to determine how to make this happen.  But, it opens up the larger opportunity for widespread certificate management in Cloud Forms.
Comment 11 Greg Blomquist 2012-05-23 21:18:09 UTC 
Assigning to Dan.
Comment 12 Dan Radez 2012-05-25 11:43:39 UTC 
fixed in da87064e28d588925959e270f66d7183a6500295
built as 0.4.9-1
Comment 14 dgao 2012-06-13 17:29:19 UTC 
[root@10-16-120-177 ~]# cat /var/log/audrey.log 
2012-06-13 13:26:44,423 - INFO    : audrey:1305 Invoked audrey_script_main
2012-06-13 13:26:44,685 - INFO    : audrey:1334 
<Instance of: CSClient
	Version: 1
	Config Server Endpoint: https://deaddonkey.usersys.redhat.com
	Config Server oAuth Key: af6caa10-b56d-11e1-9376-e83935c21f2c
	Config Server oAuth Secret: dNq4bvMxPoKwr3tFuChikdIe5nQYCRuqejSSKuzIOzT2
	Config Server Params: 
	Config Server Configs: 
	Temporary Directory: 
	Tarball Name: 
eot>
2012-06-13 13:26:44,686 - INFO    : audrey:951 Invoked CSClient.get_cs_tooling()
2012-06-13 13:26:44,886 - INFO    : audrey:683 Invoked unpack_tooling()
2012-06-13 13:26:44,888 - INFO    : audrey:908 Invoked CSClient.get_cs_configs()
2012-06-13 13:26:45,086 - INFO    : audrey:923 Invoked CSClient.get_cs_params()
2012-06-13 13:26:45,286 - INFO    : audrey:521 Invoked generate_provides()
2012-06-13 13:26:45,636 - INFO    : audrey:938 Invoked CSClient.put_cs_params_values()
[root@10-16-120-177 ~]# rpm -qa | grep "python-httplib"
python-httplib2-0.7.4-1.el6.noarch
[root@10-16-120-177 ~]# rpm -qa | grep "audrey"
aeolus-audrey-agent-0.4.9-1.el6_2.noarch
Comment 16 errata-xmlrpc 2012-12-04 15:03:48 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2012-1516.html
Note You need to log in before you can comment on or make changes to this bug.