Bug 1132471

Summary: pulp-manage-db threw a traceback: ConnectError: [Errno 336265218] _ssl.c:341: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib
Product: Red Hat Satellite Reporter: Jan Hutař <jhutar>
Component: InstallationAssignee: Ivan Necas <inecas>
Status: CLOSED CURRENTRELEASE QA Contact: Jan Hutař <jhutar>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.0.3CC: inecas, jmontleo, lzap
Target Milestone: Unspecified   
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
URL: http://projects.theforeman.org/issues/7239
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-09-11 12:21:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jan Hutař 2014-08-21 12:11:42 UTC
Description of problem:
pulp-manage-db threw a traceback: ConnectError: [Errno 336265218] _ssl.c:341: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib

inecas investigated this and his theory is this is caused by puppet choosing what happens first: qpid setting or pulp data migration


Version-Release number of selected component (if applicable):
Satellite-6.0.4-RHEL-6-20140820.2


How reproducible:
1 occurrence out of 2 attempts


Steps to Reproduce:
1. Attempt to install Satellite 6 on RHEL5 with:
   # katello-installer --foreman-admin-email 'root@localhost' \
       --foreman-admin-username 'admin' --foreman-admin-password 'changeme'


Actual results:
[...]
[31m /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[0m[31m /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[0mNotice: /Stage[main]/Foreman::Database/Foreman::Ra: 478/479, 99%, 0.0/s, elapsed: 00:10:55
Debug: Executing '/sbin/chkconfig foreman-tasks'  : 478/479, 99%, 0.0/s, elapsed: 00:11:53
Notice: /Stage[main]/Foreman::Service/Exec[restart: 478/479, 99%, 0.0/s, elapsed: 00:11:55
Debug: /Stage[main]/Certs::Foreman/Exec[foreman_ce: 478/479, 99%, 0.0/s, elapsed: 00:12:57
Notice: /Stage[main]/Certs::Foreman/Exec[foreman_c: 478/479, 99%, 0.0/s, elapsed: 00:13:56
E, [2014-08-20T21:09:16.481482 #13443] ERROR -- : : 478/479, 99%, 0.0/s, elapsed: 00:14:58
[31m /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[<fqdn>]: Could not evaluate: 500 Internal Server Error
[0m[32mDone                                              [0m: 479/479, 100%, 0.0/s, elapsed: 00:15:03
[32mDone                                              [0m: 479/479, 100%, 0.0/s, elapsed: 00:15:03
  [1m[31mSomething went wrong![0m Check the log for ERROR-level output
  The full log is at [1m[36m/var/log/katello-installer/katello-installer.log[0m

inecas noticed this in /var/log/messages:
[...]
Aug 20 21:02:43 <host> pulp: pulp.server.db.manage:INFO: Applying pulp.server.db.migrations version 9
Aug 20 21:02:43 <host> pulp: pulp.server.db.manage:CRITICAL: Applying migration pulp.server.db.migrations.0009_qpid_queues failed.
Aug 20 21:02:43 <host> qpidd[8670]: 2014-08-20 21:02:43 [System] error Error reading socket: Success(0)
Aug 20 21:02:43 <host> pulp: pulp.server.db.manage:CRITICAL: [Errno 336265218] _ssl.c:341: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib
Aug 20 21:02:43 <host> pulp: pulp.server.db.manage:CRITICAL: Traceback (most recent call last):
Aug 20 21:02:43 <host> pulp: pulp.server.db.manage:CRITICAL:   File "/usr/lib/python2.6/site-packages/pulp/server/db/manage.py", line 86, in migrate_database
Aug 20 21:02:43 <host> pulp: pulp.server.db.manage:CRITICAL:     update_current_version=not options.test)
Aug 20 21:02:43 <host> pulp: pulp.server.db.manage:CRITICAL:   File "/usr/lib/python2.6/site-packages/pulp/server/db/migrate/models.py", line 161, in apply_migration
Aug 20 21:02:43 <host> pulp: pulp.server.db.manage:CRITICAL:     migration.migrate()
Aug 20 21:02:43 <host> pulp: pulp.server.db.manage:CRITICAL:   File "/usr/lib/python2.6/site-packages/pulp/server/db/migrations/0009_qpid_queues.py", line 67, in migrate
Aug 20 21:02:43 <host> pulp: pulp.server.db.manage:CRITICAL:     connection.attach()
Aug 20 21:02:43 <host> pulp: pulp.server.db.manage:CRITICAL:   File "<string>", line 6, in attach
Aug 20 21:02:43 <host> pulp: pulp.server.db.manage:CRITICAL:   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 286, in attach
Aug 20 21:02:43 <host> pulp: pulp.server.db.manage:CRITICAL:     if not self._ewait(lambda: self._transport_connected and not self._unlinked(), timeout=timeout):
Aug 20 21:02:43 <host> pulp: pulp.server.db.manage:CRITICAL:   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 219, in _ewait
Aug 20 21:02:43 <host> pulp: pulp.server.db.manage:CRITICAL:     self.check_error()
Aug 20 21:02:43 <host> pulp: pulp.server.db.manage:CRITICAL:   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 212, in check_error
Aug 20 21:02:43 <host> pulp: pulp.server.db.manage:CRITICAL:     raise e
Aug 20 21:02:43 <host> pulp: pulp.server.db.manage:CRITICAL: ConnectError: [Errno 336265218] _ssl.c:341: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib
Aug 20 21:02:43 <host> pulp: pulp.server.db.manage:CRITICAL: 
Aug 20 21:02:43 <host> pulp: pulp.server.db.manage:INFO: Applying pulp_puppet.plugins.migrations version 1
Aug 20 21:02:43 <host> pulp: pulp:INFO: Migrated puppet modules to include checksum
Aug 20 21:02:43 <host> pulp: pulp.server.db.manage:INFO: Migration to pulp_puppet.plugins.migrations version 1 complete.
Aug 20 21:02:43 <host> pulp: pulp.server.db.manage:INFO: Applying pulp_puppet.plugins.migrations version 2
[...]


Expected results:
No traceback, installation should work.

Comment 1 Jan Hutař 2014-08-21 12:15:57 UTC
Bug 1132475 was created to track fact that installer do not report more useful message.

Comment 3 Lukas Zapletal 2014-08-22 08:10:55 UTC
Just a hint - make sure there were no denials. We have a bug opened on RHEL7 qpidd policy - it is blocking it from starting.

Comment 4 Ivan Necas 2014-08-22 11:55:08 UTC
This was a rhel6 machine and the ordering was clearly wrong. Thank you puppet!

Comment 5 Ivan Necas 2014-08-22 14:59:14 UTC
Created redmine issue http://projects.theforeman.org/issues/7239 from this bug

Comment 6 Ivan Necas 2014-08-22 15:09:44 UTC
Fix proposed here https://github.com/Katello/puppet-certs/pull/32

Comment 7 Ivan Necas 2014-08-27 15:33:06 UTC
Katello-installer update here https://github.com/Katello/katello-installer/pull/112

Comment 8 Ivan Necas 2014-08-27 16:51:43 UTC
Updates merged in https://github.com/Katello/katello-installer/pull/112

Comment 12 Bryan Kearney 2014-09-11 12:21:59 UTC
This was delivered with Satellite 6.0 which was released on 10 September 2014.