Bug 1361196 - Upgrade to 6.2 is failing due to Pulp message bus connection issue
Summary: Upgrade to 6.2 is failing due to Pulp message bus connection issue
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Upgrades
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
high
high vote
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: jcallaha
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-28 13:29 UTC by Andy MacMurray
Modified: 2019-11-14 08:49 UTC (History)
22 users (show)

Fixed In Version: python-kombu-3.0.33-8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-22 06:35:59 UTC
Target Upstream Version:


Attachments (Terms of Use)
Satellite log (1.81 MB, text/plain)
2016-07-28 13:29 UTC, Andy MacMurray
no flags Details
Production log (654.45 KB, text/plain)
2016-07-28 13:30 UTC, Andy MacMurray
no flags Details
python-kombu EL6 hotfix package (520.04 KB, application/octet-stream)
2016-08-17 04:34 UTC, Mike McCune
no flags Details
python-kombu EL7 hotfix package (485.94 KB, application/octet-stream)
2016-08-17 04:34 UTC, Mike McCune
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1643 normal SHIPPED_LIVE Satellite 6.2 Upgrades Bugfixes 2016-08-22 10:35:37 UTC
Pulp Redmine 2124 Urgent CLOSED - CURRENTRELEASE Pulp status check leaves open Pipes 2016-08-19 14:00:27 UTC

Description Andy MacMurray 2016-07-28 13:29:55 UTC
Created attachment 1185106 [details]
Satellite log

Description of problem: We've encountered two occurances of the upgrade failing due to Pulp message bus connection issues.

[ INFO 2016-07-28 07:52:31 main] Elasticsearch has been removed as a dependency.  The database files can be removed manually with #rm -rf /var/lib/elasticsearch.  Some packages are no longer needed and can be removed:  #rpm -e ruby193-rubygem-tire tfm-rubygem-tire elasticsearch sigar-java sigar snappy-java lucene4-contrib lucene4
[ INFO 2016-07-28 07:52:31 main] Upgrade Step: add_export_distributor (this may take a while) ...
[ERROR 2016-07-28 09:00:12 main] rake aborted!
There was an issue with the backend service pulp: Pulp message bus connection issue.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/app/lib/actions/middleware/backend_services_check.rb:16:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/app/lib/actions/middleware/backend_services_check.rb:14:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/app/lib/actions/middleware/backend_services_check.rb:14:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/app/lib/actions/middleware/remote_action.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:34:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:34:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:460:in `block (2 levels) in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:281:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:281:in `switch_flow'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:364:in `concurrence'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:459:in `block in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:458:in `execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:260:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/plan_step.rb:53:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:392:in `plan_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/app/lib/actions/katello/repository/refresh_repostiory.rb:8:in `block in plan'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:72:in `as'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:78:in `as_anonymous_admin'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/app/lib/actions/katello/repository/refresh_repostiory.rb:6:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:461:in `block (3 levels) in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:34:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:34:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:460:in `block (2 levels) in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:281:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:281:in `switch_flow'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:364:in `concurrence'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:459:in `block in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:458:in `execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:260:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/plan_step.rb:53:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:191:in `block (2 levels) in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:281:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:281:in `switch_flow'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:271:in `with_planning_scope'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:190:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:42:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/common/transaction.rb:16:in `block in rollback_on_error'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `block in transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:209:in `within_new_transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/transaction_adapters/active_record.rb:5:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/common/transaction.rb:15:in `rollback_on_error'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/common/transaction.rb:5:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:189:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/world.rb:168:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/world.rb:166:in `tap'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/world.rb:166:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/world.rb:142:in `trigger'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.6/lib/foreman_tasks.rb:18:in `trigger'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.6/lib/foreman_tasks.rb:24:in `trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.6/lib/foreman_tasks.rb:39:in `sync_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/lib/katello/tasks/upgrades/3.0/add_export_distributor.rake:9:in `block (5 levels) in <top (required)>'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `block (2 levels) in find_each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `block in find_each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:125:in `find_in_batches'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:51:in `find_each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/querying.rb:9:in `find_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/lib/katello/tasks/upgrades/3.0/add_export_distributor.rake:8:in `block (4 levels) in <top (required)>'
Tasks: TOP => katello:upgrades:3.0:add_export_distributor
(See full trace by running task with --trace)
Refreshing existing repositories to add export distributor

[ERROR 2016-07-28 09:00:12 main] Upgrade step add_export_distributor failed. Check logs for more information.
[DEBUG 2016-07-28 09:00:12 main] Exit with status code: 1 (signal was 1)
[DEBUG 2016-07-28 09:00:12 main] Cleaning /etc/katello-installer/d20160728-28117-eu5lyg
[DEBUG 2016-07-28 09:00:12 main] Cleaning /tmp/default_values.yaml



Version-Release number of selected component (if applicable):


How reproducible: 2/2 so far.


Steps to Reproduce:
1. 
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Andy MacMurray 2016-07-28 13:30:36 UTC
Created attachment 1185107 [details]
Production log

Comment 2 Chris Duryee 2016-07-28 13:41:36 UTC
additional info from irc, this happens about six hours into the upgrade process. Also this happens on el6. It's unknown if this is also on el7.

Comment 3 Ivan Necas 2016-07-29 11:58:56 UTC
Could you upload the whole sosreport for further analysis?

Comment 4 Andy MacMurray 2016-07-29 12:07:30 UTC
What options do you recommend running sosreport with?

Also FYI - Ohad and Justin Sherrill have been looking at this instance and investigating.

Thanks,
 - Andy

Comment 7 Justin Sherrill 2016-07-29 18:54:22 UTC
To add a bit of information, at the time of the failure pulp seems to throw:


 qpidd[12237]: 2016-07-29 11:55:01 [System] error Error reading socket: Success(0)
 pulp: pulp.server.managers.status:ERROR: (9893-87936) [Errno 185090050] _ssl.c:330: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib
 pulp: pulp.server.managers.status:ERROR: (9893-87936) Traceback (most recent call last):
 pulp: pulp.server.managers.status:ERROR: (9893-87936)   File "/usr/lib/python2.6/site-packages/pulp/server/managers/status.py", line 58, in get_broker_conn_status
 pulp: pulp.server.managers.status:ERROR: (9893-87936)     conn.connect()
 pulp: pulp.server.managers.status:ERROR: (9893-87936)   File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 237, in connect
 pulp: pulp.server.managers.status:ERROR: (9893-87936)     return self.connection
 pulp: pulp.server.managers.status:ERROR: (9893-87936)   File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 741, in connection
 pulp: pulp.server.managers.status:ERROR: (9893-87936)     self._connection = self._establish_connection()
 pulp: pulp.server.managers.status:ERROR: (9893-87936)   File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 696, in _establish_connection
 pulp: pulp.server.managers.status:ERROR: (9893-87936)     conn = self.transport.establish_connection()
 pulp: pulp.server.managers.status:ERROR: (9893-87936)   File "/usr/lib/python2.6/site-packages/kombu/transport/qpid.py", line 1600, in establish_connection
 pulp: pulp.server.managers.status:ERROR: (9893-87936)     conn = self.Connection(**opts)
 pulp: pulp.server.managers.status:ERROR: (9893-87936)   File "/usr/lib/python2.6/site-packages/kombu/transport/qpid.py", line 1261, in __init__
 pulp: pulp.server.managers.status:ERROR: (9893-87936)     self._qpid_conn = establish(**self.connection_options)
 pulp: pulp.server.managers.status:ERROR: (9893-87936)   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 112, in establish
 pulp: pulp.server.managers.status:ERROR: (9893-87936)     conn.open(timeout=timeout)
 pulp: pulp.server.managers.status:ERROR: (9893-87936)   File "<string>", line 6, in open
 pulp: pulp.server.managers.status:ERROR: (9893-87936)   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 323, in open
 pulp: pulp.server.managers.status:ERROR: (9893-87936)     self.attach(timeout=timeout)
 pulp: pulp.server.managers.status:ERROR: (9893-87936)   File "<string>", line 6, in attach
 pulp: pulp.server.managers.status:ERROR: (9893-87936)   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 341, in attach
 pulp: pulp.server.managers.status:ERROR: (9893-87936)     if not self._ewait(lambda: self._transport_connected and not self._unlinked(), timeout=timeout):
 pulp: pulp.server.managers.status:ERROR: (9893-87936)   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 274, in _ewait
 pulp: pulp.server.managers.status:ERROR: (9893-87936)     self.check_error()
 pulp: pulp.server.managers.status:ERROR: (9893-87936)   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 267, in check_error
 pulp: pulp.server.managers.status:ERROR: (9893-87936)     raise e
 pulp: pulp.server.managers.status:ERROR: (9893-87936) ConnectError: [Errno 185090050] _ssl.c:330: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib


And then when i make requests to  /pulp/api/v2/status/ the failure responses trigger:


ERROR: (11965-28352) [Errno 24] Too many open files
ERROR: (11965-28352) Traceback (most recent call last):
ERROR: (11965-28352)   File "/usr/lib/python2.6/site-packages/pulp/server/managers/status.py", line 58, in get_broker_conn_status
ERROR: (11965-28352)     conn.connect()
ERROR: (11965-28352)   File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 237, in connect
ERROR: (11965-28352)     return self.connection
ERROR: (11965-28352)   File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 741, in connection
ERROR: (11965-28352)     self._connection = self._establish_connection()
ERROR: (11965-28352)   File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 696, in _establish_connection
ERROR: (11965-28352)     conn = self.transport.establish_connection()
ERROR: (11965-28352)   File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 769, in transport
ERROR: (11965-28352)     self._transport = self.create_transport()
ERROR: (11965-28352)   File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 514, in create_transport
ERROR: (11965-28352)     return self.get_transport_cls()(client=self)
ERROR: (11965-28352)   File "/usr/lib/python2.6/site-packages/kombu/transport/qpid.py", line 1396, in __init__
ERROR: (11965-28352)     self.r, self._w = os.pipe()
ERROR: (11965-28352) OSError: [Errno 24] Too many open files



This may be related to https://bugzilla.redhat.com/show_bug.cgi?id=1328984


In addition, on several occasions I've seen apache segfault:

 kernel: httpd[10215]: segfault at 28 ip 00007fd20e0cd11b sp 00007fd1da5f5cc0 error 4 in libapr-1.so.0.3.9[7fd20e0ae000+2b000]

Once this was right after the above failure, other times it occurred independently of the above failure.

Comment 8 pulp-infra@redhat.com 2016-08-02 19:30:33 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 9 pulp-infra@redhat.com 2016-08-02 19:30:36 UTC
The Pulp upstream bug priority is at Urgent. Updating the external tracker on this bug.

Comment 10 pulp-infra@redhat.com 2016-08-02 23:00:25 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 11 Justin Sherrill 2016-08-05 14:11:04 UTC
Workaround:

replace 

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.*/lib/katello/tasks/upgrades/3.0/add_export_distributor.rake

with these contents:

https://gist.githubusercontent.com/jlsherrill/fe10aebb0ca512d4c575232df2e26386/raw/983b2ed51ed18713a9f43e6627737ba04ac60132/export_distributor_update.txt


This change causes repositories to not be updated if they already were updated.  So you can run:

foreman-rake katello:upgrades:3.0:add_export_distributor 
service httpd restart

multiple times.  In my experience it would update about 500 repositories before hitting the issue, so if 1000 repos exist in the database it would take 2-3 runs to get them all.  Eventually it should finish them all.  Its very important to restart httpd after each run.

Comment 12 pulp-infra@redhat.com 2016-08-05 16:00:27 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 21 pulp-infra@redhat.com 2016-08-15 14:00:32 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 22 pulp-infra@redhat.com 2016-08-15 17:00:35 UTC
The Pulp upstream bug status is at VERIFIED. Updating the external tracker on this bug.

Comment 23 Mike McCune 2016-08-17 04:31:55 UTC
*** HOTFIX Packages Available ***

These hotfix packages resolve the apache segfault and subsequent "Pulp message bus connection issue"

Instructions for installation:

1) Download the attached hotfix package that corresponds to your version of Red Hat Enterprise Linux (.el7 or .el6)

2) rpm -UVh <python-kombu package you downloaded>

3) katello-service restart

4) continue operations

Comment 24 Mike McCune 2016-08-17 04:34:09 UTC
Created attachment 1191467 [details]
python-kombu EL6 hotfix package

Comment 25 Mike McCune 2016-08-17 04:34:50 UTC
Created attachment 1191468 [details]
python-kombu EL7 hotfix package

Comment 28 pulp-infra@redhat.com 2016-08-19 14:00:28 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 29 jcallaha 2016-08-19 19:48:42 UTC
Verified in Satellite 6.2.1 Snap 1.3

Steps:
Terminal 1 - for i in {0..199}; do hammer ping; done;
Terminal 2 - journalctl -f

monitor terminal 2 for segfaults

Results:
no segfaults seen while monitoring. I then reran the loop with 1,000 iterations and saw no segfaults.

Comment 31 errata-xmlrpc 2016-08-22 06:35:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1643


Note You need to log in before you can comment on or make changes to this bug.