Bug 1361196
| Summary: | Upgrade to 6.2 is failing due to Pulp message bus connection issue | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Andy MacMurray <amacmurr> | ||||||||||
| Component: | Upgrades | Assignee: | satellite6-bugs <satellite6-bugs> | ||||||||||
| Status: | CLOSED ERRATA | QA Contact: | jcallaha | ||||||||||
| Severity: | high | Docs Contact: | |||||||||||
| Priority: | high | ||||||||||||
| Version: | 6.2.0 | CC: | amacmurr, arahaman, bbuckingham, bmbouter, cduryee, chrobert, daviddavis, dkliban, ggainey, inecas, ipanova, jalviso, jcallaha, jsherril, ktordeur, mhrivnak, mmccune, ohadlevy, oshtaier, pcreech, rchan, sthirugn, ttereshc, xdmoon, zhunting | ||||||||||
| Target Milestone: | Unspecified | Keywords: | Triaged | ||||||||||
| Target Release: | Unused | ||||||||||||
| Hardware: | Unspecified | ||||||||||||
| OS: | Unspecified | ||||||||||||
| Whiteboard: | |||||||||||||
| Fixed In Version: | python-kombu-3.0.33-8 | Doc Type: | If docs needed, set a value | ||||||||||
| Doc Text: | Story Points: | --- | |||||||||||
| Clone Of: | Environment: | ||||||||||||
| Last Closed: | 2016-08-22 06:35:59 UTC | Type: | Bug | ||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||
| Documentation: | --- | CRM: | |||||||||||
| Verified Versions: | Category: | --- | |||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
| Embargoed: | |||||||||||||
| Attachments: |
|
||||||||||||
Created attachment 1185107 [details]
Production log
additional info from irc, this happens about six hours into the upgrade process. Also this happens on el6. It's unknown if this is also on el7. Could you upload the whole sosreport for further analysis? What options do you recommend running sosreport with? Also FYI - Ohad and Justin Sherrill have been looking at this instance and investigating. Thanks, - Andy To add a bit of information, at the time of the failure pulp seems to throw: qpidd[12237]: 2016-07-29 11:55:01 [System] error Error reading socket: Success(0) pulp: pulp.server.managers.status:ERROR: (9893-87936) [Errno 185090050] _ssl.c:330: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib pulp: pulp.server.managers.status:ERROR: (9893-87936) Traceback (most recent call last): pulp: pulp.server.managers.status:ERROR: (9893-87936) File "/usr/lib/python2.6/site-packages/pulp/server/managers/status.py", line 58, in get_broker_conn_status pulp: pulp.server.managers.status:ERROR: (9893-87936) conn.connect() pulp: pulp.server.managers.status:ERROR: (9893-87936) File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 237, in connect pulp: pulp.server.managers.status:ERROR: (9893-87936) return self.connection pulp: pulp.server.managers.status:ERROR: (9893-87936) File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 741, in connection pulp: pulp.server.managers.status:ERROR: (9893-87936) self._connection = self._establish_connection() pulp: pulp.server.managers.status:ERROR: (9893-87936) File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 696, in _establish_connection pulp: pulp.server.managers.status:ERROR: (9893-87936) conn = self.transport.establish_connection() pulp: pulp.server.managers.status:ERROR: (9893-87936) File "/usr/lib/python2.6/site-packages/kombu/transport/qpid.py", line 1600, in establish_connection pulp: pulp.server.managers.status:ERROR: (9893-87936) conn = self.Connection(**opts) pulp: pulp.server.managers.status:ERROR: (9893-87936) File "/usr/lib/python2.6/site-packages/kombu/transport/qpid.py", line 1261, in __init__ pulp: pulp.server.managers.status:ERROR: (9893-87936) self._qpid_conn = establish(**self.connection_options) pulp: pulp.server.managers.status:ERROR: (9893-87936) File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 112, in establish pulp: pulp.server.managers.status:ERROR: (9893-87936) conn.open(timeout=timeout) pulp: pulp.server.managers.status:ERROR: (9893-87936) File "<string>", line 6, in open pulp: pulp.server.managers.status:ERROR: (9893-87936) File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 323, in open pulp: pulp.server.managers.status:ERROR: (9893-87936) self.attach(timeout=timeout) pulp: pulp.server.managers.status:ERROR: (9893-87936) File "<string>", line 6, in attach pulp: pulp.server.managers.status:ERROR: (9893-87936) File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 341, in attach pulp: pulp.server.managers.status:ERROR: (9893-87936) if not self._ewait(lambda: self._transport_connected and not self._unlinked(), timeout=timeout): pulp: pulp.server.managers.status:ERROR: (9893-87936) File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 274, in _ewait pulp: pulp.server.managers.status:ERROR: (9893-87936) self.check_error() pulp: pulp.server.managers.status:ERROR: (9893-87936) File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 267, in check_error pulp: pulp.server.managers.status:ERROR: (9893-87936) raise e pulp: pulp.server.managers.status:ERROR: (9893-87936) ConnectError: [Errno 185090050] _ssl.c:330: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib And then when i make requests to /pulp/api/v2/status/ the failure responses trigger: ERROR: (11965-28352) [Errno 24] Too many open files ERROR: (11965-28352) Traceback (most recent call last): ERROR: (11965-28352) File "/usr/lib/python2.6/site-packages/pulp/server/managers/status.py", line 58, in get_broker_conn_status ERROR: (11965-28352) conn.connect() ERROR: (11965-28352) File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 237, in connect ERROR: (11965-28352) return self.connection ERROR: (11965-28352) File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 741, in connection ERROR: (11965-28352) self._connection = self._establish_connection() ERROR: (11965-28352) File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 696, in _establish_connection ERROR: (11965-28352) conn = self.transport.establish_connection() ERROR: (11965-28352) File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 769, in transport ERROR: (11965-28352) self._transport = self.create_transport() ERROR: (11965-28352) File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 514, in create_transport ERROR: (11965-28352) return self.get_transport_cls()(client=self) ERROR: (11965-28352) File "/usr/lib/python2.6/site-packages/kombu/transport/qpid.py", line 1396, in __init__ ERROR: (11965-28352) self.r, self._w = os.pipe() ERROR: (11965-28352) OSError: [Errno 24] Too many open files This may be related to https://bugzilla.redhat.com/show_bug.cgi?id=1328984 In addition, on several occasions I've seen apache segfault: kernel: httpd[10215]: segfault at 28 ip 00007fd20e0cd11b sp 00007fd1da5f5cc0 error 4 in libapr-1.so.0.3.9[7fd20e0ae000+2b000] Once this was right after the above failure, other times it occurred independently of the above failure. The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug. The Pulp upstream bug priority is at Urgent. Updating the external tracker on this bug. The Pulp upstream bug status is at POST. Updating the external tracker on this bug. Workaround: replace /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.*/lib/katello/tasks/upgrades/3.0/add_export_distributor.rake with these contents: https://gist.githubusercontent.com/jlsherrill/fe10aebb0ca512d4c575232df2e26386/raw/983b2ed51ed18713a9f43e6627737ba04ac60132/export_distributor_update.txt This change causes repositories to not be updated if they already were updated. So you can run: foreman-rake katello:upgrades:3.0:add_export_distributor service httpd restart multiple times. In my experience it would update about 500 repositories before hitting the issue, so if 1000 repos exist in the database it would take 2-3 runs to get them all. Eventually it should finish them all. Its very important to restart httpd after each run. The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug. The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug. The Pulp upstream bug status is at VERIFIED. Updating the external tracker on this bug. *** HOTFIX Packages Available *** These hotfix packages resolve the apache segfault and subsequent "Pulp message bus connection issue" Instructions for installation: 1) Download the attached hotfix package that corresponds to your version of Red Hat Enterprise Linux (.el7 or .el6) 2) rpm -UVh <python-kombu package you downloaded> 3) katello-service restart 4) continue operations Created attachment 1191467 [details]
python-kombu EL6 hotfix package
Created attachment 1191468 [details]
python-kombu EL7 hotfix package
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug. Verified in Satellite 6.2.1 Snap 1.3
Steps:
Terminal 1 - for i in {0..199}; do hammer ping; done;
Terminal 2 - journalctl -f
monitor terminal 2 for segfaults
Results:
no segfaults seen while monitoring. I then reran the loop with 1,000 iterations and saw no segfaults.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1643 |
Created attachment 1185106 [details] Satellite log Description of problem: We've encountered two occurances of the upgrade failing due to Pulp message bus connection issues. [ INFO 2016-07-28 07:52:31 main] Elasticsearch has been removed as a dependency. The database files can be removed manually with #rm -rf /var/lib/elasticsearch. Some packages are no longer needed and can be removed: #rpm -e ruby193-rubygem-tire tfm-rubygem-tire elasticsearch sigar-java sigar snappy-java lucene4-contrib lucene4 [ INFO 2016-07-28 07:52:31 main] Upgrade Step: add_export_distributor (this may take a while) ... [ERROR 2016-07-28 09:00:12 main] rake aborted! There was an issue with the backend service pulp: Pulp message bus connection issue. /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/app/lib/actions/middleware/backend_services_check.rb:16:in `block in plan' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/app/lib/actions/middleware/backend_services_check.rb:14:in `each' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/app/lib/actions/middleware/backend_services_check.rb:14:in `plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/app/lib/actions/middleware/remote_action.rb:9:in `plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:34:in `plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:34:in `plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/world.rb:30:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:460:in `block (2 levels) in execute_plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:281:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:281:in `switch_flow' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:364:in `concurrence' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:459:in `block in execute_plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `block in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:458:in `execute_plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:260:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `block in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/plan_step.rb:53:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:392:in `plan_action' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/app/lib/actions/katello/repository/refresh_repostiory.rb:8:in `block in plan' /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:72:in `as' /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:78:in `as_anonymous_admin' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/app/lib/actions/katello/repository/refresh_repostiory.rb:6:in `plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:461:in `block (3 levels) in execute_plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:34:in `plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:34:in `plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/world.rb:30:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:460:in `block (2 levels) in execute_plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:281:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:281:in `switch_flow' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:364:in `concurrence' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:459:in `block in execute_plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `block in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:458:in `execute_plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:260:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `block in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/plan_step.rb:53:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:191:in `block (2 levels) in plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:281:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:281:in `switch_flow' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:271:in `with_planning_scope' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:190:in `block in plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:42:in `plan_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/common/transaction.rb:16:in `block in rollback_on_error' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `block in transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:209:in `within_new_transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `transaction' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in `transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/transaction_adapters/active_record.rb:5:in `transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/common/transaction.rb:15:in `rollback_on_error' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/common/transaction.rb:5:in `plan_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/world.rb:30:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan.rb:189:in `plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/world.rb:168:in `block in plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/world.rb:166:in `tap' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/world.rb:166:in `plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/world.rb:142:in `trigger' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.6/lib/foreman_tasks.rb:18:in `trigger' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.6/lib/foreman_tasks.rb:24:in `trigger_task' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.6/lib/foreman_tasks.rb:39:in `sync_task' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/lib/katello/tasks/upgrades/3.0/add_export_distributor.rake:9:in `block (5 levels) in <top (required)>' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `block (2 levels) in find_each' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `each' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `block in find_each' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:125:in `find_in_batches' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:51:in `find_each' /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/querying.rb:9:in `find_each' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.68/lib/katello/tasks/upgrades/3.0/add_export_distributor.rake:8:in `block (4 levels) in <top (required)>' Tasks: TOP => katello:upgrades:3.0:add_export_distributor (See full trace by running task with --trace) Refreshing existing repositories to add export distributor [ERROR 2016-07-28 09:00:12 main] Upgrade step add_export_distributor failed. Check logs for more information. [DEBUG 2016-07-28 09:00:12 main] Exit with status code: 1 (signal was 1) [DEBUG 2016-07-28 09:00:12 main] Cleaning /etc/katello-installer/d20160728-28117-eu5lyg [DEBUG 2016-07-28 09:00:12 main] Cleaning /tmp/default_values.yaml Version-Release number of selected component (if applicable): How reproducible: 2/2 so far. Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: