Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1220771 - On restarting the qpidd and other pulp services on Satellite-server raises some error related to qpid/pulp along with traceback
Summary: On restarting the qpidd and other pulp services on Satellite-server raises so...
Keywords:
Status: CLOSED DUPLICATE of bug 1269352
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Infrastructure
Version: 6.1.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: Unspecified
Assignee: Stephen Benjamin
QA Contact: Sachin Ghai
URL:
Whiteboard:
Depends On:
Blocks: 1195450
TreeView+ depends on / blocked
 
Reported: 2015-05-12 11:45 UTC by Sachin Ghai
Modified: 2019-07-11 09:08 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-10-15 13:40:02 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
error in /var/log/messages on restarting qpid and pulp services on sat6 (52.14 KB, text/plain)
2015-05-12 11:45 UTC, Sachin Ghai 		no flags Details
View All

Description Sachin Ghai 2015-05-12 11:45:30 UTC 
Created attachment 1024548 [details]
error in /var/log/messages on restarting qpid and pulp services on sat6

Description of problem:
On restarting following services on satellite6 server throws following error in /var/log/messages:


May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:WARNING: (19753-25856) ConnectionError: connection aborted
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:WARNING: (19678-84224) consumer: Connection to broker lost. Trying to re-establish the connection...
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:WARNING: (19678-84224) Traceback (most recent call last):
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:WARNING: (19678-84224)   File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 278, in start
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:WARNING: (19678-84224)     blueprint.start(self)
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:WARNING: (19678-84224)   File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:WARNING: (19678-84224)     step.start(parent)
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:WARNING: (19678-84224)   File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 821, in start
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:WARNING: (19678-84224)     c.loop(*c.loop_args())
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:WARNING: (19678-84224)   File "/usr/lib/python2.7/site-packages/celery/worker/loops.py", line 72, in asynloop
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:WARNING: (19678-84224)     next(loop)
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:WARNING: (19678-84224)   File "/usr/lib/python2.7/site-packages/kombu/async/hub.py", line 324, in create_loop
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:WARNING: (19678-84224)     cb(*cbargs)
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:WARNING: (19678-84224)   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1559, in on_readable
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:WARNING: (19678-84224)     raise self.session.saved_exception
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:WARNING: (19678-84224) ConnectionError: connection aborted
May 12 16:50:58 dhcp207-123 pulp: pulp.server.async.scheduler:ERROR: connection aborted
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:ERROR: (19747-23680) consumer: Cannot connect to qpid://guest.eng.pnq.redhat.com:5671//: [Errno 104] Connection reset by peer.
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:ERROR: (19747-23680) Trying again in 2.00 seconds...
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:ERROR: (19747-23680)
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:ERROR: (19745-76192) consumer: Cannot connect to qpid://guest.eng.pnq.redhat.com:5671//: [Errno 104] Connection reset by peer.
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:ERROR: (19745-76192) Trying again in 2.00 seconds...
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:ERROR: (19753-25856) consumer: Cannot connect to qpid://guest.eng.pnq.redhat.com:5671//: [Errno 104] Connection reset by peer.
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:ERROR: (19745-76192)
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:ERROR: (19753-25856) Trying again in 2.00 seconds...
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:ERROR: (19753-25856)
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:ERROR: (19743-40512) consumer: Cannot connect to qpid://guest.eng.pnq.redhat.com:5671//: [Errno 104] Connection reset by peer.
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:ERROR: (19743-40512) Trying again in 2.00 seconds...
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:ERROR: (19749-68192) consumer: Cannot connect to qpid://guest.eng.pnq.redhat.com:5671//: [Errno 104] Connection reset by peer.
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:ERROR: (19678-84224) consumer: Cannot connect to qpid://guest.eng.pnq.redhat.com:5671//: [Errno 104] Connection reset by peer.
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:ERROR: (19749-68192) Trying again in 2.00 seconds...
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:ERROR: (19751-89888) consumer: Cannot connect to qpid://guest.eng.pnq.redhat.com:5671//: [Errno 104] Connection reset by peer.
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:ERROR: (19678-84224) Trying again in 2.00 seconds...
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:ERROR: (19749-68192)
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:ERROR: (19751-89888) Trying again in 2.00 seconds...
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:ERROR: (19678-84224)
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:ERROR: (19751-89888)
May 12 16:50:58 dhcp207-123 pulp: celery.worker.consumer:ERROR: (19743-40512)
May 12 16:50:58 dhcp207-123 qdrouterd: Tue May 12 16:50:58 2015 ROUTER (info) Removing Prefix 'pulp.' for routed links to 'broker'
May 12 16:50:58 dhcp207-123 qdrouterd: Tue May 12 16:50:58 2015 ROUTER (info) Removing Prefix 'qmf.' for routed links to 'broker'
May 12 16:50:58 dhcp207-123 systemd: Starting An AMQP message broker daemon....
May 12 16:50:58 dhcp207-123 systemd: Started An AMQP message broker daemon..
May 12 16:50:58 dhcp207-123 systemd: Stopping Pulp Resource Manager...
May 12 16:50:58 dhcp207-123 celery: Please enter your password: Please enter your password: Please enter your password:
May 12 16:50:58 dhcp207-123 celery: worker: Warm shutdown (MainProcess)
May 12 16:50:59 dhcp207-123 celery: resource_manager.eng.pnq.redhat.com ready.

Version-Release number of selected component (if applicable):
sat6.1 GA snap3 

How reproducible:
always

Steps to Reproduce:
1.  for i in qpidd pulp_resource_manager pulp_workers pulp_celerybeat; do service $i restart; done
2.
3.

Actual results:
errors in /var/log/messages

Expected results:


Additional info:
Comment 2 Brad Buckingham 2015-05-12 17:10:27 UTC 
With Satellite 6.1, the proper way to restart services on the capsule will be via 'katello-service restart'.  Do you see the same errors when using that tool?

If not, I'd recommend we close this one out as user could get errors restarting individual services.
Comment 4 Stephen Benjamin 2015-06-08 12:20:19 UTC 
I don't see this on the latest GA snap 7, it all seems to be working fine, can you check again if the problem's gone away?
Comment 5 Sachin Ghai 2015-06-08 12:38:33 UTC 
I'll re-test with snap7. thanks.
Comment 6 Sachin Ghai 2015-06-09 06:06:07 UTC 
I can see error related to qpid with sat6.1 ga snap7

--
Jun  9 02:02:19 cloud-qe-14 pulp: celery.worker.consumer:ERROR: (7740-47776) consumer: Cannot connect to qpid://guest.lab.eng.bos.redhat.com:5671//: [Errno 104] Connection reset by peer.
Jun  9 02:02:19 cloud-qe-14 pulp: celery.worker.consumer:ERROR: (7743-02272) consumer: Cannot connect to qpid://guest.lab.eng.bos.redhat.com:5671//: [Errno 104] Connection reset by peer.
Jun  9 02:02:19 cloud-qe-14 pulp: celery.worker.consumer:ERROR: (7748-30816) consumer: Cannot connect to qpid://guest.lab.eng.bos.redhat.com:5671//: [Errno 104] Connection reset by peer.
Jun  9 02:02:19 cloud-qe-14 pulp: celery.worker.consumer:ERROR: (7743-02272) Trying again in 2.00 seconds...
Jun  9 02:02:19 cloud-qe-14 pulp: celery.worker.consumer:ERROR: (7740-47776) Trying again in 2.00 seconds...
Jun  9 02:02:19 cloud-qe-14 pulp: celery.worker.consumer:ERROR: (7748-30816) Trying again in 2.00 seconds...
Jun  9 02:02:19 cloud-qe-14 pulp: celery.worker.consumer:ERROR: (7743-02272)
Jun  9 02:02:19 cloud-qe-14 pulp: celery.worker.consumer:ERROR: (7740-47776)
Jun  9 02:02:19 cloud-qe-14 pulp: celery.worker.consumer:ERROR: (7748-30816)
Jun  9 02:02:19 cloud-qe-14 pulp: celery.worker.consumer:ERROR: (7679-66816) consumer: Cannot connect to qpid://guest.lab.eng.bos.redhat.com:5671//: [Errno 104] Connection reset by peer.
Jun  9 02:02:19 cloud-qe-14 pulp: celery.worker.consumer:ERROR: (7745-52384) consumer: Cannot connect to qpid://guest.lab.eng.bos.redhat.com:5671//: [Errno 104] Connection reset by peer.
Jun  9 02:02:19 cloud-qe-14 pulp: celery.worker.consumer:ERROR: (7745-52384) Trying again in 2.00 seconds...
Jun  9 02:02:19 cloud-qe-14 pulp: celery.worker.consumer:ERROR: (7745-52384)
Jun  9 02:02:19 cloud-qe-14 qdrouterd: Tue Jun  9 02:02:19 2015 ROUTER (info) Removing Prefix 'pulp.' for routed links to 'broker'
Jun  9 02:02:19 cloud-qe-14 qdrouterd: Tue Jun  9 02:02:19 2015 ROUTER (info) Removing Prefix 'qmf.' for routed links to 'broker'
Jun  9 02:02:19 cloud-qe-14 pulp: celery.worker.consumer:ERROR: (7679-66816) Trying again in 2.00 seconds...
Jun  9 02:02:19 cloud-qe-14 pulp: celery.worker.consumer:ERROR: (7679-66816)
Comment 7 Stephen Benjamin 2015-06-09 14:29:28 UTC 
Oh I see! Sorry, I misunderstood.  This is expected.  qpidd is restarted, so the workers lose their connection.

You could avoid the errors by doing this in three steps:

1. for i in pulp_resource_manager pulp_workers pulp_celerybeat; do service $i stop; done

2. service qpidd restart

3. for i in pulp_resource_manager pulp_workers pulp_celerybeat; do service $i start; done


I'll leave this as a bug but for sat-future if that's OK.  We could modify katello-service to do things in exactly this order, but the errors don't mean much, the workers lose their connections and then get restarted and reconnect and everything is healthy afterwards.
Comment 9 Stephen Benjamin 2015-10-15 13:40:02 UTC 
BZ1269352 should fix the ordering so stop/start/restart all work as expected

*** This bug has been marked as a duplicate of bug 1269352 ***
Note You need to log in before you can comment on or make changes to this bug.