Bug 1205737 - Various errors in logs during upgrades due to services running
Summary: Various errors in logs during upgrades due to services running
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Upgrades
Version: 6.1.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: Unspecified
Assignee: Stephen Benjamin
QA Contact: Sachin Ghai
URL: http://projects.theforeman.org/issues...
Whiteboard:
: 1195463 1205693 (view as bug list)
Depends On:
Blocks: 1195450
TreeView+ depends on / blocked
 
Reported: 2015-03-25 14:46 UTC by Lukáš Hellebrandt
Modified: 2020-08-13 08:13 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-08-12 14:00:49 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
var-log-foreman-production.log (53.67 KB, text/plain)
2015-03-25 14:46 UTC, Lukáš Hellebrandt
no flags Details
hammer.log (1.06 KB, text/html)
2015-03-25 14:47 UTC, Lukáš Hellebrandt
no flags Details
hammer_serverside.log (52.69 KB, text/plain)
2015-03-25 14:47 UTC, Lukáš Hellebrandt
no flags Details
reg_serverside.log (76.97 KB, text/plain)
2015-03-25 14:47 UTC, Lukáš Hellebrandt
no flags Details
rhsm.log (10.42 KB, text/plain)
2015-03-25 14:48 UTC, Lukáš Hellebrandt
no flags Details
error in /var/log/messages on restarting services on sat6 (52.14 KB, text/plain)
2015-05-12 11:27 UTC, Sachin Ghai
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 10221 0 None None None 2016-04-22 16:41:52 UTC

Description Lukáš Hellebrandt 2015-03-25 14:46:41 UTC
Created attachment 1006345 [details]
var-log-foreman-production.log

Description of problem:
At least the following entities can't be listed: activation keys, products, repositories (in their tab and in content view), manifest.

In WebUI, all these pages show just "Loading" with infinite spinner. Also, Hammer can't list them.

Let AKs be an example:

WebUI - doesn't load and traceback occurs in /var/log/foreman/production.log (attached). In organization without any AKs, the page loads correctly but it's not possible to add any AK (clicking "save" has no effect).

I can't list AKs using "hammer activation-key list --organization-id <ID>": hammer.log shows output of the command, hammer_serverside.log is the server side log

The AK can't be used for content host registration, see reg_serverside.log , the command output is "Remote server error. Please check the connection details, or see /var/log/rhsm/rhsm.log for more information.". Also attaching the mentioned log.

The same/similar applies to other listed entity types.

Version-Release number of selected component (if applicable):
Compose 20150320.1

How reproducible:
Always

Steps to Reproduce:
1. Have a sat6.0.8 with some data
2. Add compose repo
3. yum update -y ; katello-service restart
4. See bug description and try yourself

Actual results:
Multiple entities can't be manipulated/listed. Traceback.

Expected results:
It should be possible. And we don't like tracebacks.

Comment 1 Lukáš Hellebrandt 2015-03-25 14:47:08 UTC
Created attachment 1006346 [details]
hammer.log

Comment 2 Lukáš Hellebrandt 2015-03-25 14:47:31 UTC
Created attachment 1006347 [details]
hammer_serverside.log

Comment 3 Lukáš Hellebrandt 2015-03-25 14:47:52 UTC
Created attachment 1006348 [details]
reg_serverside.log

Comment 4 Lukáš Hellebrandt 2015-03-25 14:48:18 UTC
Created attachment 1006349 [details]
rhsm.log

Comment 6 Stephen Benjamin 2015-04-15 11:46:59 UTC
In comment #1:
> 3. yum update -y ; katello-service restart

This is not the correct upgrade procedure, not sure if we have this documented for downstream users anywhere, but for 6.0 -> 6.1 you need to rerun the installer after doing `yum -y update`

  katello-installer --upgrade

I imagine nearly everything will be broken if you didn't do that.

Here's upstream docs about it:
 http://www.katello.org/docs/2.2/upgrade/index.html

Comment 8 Lukáš Hellebrandt 2015-04-15 20:18:20 UTC
Either I forgot to perform that step, or I forgot to mention it. When trying to reproduce, this bug still persisted after runing 'katello-installer --upgrade' after update. However, stopping the katello service before updating fixed it for me, i.e.:
"katelo-service stop; yum update -y; katello-installer --upgrade; katello-service restart"
worked and fixed the bug. Might only be my case, although I tried it multiple times.

So for me, if stopping katello-service before update is a correct procedure, this is a NOTABUG.

Comment 9 Stephen Benjamin 2015-04-21 11:32:08 UTC
Created redmine issue http://projects.theforeman.org/issues/10221 from this bug

Comment 10 Stephen Benjamin 2015-04-21 11:40:05 UTC
*** Bug 1205693 has been marked as a duplicate of this bug. ***

Comment 11 Bryan Kearney 2015-05-04 20:05:45 UTC
Moving to POST since upstream bug http://projects.theforeman.org/issues/10221 has been closed
-------------
Anonymous
Applied in changeset commit:katello-installer|9150d2751f2dcfd5ba29734317bf4d943a37489f.

Comment 14 Sachin Ghai 2015-05-08 12:21:26 UTC
Verified with setup upgraded from sat6.0.8 --> sat6.1 GA compose snap3 compose2


I'm assuming here I need to restart the katello-service after upgrade. All services restarted successfully

[root@sat6-]# katello-service restart
Stopping Qpid AMQP daemon:                                 [  OK  ]
Starting Qpid AMQP daemon:                                 [  OK  ]
Shutting down qdrouterd services:                          [  OK  ]
Starting qdrouterd services:                               [  OK  ]
Stopping elasticsearch:                                    [  OK  ]
Starting elasticsearch:                                    [  OK  ]
Stopping tomcat6:                                          [  OK  ]
Starting tomcat6:                                          [  OK  ]
celery multi v3.1.11 (Cipater)
> Stopping nodes...
	> reserved_resource_worker-1.redhat.com: QUIT -> 28656
	> reserved_resource_worker-2.redhat.com: QUIT -> 28587
	> reserved_resource_worker-3.redhat.com: QUIT -> 28627
	> reserved_resource_worker-0.redhat.com: QUIT -> 28566
> Waiting for 4 nodes -> 28656, 28587, 28627, 28566........
	> reserved_resource_worker-1.redhat.com: OK
> Restarting node reserved_resource_worker-1.redhat.com: OK
> Waiting for 3 nodes -> 28587, 28627, 28566....
	> reserved_resource_worker-2.redhat.com: OK
> Restarting node reserved_resource_worker-2.redhat.com: OK
> Waiting for 2 nodes -> 28627, 28566....
	> reserved_resource_worker-3.redhat.com: OK
> Restarting node reserved_resource_worker-3.redhat.com: OK
> Waiting for 1 node -> 28566....
	> reserved_resource_worker-0.redhat.com: OK
> Restarting node reserved_resource_worker-0.redhat.com: OK

celery init v10.0.
Using config script: /etc/default/pulp_workers
Stopping foreman-proxy:                                    [  OK  ]
Starting foreman-proxy:                                    [  OK  ]
celery multi v3.1.11 (Cipater)
> Stopping nodes...
	> resource_manager.redhat.com: QUIT -> 28753
> Waiting for 1 node -> 28753.....
	> resource_manager.redhat.com: OK
> Restarting node resource_manager.redhat.com: OK

celery init v10.0.
Using config script: /etc/default/pulp_resource_manager
Stopping mongod:                                           [  OK  ]
Starting mongod:                                           [  OK  ]
Waiting for mongod to become available:                    [  OK  ]
celery init v10.0.
Using configuration: /etc/default/pulp_workers, /etc/default/pulp_celerybeat
Restarting celery periodic task scheduler
Stopping pulp_celerybeat... OK
Starting pulp_celerybeat...
[Fri May 08 08:16:53 2015] [warn] module passenger_module is already loaded, skipping
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]
Starting foreman-tasks:                                    [  OK  ]
Success!

Comment 15 Sachin Ghai 2015-05-08 12:22:28 UTC
But with restart, I found these errors in logs:

May  8 08:15:30 sat6-qe-rhel66 qpidd[27906]: 2015-05-08 08:15:30 [Security] error Rejected un-encrypted connection.
May  8 08:15:30 sat6-qe-rhel66 qpidd[27906]: 2015-05-08 08:15:30 [Protocol] error Connection qpid.127.0.0.1:5672-127.0.0.1:60133 closed by error: connection-forced: Connection must be encrypted.(320)
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28566-09632) consumer: Connection to broker lost. Trying to re-establish the connection...
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28566-09632) Traceback (most recent call last):
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28566-09632)   File "/usr/lib/python2.6/site-packages/celery/worker/consumer.py", line 278, in start
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28566-09632)     blueprint.start(self)
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28566-09632)   File "/usr/lib/python2.6/site-packages/celery/bootsteps.py", line 123, in start
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28566-09632)     step.start(parent)
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28566-09632)   File "/usr/lib/python2.6/site-packages/celery/worker/consumer.py", line 821, in start
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28566-09632)     c.loop(*c.loop_args())
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28566-09632)   File "/usr/lib/python2.6/site-packages/celery/worker/loops.py", line 72, in asynloop
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28566-09632)     next(loop)
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28566-09632)   File "/usr/lib/python2.6/site-packages/kombu/async/hub.py", line 324, in create_loop
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28566-09632)     cb(*cbargs)
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28566-09632)   File "/usr/lib/python2.6/site-packages/kombu/transport/qpid.py", line 1559, in on_readable
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28566-09632)     raise self.session.saved_exception
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28566-09632) ConnectionError: connection aborted
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28656-29568) consumer: Connection to broker lost. Trying to re-establish the connection...
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28656-29568) Traceback (most recent call last):
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28656-29568)   File "/usr/lib/python2.6/site-packages/celery/worker/consumer.py", line 278, in start
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28656-29568)     blueprint.start(self)
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28656-29568)   File "/usr/lib/python2.6/site-packages/celery/bootsteps.py", line 123, in start
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28656-29568)     step.start(parent)
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28656-29568)   File "/usr/lib/python2.6/site-packages/celery/worker/consumer.py", line 821, in start
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28656-29568)     c.loop(*c.loop_args())
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28656-29568)   File "/usr/lib/python2.6/site-packages/celery/worker/loops.py", line 72, in asynloop
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28656-29568)     next(loop)
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28656-29568)   File "/usr/lib/python2.6/site-packages/kombu/async/hub.py", line 324, in create_loop
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28656-29568)     cb(*cbargs)
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28656-29568)   File "/usr/lib/python2.6/site-packages/kombu/transport/qpid.py", line 1559, in on_readable
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28656-29568)     raise self.session.saved_exception
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28656-29568) ConnectionError: connection aborted
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28627-29472) consumer: Connection to broker lost. Trying to re-establish the connection...
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28627-29472) Traceback (most recent call last):
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28627-29472)   File "/usr/lib/python2.6/site-packages/celery/worker/consumer.py", line 278, in start
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28627-29472)     blueprint.start(self)
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28627-29472)   File "/usr/lib/python2.6/site-packages/celery/bootsteps.py", line 123, in start
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28627-29472)     step.start(parent)
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28627-29472)   File "/usr/lib/python2.6/site-packages/celery/worker/consumer.py", line 821, in start
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28627-29472)     c.loop(*c.loop_args())
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28627-29472)   File "/usr/lib/python2.6/site-packages/celery/worker/loops.py", line 72, in asynloop
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28627-29472)     next(loop)
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28627-29472)   File "/usr/lib/python2.6/site-packages/kombu/async/hub.py", line 324, in create_loop
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28587-29248) consumer: Connection to broker lost. Trying to re-establish the connection...
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28587-29248) Traceback (most recent call last):
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28587-29248)   File "/usr/lib/python2.6/site-packages/celery/worker/consumer.py", line 278, in start
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28587-29248)     blueprint.start(self)
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28587-29248)   File "/usr/lib/python2.6/site-packages/celery/bootsteps.py", line 123, in start
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28587-29248)     step.start(parent)
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28587-29248)   File "/usr/lib/python2.6/site-packages/celery/worker/consumer.py", line 821, in start
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28587-29248)     c.loop(*c.loop_args())
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28587-29248)   File "/usr/lib/python2.6/site-packages/celery/worker/loops.py", line 72, in asynloop
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28587-29248)     next(loop)
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28587-29248)   File "/usr/lib/python2.6/site-packages/kombu/async/hub.py", line 324, in create_loop
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28587-29248)     cb(*cbargs)
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28587-29248)   File "/usr/lib/python2.6/site-packages/kombu/transport/qpid.py", line 1559, in on_readable
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28587-29248)     raise self.session.saved_exception
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28587-29248) ConnectionError: connection aborted
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28627-29472)     cb(*cbargs)
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28627-29472)   File "/usr/lib/python2.6/site-packages/kombu/transport/qpid.py", line 1559, in on_readable
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28627-29472)     raise self.session.saved_exception
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28627-29472) ConnectionError: connection aborted
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28753-83264) consumer: Connection to broker lost. Trying to re-establish the connection...
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28753-83264) Traceback (most recent call last):
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28753-83264)   File "/usr/lib/python2.6/site-packages/celery/worker/consumer.py", line 278, in start
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28753-83264)     blueprint.start(self)
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28753-83264)   File "/usr/lib/python2.6/site-packages/celery/bootsteps.py", line 123, in start
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28753-83264)     step.start(parent)
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28753-83264)   File "/usr/lib/python2.6/site-packages/celery/worker/consumer.py", line 821, in start
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28753-83264)     c.loop(*c.loop_args())
May  8 08:15:30 sat6-qe-rhel66 pulp: pulp.server.async.scheduler:ERROR: connection aborted
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28753-83264)   File "/usr/lib/python2.6/site-packages/celery/worker/loops.py", line 72, in asynloop
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28753-83264)     next(loop)
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28753-83264)   File "/usr/lib/python2.6/site-packages/kombu/async/hub.py", line 324, in create_loop
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28753-83264)     cb(*cbargs)
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28753-83264)   File "/usr/lib/python2.6/site-packages/kombu/transport/qpid.py", line 1559, in on_readable
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28753-83264)     raise self.session.saved_exception
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:WARNING: (28753-83264) ConnectionError: connection aborted
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:ERROR: (28753-83264) consumer: Cannot connect to qpid://guest.redhat.com:5671//: [Errno 104] Connection reset by peer.
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:ERROR: (28656-29568) consumer: Cannot connect to qpid://guest.redhat.com:5671//: [Errno 104] Connection reset by peer.
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:ERROR: (28627-29472) consumer: Cannot connect to qpid://guest.redhat.com:5671//: [Errno 104] Connection reset by peer.
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:ERROR: (28627-29472) Trying again in 2.00 seconds...
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:ERROR: (28627-29472) 
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:ERROR: (28656-29568) Trying again in 2.00 seconds...
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:ERROR: (28656-29568) 
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:ERROR: (28753-83264) Trying again in 2.00 seconds...
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:ERROR: (28753-83264) 
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:ERROR: (28587-29248) consumer: Cannot connect to qpid://guest.redhat.com:5671//: [Errno 111] Connection refused.
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:ERROR: (28587-29248) Trying again in 2.00 seconds...
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:ERROR: (28587-29248) 
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:ERROR: (28566-09632) consumer: Cannot connect to qpid://guest.redhat.com:5671//: [Errno 111] Connection refused.
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:ERROR: (28566-09632) Trying again in 2.00 seconds...
May  8 08:15:30 sat6-qe-rhel66 pulp: celery.worker.consumer:ERROR: (28566-09632) 
May  8 08:15:32 sat6-qe-rhel66 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism PLAIN
May  8 08:15:32 sat6-qe-rhel66 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism PLAIN
May  8 08:15:32 sat6-qe-rhel66 pulp: celery.worker.consumer:INFO: Connected to qpid://guest.redhat.com:5671//
May  8 08:15:32 sat6-qe-rhel66 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism PLAIN
May  8 08:15:32 sat6-qe-rhel66 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism PLAIN
May  8 08:15:32 sat6-qe-rhel66 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism PLAIN
May  8 08:15:32 sat6-qe-rhel66 pulp: celery.worker.consumer:INFO: Connected to qpid://guest.redhat.com:5671//
May  8 08:15:32 sat6-qe-rhel66 pulp: celery.worker.consumer:INFO: Connected to qpid://guest.redhat.com:5671//
May  8 08:15:32 sat6-qe-rhel66 pulp: celery.worker.consumer:INFO: Connected to qpid://guest.redhat.com:5671//
May  8 08:15:32 sat6-qe-rhel66 pulp: celery.worker.consumer:INFO: Connected to qpid://guest.redhat.com:5671//
May  8 08:15:32 sat6-qe-rhel66 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism PLAIN
May  8 08:15:32 sat6-qe-rhel66 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism PLAIN
May  8 08:15:32 sat6-qe-rhel66 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism PLAIN
May  8 08:15:32 sat6-qe-rhel66 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism PLAIN
May  8 08:15:32 sat6-qe-rhel66 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism PLAIN
May  8 08:15:40 sat6-qe-rhel66 pulp: gofer.messaging.adapter.qpid.connection:INFO: connecting: URL: ssl://sat6-qe-rhel66.usersys.redhat.com|SSL: ca: /etc/pki/katello/certs/katello-default-ca.crt|key: None|certificate: /etc/pki/katello/qpid_client_striped.crt|host-validation: False
May  8 08:15:40 sat6-qe-rhel66 pulp: gofer.messaging.adapter.qpid.connection:INFO: connected: ssl://sat6-qe-rhel66.usersys.redhat.com
May  8 08:15:40 sat6-qe-rhel66 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism PLAIN
May  8 08:15:40 sat6-qe-rhel66 pulp: pulp.server.async.scheduler:INFO: Event Monitor Starting

Comment 16 Sachin Ghai 2015-05-08 12:23:34 UTC
@Stephen: Hey could you please take a look at comment 14 and comment 15. Thanks

Comment 17 Stephen Benjamin 2015-05-11 07:33:41 UTC
Restarting services isn't required, the installer takes care of it, but you still shouldn't see errors.

Something odd is going on there with qpid, it's like it takes some time after the restart for it to enable SSL.

Can you see the problem if you do something like this outside of an upgrade?


for i in qpidd pulp_resource_manager pulp_workers pulp_celerybeat; do servive $i restart; done

Comment 18 Stephen Benjamin 2015-05-11 20:35:22 UTC
*** Bug 1195463 has been marked as a duplicate of this bug. ***

Comment 19 Sachin Ghai 2015-05-12 11:26:04 UTC
@Stephen: Yeah.. If I restart qpidd and all above mentioned services in comment17 on another sat6.1 server (which is not upgraded), I can see similar errors there. Please see attached logs there. I'll file seperate bz for that to track the issue since its not specific to upgrade.

Comment 20 Sachin Ghai 2015-05-12 11:27:01 UTC
Created attachment 1024540 [details]
error in /var/log/messages on restarting services on sat6

Comment 21 Sachin Ghai 2015-05-12 11:33:52 UTC
The reported issue in bz description is no longer reproducible after upgrading sat6.0.8 to sat6.1 GA snap3 c2.. And the issue stated in comment 15 is not specific to upgrade. Its anywayz reproducible on fresh sat6.1 GA snap3 setup. So will track that issue in new bz. 

Moving this bz to verified.

Comment 22 Sachin Ghai 2015-05-12 11:46:16 UTC
Bz filed for issue in comment 15
https://bugzilla.redhat.com/show_bug.cgi?id=1220771

Comment 23 Bryan Kearney 2015-08-11 13:35:31 UTC
This bug is slated to be released with Satellite 6.1.

Comment 24 Bryan Kearney 2015-08-12 14:00:49 UTC
This bug was fixed in version 6.1.1 of Satellite which was released on 12 August, 2015.

Comment 25 Juan Manuel Parrilla Madrid 2015-08-13 11:49:46 UTC
Hi there,

I am in the same situation, I use the public release 6.1 of Satellite and I give an error in the installation:

Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_mongo...
Upgrade Step: migrate_pulp...
Upgrade Step: start_httpd...
Upgrade Step: migrate_candlepin...
Upgrade Step: migrate_foreman...
Upgrade Step: Running installer...
Installing             Done                                               [100%] [....................................................................................]
  The full log is at /var/log/katello-installer/katello-installer.log
Upgrade Step: restart_services...
Upgrade step restart_services failed. Check logs for more information.
/usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/logger.rb:104:in `error': wrong number of arguments (44 for 1) (ArgumentError)
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/logger.rb:104:in `send'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/logger.rb:104:in `dump_buffer'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/logger.rb:104:in `each'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/logger.rb:104:in `dump_buffer'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/logger.rb:103:in `each'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/logger.rb:103:in `dump_buffer'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/logger.rb:94:in `dump_errors'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/logger.rb:99:in `dump_errors'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/exit_handler.rb:26:in `exit'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:122:in `exit'
	from /usr/share/katello-installer/hooks/post/30-upgrade.rb:34:in `fail_and_exit'
	from /usr/share/katello-installer/hooks/post/30-upgrade.rb:28:in `upgrade_step'
	from /usr/share/katello-installer/hooks/post/30-upgrade.rb:38:in `load'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/hook_context.rb:13:in `instance_exec'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/hook_context.rb:13:in `execute'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:49:in `execute'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:47:in `each'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:47:in `execute'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:339:in `run_installation'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/exit_handler.rb:24:in `call'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/exit_handler.rb:24:in `exit'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:122:in `exit'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:338:in `run_installation'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:109:in `execute'
	from /usr/lib/ruby/gems/1.8/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run'
	from /usr/lib/ruby/gems/1.8/gems/clamp-0.6.2/lib/clamp/command.rb:125:in `run'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:116:in `run'
	from /usr/sbin/katello-installer:46


I follow the sentence like documentation with repositories of 6.1 enabled:

"katelo-service stop; yum update -y; katello-installer --upgrade; katello-service restart"

when I try to restart the processes into sat 6.1, this is the output:



Stopping mongod:                                           [  OK  ]
Starting mongod:                                           [  OK  ]
Waiting for mongod to become available:                    [  OK  ]
Stopping Qpid AMQP daemon:                                 [  OK  ]
Starting Qpid AMQP daemon:                                 [  OK  ]
Shutting down qdrouterd services:                          [  OK  ]
Starting qdrouterd services:                               [  OK  ]
celery multi v3.1.11 (Cipater)
> Stopping nodes...
	> resource_manager.gasnatural.com: QUIT -> 21589
> Waiting for 1 node -> 21589.....
	> resource_manager.gasnatural.com: OK

celery multi v3.1.11 (Cipater)
> Starting nodes...
	> resource_manager.gasnatural.com: OK
celery init v10.0.
Using config script: /etc/default/pulp_resource_manager
celery init v10.0.
Using configuration: /etc/default/pulp_workers, /etc/default/pulp_celerybeat
Restarting celery periodic task scheduler
Stopping pulp_celerybeat... OK
Starting pulp_celerybeat...
celery multi v3.1.11 (Cipater)
> Stopping nodes...
	> reserved_resource_worker-0.gasnatural.com: QUIT -> 22008
	> reserved_resource_worker-3.gasnatural.com: QUIT -> 22091
	> reserved_resource_worker-2.gasnatural.com: QUIT -> 22062
	> reserved_resource_worker-1.gasnatural.com: QUIT -> 22033
> Waiting for 4 nodes -> 22008, 22091, 22062, 22033........
	> reserved_resource_worker-0.gasnatural.com: OK
> Waiting for 3 nodes -> 22091, 22062, 22033....
	> reserved_resource_worker-3.gasnatural.com: OK
> Waiting for 2 nodes -> 22062, 22033....
	> reserved_resource_worker-2.gasnatural.com: OK
> Waiting for 1 node -> 22033....
	> reserved_resource_worker-1.gasnatural.com: OK

celery multi v3.1.11 (Cipater)
> Starting nodes...
	> reserved_resource_worker-0.gasnatural.com: OK
	> reserved_resource_worker-1.gasnatural.com: OK
	> reserved_resource_worker-2.gasnatural.com: OK
	> reserved_resource_worker-3.gasnatural.com: OK
celery init v10.0.
Using config script: /etc/default/pulp_workers
Stopping elasticsearch:                                    [  OK  ]
Starting elasticsearch:                                    [  OK  ]
Stopping foreman-proxy:                                    [  OK  ]
Starting foreman-proxy:                                    [  OK  ]
Stopping tomcat6: waiting for processes 21438 to exit
killing 21438 which did not stop after 30 seconds          [WARNING]
                                                           [  OK  ]
Starting tomcat6:                                          [  OK  ]
Starting foreman-tasks:                                    [  OK  ]
[Thu Aug 13 13:47:13 2015] [warn] module passenger_module is already loaded, skipping
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]
Some services failed: tomcat6


Any ideas?

Kind regards


Note You need to log in before you can comment on or make changes to this bug.