Bug 1243979 - celery startup never returns if mongodb is down
Summary: celery startup never returns if mongodb is down
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Pulp
Version: 6.1.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium vote
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Lukas Pramuk
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-07-16 18:28 UTC by Adam Price
Modified: 2019-09-26 14:46 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-27 09:16:39 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Pulp Redmine 1290 Normal CLOSED - CURRENTRELEASE missing DB during boot causes resource manager to hang 2016-03-23 18:30:45 UTC
Red Hat Product Errata RHBA-2016:1501 normal SHIPPED_LIVE Red Hat Satellite 6.2 Capsule and Server 2016-07-27 12:28:58 UTC

Description Adam Price 2015-07-16 18:28:09 UTC
Description of problem:


Version-Release number of selected component (if applicable):
Satellite 6.1 Snap12


How reproducible:

Steps to Reproduce:
1. Ensure mongod cannot start properly. For instance, shutdown mongod then create a bogus /var/lib/mongod/mongod.lock pid file.
2. $ katello-service restart

Actual results:

$ katello-service restart
Stopping Qpid AMQP daemon:                                 [  OK  ]
Starting Qpid AMQP daemon:                                 [  OK  ]
Shutting down qdrouterd services:                          [  OK  ]
Starting qdrouterd services:                               [  OK  ]
celery multi v3.1.11 (Cipater)
> Stopping nodes...
        > resource_manager@hostname: QUIT -> 9704
> Waiting for 1 node -> 9704.....
        > resource_manager@hostname: OK

celery multi v3.1.11 (Cipater)
> Starting nodes...
        > resource_manager@hostname: OK
celery init v10.0.
Using config script: /etc/default/pulp_resource_manager
Stopping mongod:                                           [  OK  ]
Starting mongod:                                           [  OK  ]
Waiting for mongod to become available:                    [FAILED]
Stopping elasticsearch:                                    [  OK  ]
Starting elasticsearch:                                    [  OK  ]
Stopping tomcat6:                                          [  OK  ]
Starting tomcat6:                                          [  OK  ]
Stopping foreman-proxy:                                    [  OK  ]
Starting foreman-proxy:                                    [  OK  ]
celery init v10.0.
Using configuration: /etc/default/pulp_workers, /etc/default/pulp_celerybeat
Restarting celery periodic task scheduler
Stopping pulp_celerybeat... OK
Starting pulp_celerybeat...
celery multi v3.1.11 (Cipater)
> Stopping nodes...
        > reserved_resource_worker-0@hostname: QUIT -> 9936
        > reserved_resource_worker-1@hostname: QUIT -> 9962
> Waiting for 2 nodes -> 9936, 9962......
        > reserved_resource_worker-0@hostname: OK
> Waiting for 1 node -> 9962....
        > reserved_resource_worker-1@hostname: OK

celery multi v3.1.11 (Cipater)
> Starting nodes...
        > reserved_resource_worker-0@hostname: No handlers could be found for logger "pulp.server.db.connection"

*** gets stuck here indefinitely until user CTRL-c's ***

Expected results:
Even if a database connection cannot be established the startup should still return.


Additional info:

Comment 1 RHEL Program Management 2015-07-16 18:34:17 UTC
Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

Comment 3 Michael Hrivnak 2015-10-01 14:54:59 UTC
Was this on RHEL6? When you respond, please also add that info to the upstream bug report (linked above under "External Trackers").

Comment 4 pulp-infra@redhat.com 2015-10-01 15:00:21 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 5 pulp-infra@redhat.com 2015-10-01 15:00:23 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 6 Adam Price 2015-10-05 14:13:07 UTC
this was occurring on EL6.

Comment 7 pulp-infra@redhat.com 2015-12-22 17:30:24 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 8 Bryan Kearney 2016-01-04 15:37:42 UTC
Moving this to POST. Please verify once the 2.8 version of pulp is incorporated into teh downstream build.

Comment 9 pulp-infra@redhat.com 2016-02-23 21:00:32 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 11 pulp-infra@redhat.com 2016-03-23 18:30:46 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 13 Lukas Pramuk 2016-05-11 09:35:00 UTC
VERIFIED.

@Sat6.2.0-Snap10.1
katello-service-3.0.0-3.el6sat.noarch

# chown -R root:root /var/lib/mongodb

# katello-service restart
Stopping httpd: [  OK  ]

celery multi v3.1.11 (Cipater)
> Stopping nodes...
	> reserved_resource_worker-3@<FQDN>: QUIT -> 29260
	> reserved_resource_worker-1@<FQDN>: QUIT -> 29220
	> reserved_resource_worker-2@<FQDN>: QUIT -> 29239
	> reserved_resource_worker-0@<FQDN>: QUIT -> 29205
> Waiting for 4 nodes -> 29260, 29220, 29239, 29205........
	> reserved_resource_worker-3@<FQDN>: OK
> Waiting for 3 nodes -> 29220, 29239, 29205....
	> reserved_resource_worker-1@<FQDN>: OK
> Waiting for 2 nodes -> 29239, 29205....
	> reserved_resource_worker-2@<FQDN>: OK
> Waiting for 1 node -> 29205....
	> reserved_resource_worker-0@<FQDN>: OK

celery init v10.0.
Using config script: /etc/default/pulp_workers
Stopping tomcat6: waiting for processes 28938 to exit
[  OK  ]
Stopping foreman-proxy: [  OK  ]
celery init v10.0.
Using configuration: /etc/default/pulp_workers, /etc/default/pulp_celerybeat
Stopping pulp_celerybeat... NOT RUNNING
celery multi v3.1.11 (Cipater)
> Stopping nodes...
	> resource_manager@<FQDN>: QUIT -> 29391
> Waiting for 1 node -> 29391.....
	> resource_manager@<FQDN>: OK

celery init v10.0.
Using config script: /etc/default/pulp_resource_manager
[  OK  ] Qpid AMQP daemon: [  OK  ]
Shutting down qdrouterd services: [  OK  ]

Stopping postgresql service: [  OK  ]
Success!
Starting postgresql service: [  OK  ]
Starting mongod: [  OK  ]
Waiting for mongod to become available: [FAILED]
Starting qdrouterd services: [  OK  ]
Starting Qpid AMQP daemon: [  OK  ]
celery multi v3.1.11 (Cipater)
> Starting nodes...
	> resource_manager@<FQDN>: OK
celery init v10.0.
Using config script: /etc/default/pulp_resource_manager
celery init v10.0.
Using configuration: /etc/default/pulp_workers, /etc/default/pulp_celerybeat
Starting pulp_celerybeat...
Starting foreman-proxy: [  OK  ]
Starting tomcat6: [  OK  ]
celery multi v3.1.11 (Cipater)
> Starting nodes...
	> reserved_resource_worker-0@<FQDN>: OK
	> reserved_resource_worker-1@<FQDN>: OK
	> reserved_resource_worker-2@<FQDN>: OK
	> reserved_resource_worker-3@<FQDN>: OK
celery init v10.0.
Using config script: /etc/default/pulp_workers
Starting foreman-tasks: [  OK  ]
[Wed May 11 05:20:56 2016] [warn] module passenger_module is already loaded, skipping
Starting httpd: [  OK  ]
Some services failed to start: mongod

# echo $?
1

>>> if mongo fails to start then katello-service doesn't wait indefinitely for failing service start and informs user about it.

Comment 14 Lukas Pramuk 2016-05-11 09:37:02 UTC
pulp-server-2.8.1.3-1.el6sat.noarch ( /etc/rc.d/init.d/pulp_celerybeat )

Comment 16 errata-xmlrpc 2016-07-27 09:16:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501


Note You need to log in before you can comment on or make changes to this bug.