1128299 – Possible race condition when trying to re-run katello-installer while dynflow (sync?) tasks exist.

Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1128299 - Possible race condition when trying to re-run katello-installer while dynflow (sync?) tasks exist.

Summary: Possible race condition when trying to re-run katello-installer while dynflow...

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Red Hat Satellite
Classification:	Red Hat
Component:	Installation
Sub Component:
Version:	6.0.3
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	Unspecified
Assignee:	Ivan Necas
QA Contact:	Tazim Kolhar
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2014-08-08 20:25 UTC by Corey Welton
Modified:	2017-05-30 12:12 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2015-08-12 13:57:32 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Corey Welton 2014-08-08 20:25:13 UTC

Description of problem:
Trying to re-run katello-installer while some hung sync tasks are running seems to have caused my installer to hang indefinitely.

https://bugzilla.redhat.com/show_bug.cgi?id=1128296				
Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.  Start some sync processes, and possibly have them hang, per bug #1128296				
2.  Re-run katello-installer (perhaps to fix a bad proxy URL or something)
3.  View results

Actual results:
Installer hangs at 

Installing             Debug: Executing '/usr/share/katello-installer/mod [99%] [.......................................................................... ]  

Relevant info in installer log:

[DEBUG 2014-08-08 15:36:45 main]  Executing 'service pulp_workers status | grep "node reserved_resource_worker"'
[DEBUG 2014-08-08 15:36:45 main]  Executing '/usr/share/katello-installer/modules/service_wait/bin/service-wait pulp_workers restart'


Some interesting messages in pulp log:




Aug  8 15:37:03 beast rsyslogd-2177: imuxsock lost 19 messages from pid 27503 due to rate-limiting
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR: There are no Celery workers found in the system for reserved task work. Please ensure that there is at least one Celery worker running, and that the celerybeat service is also running.
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR: Traceback (most recent call last):
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/exception.py", line 44, in __call__
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return self.app(environ, start_response)
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/postponed.py", line 42, in __call__
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return self.app(environ, start_response)
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 279, in wsgi
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     result = self.handle_with_processors()
Aug  8 15:37:03 beast pulp: celery.worker.job:ERROR: Task pulp.server.async.tasks._reserve_resource[d288a834-04a5-44fa-93a4-d29565393ab7] raised unexpected: NoWorkers()
Aug  8 15:37:03 beast pulp: celery.worker.job:ERROR: Traceback (most recent call last):
Aug  8 15:37:03 beast pulp: celery.worker.job:ERROR:   File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task
Aug  8 15:37:03 beast pulp: celery.worker.job:ERROR:     R = retval = fun(*args, **kwargs)
Aug  8 15:37:03 beast pulp: celery.worker.job:ERROR:   File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__
Aug  8 15:37:03 beast pulp: celery.worker.job:ERROR:     return self.run(*args, **kwargs)
Aug  8 15:37:03 beast pulp: celery.worker.job:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 126, in _reserve_resource
Aug  8 15:37:03 beast pulp: celery.worker.job:ERROR:     reserved_resource.assigned_queue = resources.get_least_busy_worker().queue_name
Aug  8 15:37:03 beast pulp: celery.worker.job:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/managers/resources.py", line 39, in get_least_busy_worker
Aug  8 15:37:03 beast pulp: celery.worker.job:ERROR:     raise exceptions.NoWorkers()
Aug  8 15:37:03 beast pulp: celery.worker.job:ERROR: NoWorkers: There are no Celery workers found in the system for reserved task work. Please ensure that there is at least one Celery worker running, and that the celerybeat service is also running.
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 29, in _handle_with_processors
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return process(self.processors)
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 566, in processor
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return handler()
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 581, in processor
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     result = handler()
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 28, in process
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return self.handle()
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 230, in handle
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return self._delegate(fn, self.fvars, args)
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 422, in _delegate
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return f()
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 430, in <lambda>
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     f = lambda: self._delegate_sub_application(pat, what)
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 455, in _delegate_sub_application
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return app.handle_with_processors()
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 29, in _handle_with_processors
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return process(self.processors)
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 566, in processor
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return handler()
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 581, in processor
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     result = handler()
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 28, in process
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return self.handle()
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 230, in handle
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return self._delegate(fn, self.fvars, args)
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 420, in _delegate
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return handle_class(cls)
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 396, in handle_class
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return tocall(*args)
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/decorators.py", line 227, in _auth_decorator
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     value = method(self, *args, **kwargs)
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/repositories.py", line 732, in POST
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     async_result = repository.publish(repo_id, distributor_id, overrides)
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/celery/local.py", line 167, in <lambda>
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     __call__ = lambda x, *a, **kw: x._get_current_object()(*a, **kw)
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 306, in __call__
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return super(Task, self).__call__(*args, **kwargs)
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/celery/app/task.py", line 420, in __call__
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return self.run(*args, **kwargs)
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/tasks/repository.py", line 194, in publish
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     RESOURCE_REPOSITORY_TYPE, repo_id, tags=tags, kwargs=kwargs)
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 234, in apply_async_with_reservation
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     queue = _reserve_resource.apply_async((resource_id,), queue=RESOURCE_MANAGER_QUEUE).get()
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/celery/result.py", line 169, in get
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     no_ack=no_ack,
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/celery/backends/base.py", line 215, in wait_for
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     raise result
Aug  8 15:37:03 beast pulp: pulp.server.webservices.middleware.exception:ERROR: NoWorkers: There are no Celery workers found in the system for reserved task work. Please ensure that there is at least one Celery worker running, and that the celerybeat service is also running.
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR: There are no Celery workers found in the system for reserved task work. Please ensure that there is at least one Celery worker running, and that the celerybeat service is also running.
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR: Traceback (most recent call last):
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/exception.py", line 44, in __call__
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return self.app(environ, start_response)
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/postponed.py", line 42, in __call__
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return self.app(environ, start_response)
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 279, in wsgi
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     result = self.handle_with_processors()
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 29, in _handle_with_processors
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return process(self.processors)
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 566, in processor
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return handler()
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 581, in processor
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     result = handler()
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 28, in process
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return self.handle()
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 230, in handle
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return self._delegate(fn, self.fvars, args)
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 422, in _delegate
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return f()
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 430, in <lambda>
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     f = lambda: self._delegate_sub_application(pat, what)
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 455, in _delegate_sub_application
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return app.handle_with_processors()
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 29, in _handle_with_processors
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return process(self.processors)
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 566, in processor
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return handler()
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 581, in processor
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     result = handler()
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda>
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return p(lambda: process(processors))
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 28, in process
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return self.handle()
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 230, in handle
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return self._delegate(fn, self.fvars, args)
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 420, in _delegate
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return handle_class(cls)
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/web/application.py", line 396, in handle_class
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return tocall(*args)
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/decorators.py", line 227, in _auth_decorator
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     value = method(self, *args, **kwargs)
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/repositories.py", line 732, in POST
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     async_result = repository.publish(repo_id, distributor_id, overrides)
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/celery/local.py", line 167, in <lambda>
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     __call__ = lambda x, *a, **kw: x._get_current_object()(*a, **kw)
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 306, in __call__
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return super(Task, self).__call__(*args, **kwargs)
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/celery/app/task.py", line 420, in __call__
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     return self.run(*args, **kwargs)
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/tasks/repository.py", line 194, in publish
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     RESOURCE_REPOSITORY_TYPE, repo_id, tags=tags, kwargs=kwargs)
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 234, in apply_async_with_reservation
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     queue = _reserve_resource.apply_async((resource_id,), queue=RESOURCE_MANAGER_QUEUE).get()
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/celery/result.py", line 169, in get
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     no_ack=no_ack,
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:   File "/usr/lib/python2.6/site-packages/celery/backends/base.py", line 215, in wait_for
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR:     raise result
Aug  8 15:37:04 beast pulp: pulp.server.webservices.middleware.exception:ERROR: NoWorkers: There are no Celery workers found in the system for reserved task work. Please ensure that there is at least one Celery worker running, and that the celerybeat service is also running.
Aug  8 15:54:18 beast puppet-agent[8460]: (/File[/var/lib/puppet/lib]) Could not evaluate: Could not retrieve information from environment production source(s) puppet://beast.idm.lab.bos.redhat.com/plugins
Aug  8 15:54:21 beast puppet-master[29082]: Compiled catalog for beast.idm.lab.bos.redhat.com in environment production in 0.02 seconds

Expected results:

So I think what is happening....?

* We restart katello-services
* Dynflow immediately tries to restart the sync tasks
* Sync tasks require workers, but works service hasn't restarted yet
* Workers service never restarts, possibly due to a race condition per the three bullet points above?
* katello-installer never returns



Additional info:

Comment 3 Ivan Necas 2015-04-23 08:01:27 UTC

There were significant improvements made both on dynflow as well as pulp tasking. Putting ON_QA as I was not able to reproduce the issue and there is quite fair chance it was resolved in the 6.1 as part of other fixes

Comment 4 Tazim Kolhar 2015-04-27 10:45:51 UTC

VERIFIED :

 rpm -qa | grep foreman
foreman-vmware-1.7.2.15-1.el7sat.noarch
foreman-postgresql-1.7.2.15-1.el7sat.noarch
qeblade36.rhq.lab.eng.bos.redhat.com-foreman-proxy-1.0-1.noarch
foreman-1.7.2.15-1.el7sat.noarch
foreman-proxy-1.7.2.4-1.el7sat.noarch
qeblade36.rhq.lab.eng.bos.redhat.com-foreman-client-1.0-1.noarch
ruby193-rubygem-foreman_docker-1.2.0.7-1.el7sat.noarch
ruby193-rubygem-foreman_hooks-0.3.7-2.el7sat.noarch
rubygem-hammer_cli_foreman_discovery-0.0.1.4-1.el7sat.noarch
foreman-selinux-1.7.2.13-1.el7sat.noarch
ruby193-rubygem-foreman_gutterball-0.0.1.9-1.el7sat.noarch
puppet-foreman_scap_client-0.3.3-7.el7sat.noarch
foreman-gce-1.7.2.15-1.el7sat.noarch
ruby193-rubygem-foreman-redhat_access-0.1.0-1.el7sat.noarch
ruby193-rubygem-foreman-tasks-0.6.12.3-1.el7sat.noarch
rubygem-hammer_cli_foreman_tasks-0.0.3.3-1.el7sat.noarch
foreman-debug-1.7.2.15-1.el7sat.noarch
foreman-libvirt-1.7.2.15-1.el7sat.noarch
foreman-compute-1.7.2.15-1.el7sat.noarch
ruby193-rubygem-foreman_discovery-2.0.0.9-1.el7sat.noarch
rubygem-hammer_cli_foreman_bootdisk-0.1.2.5-1.el7sat.noarch
foreman-ovirt-1.7.2.15-1.el7sat.noarch
rubygem-hammer_cli_foreman-0.1.4.7-1.el7sat.noarch
ruby193-rubygem-foreman_bootdisk-4.0.2.10-1.el7sat.noarch
qeblade36.rhq.lab.eng.bos.redhat.com-foreman-proxy-client-1.0-1.noarch

steps:
1.  Start some sync processes, and possibly have them hang, 				
2. ifdown eth0
3. restart the system
4.  Re-run katello-installer (perhaps to fix a bad proxy URL or something)

# katello-installer
Installing             Info: START 620                                    [0%] [Installing             Debug: /Stage[main]/Apache::Mod::Wsgi/File[wsgi.co [0%] [Installing             Debug: /Stage[main]/Apache::Mod::Alias/Apache::Mod [0%] [Installing             Debug: /Stage[main]/Foreman_proxy::Config/Foreman_ [0%] [Installing             Debug: /Stage[main]/Pulp::Config/File[/etc/pulp/se [0%] [Installing             Debug: /Stage[main]/Postgresql::Server::Config/Con [0%] [Installing             Debug: /Stage[main]/Apache::Mod::Ssl/Apache::Mod[s [0%] [Installing             Info: Applying configuration version '1430131294'  [0%] [Installing             Info: RESOURCE Apache::Mod[suexec]                 [0%] [Installing             Info: RESOURCE File[/var/lib/puppet/concat/bin/con [0%] [Installing             Info: RESOURCE Package[mongodb_client]             [0%] [Installing             Info: RESOURCE Apache::Vhost[foreman]              [1%] [Installing             Info: RESOURCE Apache::Mod[access_compat]          [3%] [Installing             Info: RESOURCE File[/etc/pki/katello]              [8%] [Installing             Info: RESOURCE Ca[katello-default-ca]              [9%] [Installing             Info: RESOURCE Privkey[/etc/pki/katello/private/ka [9%] [Installing             Info: RESOURCE Pubkey[/etc/puppet/client_cert.pem] [10%] Installing             Info: RESOURCE Package[elasticsearch]              [12%] Installing             Info: RESOURCE Cert[qeblade36.rhq.lab.eng.bos.redh [14%] Installing             Info: RESOURCE Concat_fragment[qdrouter+header.con [15%] Installing             Info: RESOURCE Package[foreman-proxy]              [16%] Installing             Info: RESOURCE File[expires.load]                  [21%] Installing             Info: RESOURCE File[authz_groupfile.load]          [22%] Installing             Info: RESOURCE Apache::Mod[ext_filter]             [31%] Installing             Info: RESOURCE Apache::Mod[dav_fs]                 [32%] Installing             Info: RESOURCE File[/var/lib/puppet/concat/_var_li [38%] Installing             Info: RESOURCE Exec[restart-systemd]               [39%] Installing             Info: RESOURCE File[ssl.load]                      [41%] Installing             Info: RESOURCE File[/var/cache/tomcat]             [45%] Installing             Info: RESOURCE File[/var/lib/puppet/concat/_etc_ht [49%] Installing             Info: RESOURCE Package[gutterball]                 [52%] Installing             Info: RESOURCE Cert[qeblade36.rhq.lab.eng.bos.redh [56%] Installing             Debug: /File[/var/lib/puppet/concat/_var_lib_pgsql [58%] Installing             Info: RESOURCE Postgresql::Server::Pg_hba_rule[all [62%] Installing             Debug: /Stage[main]/Postgresql::Server::Service/Po [63%] Installing             Debug: /Stage[main]/Gutterball::Database/Postgresq [64%] Installing             Debug: /Stage[main]/Foreman::Database::Postgresql/ [65%] Installing             Debug: /Stage[main]/Candlepin::Database::Postgresq [65%] Installing             Debug: /Stage[main]/Candlepin::Database::Postgresq [65%] Installing             Debug: /Stage[main]/Candlepin::Database::Postgresq [66%] Installing             Debug: /Stage[main]/Candlepin::Database::Postgresq [66%] Installing             Debug: Executing '/usr/bin/psql -d postgres -p 543 [66%] Installing             Debug: /Stage[main]/Gutterball::Database/Postgresq [67%] Installing             Debug: /Stage[main]/Candlepin::Database::Postgresq [68%] Installing             Debug: /Stage[main]/Candlepin::Database::Postgresq [69%] Installing             Debug: /Stage[main]/Gutterball::Database/Postgresq [71%] Installing             Debug: /Stage[main]/Gutterball::Database/Postgresq [73%] Installing             Info: RESOURCE File[dav_fs.conf]                   [75%] Installing             Debug: /Stage[main]/Foreman::Database::Postgresql/ [75%] Installing             Info: RESOURCE Cert[qeblade36.rhq.lab.eng.bos.redh [78%] Installing             Info: RESOURCE Pubkey[/etc/pki/katello/qpid_router [80%] Installing             Debug: /File[/etc/httpd/conf.d/authz_owner.load]/s [82%] Installing             Debug: /File[/etc/httpd/conf.d/logio.load]/seluser [82%] Installing             Debug: /File[/etc/httpd/conf.d/passenger.load]/sel [82%] Installing             Info: RESOURCE File[/etc/httpd/conf.d/05-foreman-s [83%] Installing             Info: RESOURCE Cert[qeblade36.rhq.lab.eng.bos.redh [84%] Installing             Info: RESOURCE Concat_fragment[puppet.conf+10-main [86%] Installing             Info: RESOURCE File[/var/lib/puppet/ssl/private_ke [90%] Installing             Debug: Prefetching crontab resources for cron      [90%] Installing             Info: RESOURCE User[foreman]                       [92%] Installing             Info: RESOURCE Foreman_config_entry[db_pending_mig [94%] Installing             Debug: Executing '/usr/bin/systemctl is-enabled pu [96%] Installing             Info: RESOURCE Service[puppetmaster]               [96%] Installing             Info: RESOURCE Foreman_proxy::Settings_file[puppet [96%] Installing             Info: RESOURCE File[/etc/foreman-proxy/settings.d/ [99%] Installing             Info: /Stage[main]/Foreman_proxy::Config/Foreman_p [99%] Installing             Notice: /Stage[main]/Foreman_proxy::Service/Servic [99%] Installing             Info: RESOURCE Cert[qeblade36.rhq.lab.eng.bos.redh [99%] Installing             Debug: /Stage[main]/Certs::Gutterball/Certs::Sslto [99%] Installing             Info: RESOURCE Exec[add-private-key-to-nss-db]     [99%] Installing             Info: RESOURCE Privkey[/etc/pki/katello/private/ja [99%] Installing             Debug: /Stage[main]/Certs::Candlepin/Exec[create c [99%] Installing             Debug: /Stage[main]/Certs::Candlepin/Exec[import c [99%] Installing             Notice: /Stage[main]/Certs::Candlepin/Exec[import  [99%] Installing             Info: RESOURCE Cert[qeblade36.rhq.lab.eng.bos.redh [99%] Installing             Info: RESOURCE Package[pulp-rpm-plugins]           [99%] Installing             Info: RESOURCE Key_bundle[/etc/pki/pulp/nodes/node [99%] Installing             Info: RESOURCE File[/etc/default/pulp_workers]     [99%] Installing             Info: RESOURCE Service[httpd]                      [99%] Installing             Info: RESOURCE Qpid::Router::Ssl_profile[client]   [99%] Installing             Info: RESOURCE Service[qdrouterd]                  [99%] Installing             Debug: /Stage[main]/Katello::Qpid/Exec[create kate [99%] Installing             Debug: /Stage[main]/Katello::Qpid/Exec[bind katell [99%] Installing             Notice: /Stage[main]/Katello::Qpid/Exec[bind katel [99%] Installing             Info: RESOURCE Package[ruby193-rubygem-foreman-tas [99%] Installing             Info: RESOURCE Service[foreman]                    [99%] Installing             Debug: /Stage[main]/Certs::Foreman/Exec[foreman_ce [99%] Installing             Info: RESOURCE Foreman_smartproxy[qeblade36.rhq.la [99%] Installing             Notice: /Stage[main]/Foreman_proxy::Register/Forem [99%] Installing             Debug: Finishing transaction 39716760              [99%] Installing             Debug: Stored state in 0.11 seconds                [99%] Installing             Done                                               [100%]Installing             Done                                               [100%] []
  Success!
  * Katello is running at https://qeblade36.rhq.lab.eng.bos.redhat.com
      Initial credentials are admin / GvSWQ3evZjBcudW4
  * Capsule is running at https://qeblade36.rhq.lab.eng.bos.redhat.com:9090
  * To install additional capsule on separate machine continue by running:"

      capsule-certs-generate --capsule-fqdn "$CAPSULE" --certs-tar "~/$CAPSULE-certs.tar"

  The full log is at /var/log/katello-installer/katello-installer.log

Comment 5 Bryan Kearney 2015-08-11 13:31:42 UTC

This bug is slated to be released with Satellite 6.1.

Comment 6 Bryan Kearney 2015-08-12 13:57:32 UTC

This bug was fixed in version 6.1.1 of Satellite which was released on 12 August, 2015.

Note You need to log in before you can comment on or make changes to this bug.