Description of problem: After a pulp_worker dies, attempts are made to cancel the tasks assigned to that worker. However the cancel is called with the wrong number of arguments resulting in a traceback Version-Release number of selected component (if applicable): 6.1.3 How reproducible: Happens most times a pulp worker dies on my system Steps to Reproduce: 1. Have a pulp worker die when it has work assigned to it 2. 3. Actual results: Tasks not correctly cancelled. Expected results: Tasks correctly cancelled Additional info: Nov 06 21:11:42 stuarta-rhel7-sat6.usersys.redhat.com pulp[30873]: pulp.server.async.scheduler:ERROR: Workers 'resource_manager.redhat.com' has gone missing, removing from list of workers Nov 06 21:11:42 stuarta-rhel7-sat6.usersys.redhat.com pulp[30873]: pulp.server.async.tasks:ERROR: The worker named resource_manager.redhat.com is missing. Canceling the tasks in its queue. Nov 06 21:11:43 stuarta-rhel7-sat6.usersys.redhat.com pulp[30873]: pulp.server.async.scheduler:ERROR: Workers 'reserved_resource_worker-1.redhat.com' has gone missing, removing from list of workers Nov 06 21:11:43 stuarta-rhel7-sat6.usersys.redhat.com pulp[30873]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1.redhat.com is missing. Canceling the tasks in its queue. Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[30873]: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-3.redhat.com' discovered Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) Exception caught from plugin during publish for repo [org-stuarta-Red_Hat_OpenStack-Red_Hat_OpenStack_6_0_for_RHEL_7_RPMs_x86_64_7Server] Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) Traceback (most recent call last): Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/publish.py", line 127, in _do_publish Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) publish_report = publish_repo(transfer_repo, conduit, call_config) Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 495, in wrap_f Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) return f(*args, **kwargs) Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) File "/usr/lib/python2.7/site-packages/pulp_node/distributors/http/distributor.py", line 127, in publish_repo Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) publisher.publish(units) Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) File "/usr/lib/python2.7/site-packages/pulp_node/distributors/http/publisher.py", line 51, in publish Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) manifest_path = super(self.__class__, self).publish(units) Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) File "/usr/lib/python2.7/site-packages/pulp_node/distributors/publisher.py", line 125, in publish Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) for unit in units: Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) File "/usr/lib/python2.7/site-packages/pulp_node/conduit.py", line 111, in next Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) return self.unit_generator.next() Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) File "/usr/lib/python2.7/site-packages/pulp_node/conduit.py", line 99, in get_units Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) for metadata in cursor: Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 814, in next Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) if len(self.__data) or self._refresh(): Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 776, in _refresh Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) limit, self.__id)) Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 700, in __send_message Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) **kwargs) Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 991, in _send_message_with_response Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) return self.__send_and_receive(message, sock_info) Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 969, in __send_and_receive Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) return self.__receive_message_on_socket(1, request_id, sock_info) Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 954, in __receive_message_on_socket Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) header = self.__receive_data_on_socket(16, sock _info) Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 942, in __receive_data_on_socket Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) chunk = sock_info.sock.recv(length) Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 478, in sigterm_handler Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) handler() Nov 06 21:11:44 stuarta-rhel7-sat6.usersys.redhat.com pulp[31754]: pulp.server.managers.repo.publish:ERROR: (31754-40224) TypeError: cancel_publish_repo() takes exactly 3 arguments (1 given) Nov 06 21:11:45 stuarta-rhel7-sat6.usersys.redhat.com pulp[27151]: celery.worker.control:INFO: Terminating 85f0ab4b-ddf4-4562-b7c6-2f0329cec7e0 (15) Nov 06 21:11:45 stuarta-rhel7-sat6.usersys.redhat.com pulp[30873]: pulp.server.async.worker_watcher:INFO: New worker 'resource_manager.redhat.com' discovered Nov 06 21:11:45 stuarta-rhel7-sat6.usersys.redhat.com pulp[30873]: pulp.server.async.tasks:INFO: Task canceled: 85f0ab4b-ddf4-4562-b7c6-2f0329cec7e0. Nov 06 21:11:45 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[cfed26b1-d7c8-42c0-8a30-af7510a0687f] Nov 06 21:11:46 stuarta-rhel7-sat6.usersys.redhat.com pulp[27151]: celery.worker.control:INFO: Terminating c2732677-cb90-4791-bc78-c149254b4716 (15) Nov 06 21:11:46 stuarta-rhel7-sat6.usersys.redhat.com pulp[30873]: pulp.server.async.tasks:INFO: Task canceled: c2732677-cb90-4791-bc78-c149254b4716. Nov 06 21:11:46 stuarta-rhel7-sat6.usersys.redhat.com pulp[30873]: pulp.server.async.scheduler:ERROR: Workers 'reserved_resource_worker-0.redhat.com' has gone missing, removing from list of workers Nov 06 21:11:46 stuarta-rhel7-sat6.usersys.redhat.com pulp[30873]: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0.redhat.com is missing. Canceling the tasks in its queue. Nov 06 21:11:46 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[d4f34a09-a793-4d7b-b598-5d0b2d60fcd9] Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) Task pulp.server.managers.repo.publish.publish[b2a5e8b1-27ce-4580-9046-0511f384e23c] raised unexpected: TypeError('cancel_publish_repo() takes exactly 3 arguments (1 given)',) Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) Traceback (most recent call last): Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) R = retval = fun(*args, **kwargs) Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 328, in __call__ Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) return super(Task, self).__call__(*args, **kwargs) Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__ Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) return self.run(*args, **kwargs) Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/publish.py", line 99, in publish Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) transfer_repo, conduit, call_config) Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/publish.py", line 127, in _do_publish Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) publish_report = publish_repo(transfer_repo, conduit, call_config) Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 495, in wrap_f Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) return f(*args, **kwargs) Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) File "/usr/lib/python2.7/site-packages/pulp_node/distributors/http/distributor.py", line 127, in publish_repo Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) publisher.publish(units) Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) File "/usr/lib/python2.7/site-packages/pulp_node/distributors/http/publisher.py", line 51, in publish Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) manifest_path = super(self.__class__, self).publish(units) Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) File "/usr/lib/python2.7/site-packages/pulp_node/distributors/publisher.py", line 125, in publish Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) for unit in units: Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) File "/usr/lib/python2.7/site-packages/pulp_node/conduit.py", line 111, in next Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) return self.unit_generator.next() Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) File "/usr/lib/python2.7/site-packages/pulp_node/conduit.py", line 99, in get_units Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) for metadata in cursor: Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 814, in next Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) if len(self.__data) or self._refresh(): Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 776, in _refresh Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) limit, self.__id)) Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 700, in __send_message Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) **kwargs) Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 991, in _send_message_with_response Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) return self.__send_and_receive(message, sock_info) Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 969, in __send_and_receive Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) return self.__receive_message_on_socket(1, request_id, sock_info) Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 954, in __receive_message_on_socket Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) header = self.__receive_data_on_socket(16, sock_info) Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 942, in __receive_data_on_socket Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) chunk = sock_info.sock.recv(length) Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 478, in sigterm_handler Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) handler() Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:ERROR: (27155-40224) TypeError: cancel_publish_repo() takes exactly 3 arguments (1 given) Nov 06 21:11:47 stuarta-rhel7-sat6.usersys.redhat.com pulp[27155]: celery.worker.job:INFO: Discarding revoked task: pulp.server.managers.repo.publish.publish[e4170d4b-aa35-4c83-b510-8ce2b89455a7] ------- Corresponding dynflow error 17: Actions::Katello::Repository::Sync (error) [ 80.49s / 80.49s ] Skip Started at: 2015-11-06 21:25:49 UTC Ended at: 2015-11-06 21:27:09 UTC Real time: 80.49s Execution time (excluding suspended state): 80.49s Input: --- repository: id: 53 name: Red Hat OpenStack 6.0 for RHEL 7 RPMs x86_64 7Server label: Red_Hat_OpenStack_6_0_for_RHEL_7_RPMs_x86_64_7Server product: id: 66 name: Red Hat OpenStack label: Red_Hat_OpenStack cp_id: '191' provider: id: 4 name: Red Hat organization: id: 3 name: org-stuarta label: org-stuarta services_checked: - pulp - pulp_auth - elasticsearch id: 53 sync_result: pulp_tasks: - exception: task_type: pulp.server.managers.repo.sync.sync _href: /pulp/api/v2/tasks/cbaa0aaa-1a2e-4d18-9bcd-02e3b62d6bbf/ task_id: cbaa0aaa-1a2e-4d18-9bcd-02e3b62d6bbf tags: - pulp:repository:org-stuarta-Red_Hat_OpenStack-Red_Hat_OpenStack_6_0_for_RHEL_7_RPMs_x86_64_7Server - pulp:action:sync finish_time: '2015-11-06T20:45:47Z' _ns: task_status start_time: '2015-11-06T20:40:07Z' traceback: spawned_tasks: - _href: /pulp/api/v2/tasks/b2a5e8b1-27ce-4580-9046-0511f384e23c/ task_id: b2a5e8b1-27ce-4580-9046-0511f384e23c - _href: /pulp/api/v2/tasks/e4170d4b-aa35-4c83-b510-8ce2b89455a7/ task_id: e4170d4b-aa35-4c83-b510-8ce2b89455a7 progress_report: yum_importer: content: size_total: 0 items_left: 0 items_total: 0 state: FINISHED size_left: 0 details: rpm_total: 0 rpm_done: 0 drpm_total: 0 drpm_done: 0 error_details: [] comps: state: FINISHED distribution: items_total: 0 state: FINISHED error_details: [] items_left: 0 errata: state: FINISHED metadata: state: FINISHED queue: reserved_resource_worker-0.redhat.com.dq state: finished worker_name: reserved_resource_worker-0.redhat.com result: importer_type_id: yum_importer importer_id: yum_importer exception: repo_id: org-stuarta-Red_Hat_OpenStack-Red_Hat_OpenStack_6_0_for_RHEL_7_RPMs_x86_64_7Server removed_count: 0 started: '2015-11-06T20:40:09Z' _ns: repo_sync_results completed: '2015-11-06T20:45:44Z' traceback: summary: content: state: FINISHED comps: state: FINISHED distribution: state: FINISHED errata: state: FINISHED metadata: state: FINISHED added_count: 0 error_message: updated_count: 77 details: content: size_total: 0 items_left: 0 items_total: 0 state: FINISHED size_left: 0 details: rpm_total: 0 rpm_done: 0 drpm_total: 0 drpm_done: 0 error_details: [] comps: state: FINISHED distribution: items_total: 0 state: FINISHED error_details: [] items_left: 0 errata: state: FINISHED metadata: state: FINISHED id: 563d11789b3eb57b94f3d88c result: success error: _id: $oid: 563d0738697e0dbab8d2a894 id: 563d10279b3eb5587b70f8e9 - exception: task_type: pulp.server.managers.repo.publish.publish _href: /pulp/api/v2/tasks/b2a5e8b1-27ce-4580-9046-0511f384e23c/ task_id: b2a5e8b1-27ce-4580-9046-0511f384e23c tags: - pulp:repository:org-stuarta-Red_Hat_OpenStack-Red_Hat_OpenStack_6_0_for_RHEL_7_RPMs_x86_64_7Server - pulp:action:publish finish_time: '2015-11-06T21:11:45Z' _ns: task_status start_time: '2015-11-06T21:09:58Z' traceback: ! "Traceback (most recent call last):\n File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 240, in trace_task\n R = retval = fun(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 328, in __call__\n return super(Task, self).__call__(*args, **kwargs)\n \ File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 437, in __protected_call__\n return self.run(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/pulp/server/managers/repo/publish.py\", line 99, in publish\n transfer_repo, conduit, call_config)\n File \"/usr/lib/python2.7/site-packages/pulp/server/managers/repo/publish.py\", line 127, in _do_publish\n publish_report = publish_repo(transfer_repo, conduit, call_config)\n File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 495, in wrap_f\n return f(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/pulp_node/distributors/http/distributor.py\", line 127, in publish_repo\n publisher.publish(units)\n File \"/usr/lib/python2.7/site-packages/pulp_node/distributors/http/publisher.py\", line 51, in publish\n manifest_path = super(self.__class__, self).publish(units)\n \ File \"/usr/lib/python2.7/site-packages/pulp_node/distributors/publisher.py\", line 125, in publish\n for unit in units:\n File \"/usr/lib/python2.7/site-packages/pulp_node/conduit.py\", line 111, in next\n return self.unit_generator.next()\n File \"/usr/lib/python2.7/site-packages/pulp_node/conduit.py\", line 99, in get_units\n for metadata in cursor:\n File \"/usr/lib64/python2.7/site-packages/pymongo/cursor.py\", line 814, in next\n if len(self.__data) or self._refresh():\n File \"/usr/lib64/python2.7/site-packages/pymongo/cursor.py\", line 776, in _refresh\n limit, self.__id))\n File \"/usr/lib64/python2.7/site-packages/pymongo/cursor.py\", line 700, in __send_message\n **kwargs)\n File \"/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py\", line 991, in _send_message_with_response\n return self.__send_and_receive(message, sock_info)\n File \"/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py\", line 969, in __send_and_receive\n return self.__receive_message_on_socket(1, request_id, sock_info)\n File \"/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py\", line 954, in __receive_message_on_socket\n header = self.__receive_data_on_socket(16, sock_info)\n File \"/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py\", line 942, in __receive_data_on_socket\n chunk = sock_info.sock.recv(length)\n \ File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 478, in sigterm_handler\n handler()\nTypeError: cancel_publish_repo() takes exactly 3 arguments (1 given)\n" spawned_tasks: [] progress_report: {} queue: reserved_resource_worker-3.redhat.com.dq state: error worker_name: reserved_resource_worker-3.redhat.com result: error: code: PLP0000 data: {} description: cancel_publish_repo() takes exactly 3 arguments (1 given) sub_errors: [] _id: $oid: 563d117a697e0dbab8d2a8b0 id: 563d171c9b3eb5587b717c0d - exception: task_type: pulp.server.managers.repo.publish.publish _href: /pulp/api/v2/tasks/e4170d4b-aa35-4c83-b510-8ce2b89455a7/ task_id: e4170d4b-aa35-4c83-b510-8ce2b89455a7 tags: - pulp:repository:org-stuarta-Red_Hat_OpenStack-Red_Hat_OpenStack_6_0_for_RHEL_7_RPMs_x86_64_7Server - pulp:action:publish finish_time: start_time: traceback: spawned_tasks: [] progress_report: {} queue: reserved_resource_worker-3.redhat.com.dq state: canceled worker_name: reserved_resource_worker-3.redhat.com result: error: _id: $oid: 563d117a697e0dbab8d2a8b1 id: 563d17219b3eb5587b717c11 poll_attempts: total: 75 failed: 2 user_id: 1 current_user_id: 1 locale: en Output: --- sync_result: pulp_tasks: - exception: task_type: pulp.server.managers.repo.sync.sync _href: /pulp/api/v2/tasks/cbaa0aaa-1a2e-4d18-9bcd-02e3b62d6bbf/ task_id: cbaa0aaa-1a2e-4d18-9bcd-02e3b62d6bbf tags: - pulp:repository:org-stuarta-Red_Hat_OpenStack-Red_Hat_OpenStack_6_0_for_RHEL_7_RPMs_x86_64_7Server - pulp:action:sync finish_time: '2015-11-06T20:45:47Z' _ns: task_status start_time: '2015-11-06T20:40:07Z' traceback: spawned_tasks: - _href: /pulp/api/v2/tasks/b2a5e8b1-27ce-4580-9046-0511f384e23c/ task_id: b2a5e8b1-27ce-4580-9046-0511f384e23c - _href: /pulp/api/v2/tasks/e4170d4b-aa35-4c83-b510-8ce2b89455a7/ task_id: e4170d4b-aa35-4c83-b510-8ce2b89455a7 progress_report: yum_importer: content: size_total: 0 items_left: 0 items_total: 0 state: FINISHED size_left: 0 details: rpm_total: 0 rpm_done: 0 drpm_total: 0 drpm_done: 0 error_details: [] comps: state: FINISHED distribution: items_total: 0 state: FINISHED error_details: [] items_left: 0 errata: state: FINISHED metadata: state: FINISHED queue: reserved_resource_worker-0.redhat.com.dq state: finished worker_name: reserved_resource_worker-0.redhat.com result: importer_type_id: yum_importer importer_id: yum_importer exception: repo_id: org-stuarta-Red_Hat_OpenStack-Red_Hat_OpenStack_6_0_for_RHEL_7_RPMs_x86_64_7Server removed_count: 0 started: '2015-11-06T20:40:09Z' _ns: repo_sync_results completed: '2015-11-06T20:45:44Z' traceback: summary: content: state: FINISHED comps: state: FINISHED distribution: state: FINISHED errata: state: FINISHED metadata: state: FINISHED added_count: 0 error_message: updated_count: 77 details: content: size_total: 0 items_left: 0 items_total: 0 state: FINISHED size_left: 0 details: rpm_total: 0 rpm_done: 0 drpm_total: 0 drpm_done: 0 error_details: [] comps: state: FINISHED distribution: items_total: 0 state: FINISHED error_details: [] items_left: 0 errata: state: FINISHED metadata: state: FINISHED id: 563d11789b3eb57b94f3d88c result: success error: _id: $oid: 563d0738697e0dbab8d2a894 id: 563d10279b3eb5587b70f8e9 - exception: task_type: pulp.server.managers.repo.publish.publish _href: /pulp/api/v2/tasks/b2a5e8b1-27ce-4580-9046-0511f384e23c/ task_id: b2a5e8b1-27ce-4580-9046-0511f384e23c tags: - pulp:repository:org-stuarta-Red_Hat_OpenStack-Red_Hat_OpenStack_6_0_for_RHEL_7_RPMs_x86_64_7Server - pulp:action:publish finish_time: '2015-11-06T21:11:45Z' _ns: task_status start_time: '2015-11-06T21:09:58Z' traceback: ! "Traceback (most recent call last):\n File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 240, in trace_task\n R = retval = fun(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 328, in __call__\n return super(Task, self).__call__(*args, **kwargs)\n \ File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 437, in __protected_call__\n return self.run(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/pulp/server/managers/repo/publish.py\", line 99, in publish\n transfer_repo, conduit, call_config)\n File \"/usr/lib/python2.7/site-packages/pulp/server/managers/repo/publish.py\", line 127, in _do_publish\n publish_report = publish_repo(transfer_repo, conduit, call_config)\n File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 495, in wrap_f\n return f(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/pulp_node/distributors/http/distributor.py\", line 127, in publish_repo\n publisher.publish(units)\n File \"/usr/lib/python2.7/site-packages/pulp_node/distributors/http/publisher.py\", line 51, in publish\n manifest_path = super(self.__class__, self).publish(units)\n \ File \"/usr/lib/python2.7/site-packages/pulp_node/distributors/publisher.py\", line 125, in publish\n for unit in units:\n File \"/usr/lib/python2.7/site-packages/pulp_node/conduit.py\", line 111, in next\n return self.unit_generator.next()\n File \"/usr/lib/python2.7/site-packages/pulp_node/conduit.py\", line 99, in get_units\n for metadata in cursor:\n File \"/usr/lib64/python2.7/site-packages/pymongo/cursor.py\", line 814, in next\n if len(self.__data) or self._refresh():\n File \"/usr/lib64/python2.7/site-packages/pymongo/cursor.py\", line 776, in _refresh\n limit, self.__id))\n File \"/usr/lib64/python2.7/site-packages/pymongo/cursor.py\", line 700, in __send_message\n **kwargs)\n File \"/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py\", line 991, in _send_message_with_response\n return self.__send_and_receive(message, sock_info)\n File \"/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py\", line 969, in __send_and_receive\n return self.__receive_message_on_socket(1, request_id, sock_info)\n File \"/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py\", line 954, in __receive_message_on_socket\n header = self.__receive_data_on_socket(16, sock_info)\n File \"/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py\", line 942, in __receive_data_on_socket\n chunk = sock_info.sock.recv(length)\n \ File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 478, in sigterm_handler\n handler()\nTypeError: cancel_publish_repo() takes exactly 3 arguments (1 given)\n" spawned_tasks: [] progress_report: {} queue: reserved_resource_worker-3.redhat.com.dq state: error worker_name: reserved_resource_worker-3.redhat.com result: error: code: PLP0000 data: {} description: cancel_publish_repo() takes exactly 3 arguments (1 given) sub_errors: [] _id: $oid: 563d117a697e0dbab8d2a8b0 id: 563d171c9b3eb5587b717c0d - exception: task_type: pulp.server.managers.repo.publish.publish _href: /pulp/api/v2/tasks/e4170d4b-aa35-4c83-b510-8ce2b89455a7/ task_id: e4170d4b-aa35-4c83-b510-8ce2b89455a7 tags: - pulp:repository:org-stuarta-Red_Hat_OpenStack-Red_Hat_OpenStack_6_0_for_RHEL_7_RPMs_x86_64_7Server - pulp:action:publish finish_time: start_time: traceback: spawned_tasks: [] progress_report: {} queue: reserved_resource_worker-3.redhat.com.dq state: canceled worker_name: reserved_resource_worker-3.redhat.com result: error: _id: $oid: 563d117a697e0dbab8d2a8b1 id: 563d17219b3eb5587b717c11 poll_attempts: total: 75 failed: 2 Error: RuntimeError There was an issue with the backend service pulp: Request Timeout --- - /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.73/app/lib/actions/middleware/backend_services_check.rb:16:in `block in plan' - /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.73/app/lib/actions/middleware/backend_services_check.rb:14:in `each' - /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.73/app/lib/actions/middleware/backend_services_check.rb:14:in `plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.73/app/lib/actions/middleware/remote_action.rb:20:in `plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:29:in `plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.73/app/lib/actions/middleware/keep_locale.rb:19:in `plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/world.rb:30:in `execute' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:441:in `block (2 levels) in execute_plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan.rb:234:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan.rb:234:in `switch_flow' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:347:in `concurrence' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:440:in `block in execute_plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `block in with_error_handling' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `catch' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `with_error_handling' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:439:in `execute_plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:262:in `execute' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/plan_step.rb:46:in `block in execute' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/plan_step.rb:45:in `execute' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:375:in `plan_action' - /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.73/app/lib/actions/katello/repository/node_metadata_generate.rb:19:in `plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:442:in `block (3 levels) in execute_plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:29:in `plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.73/app/lib/actions/middleware/keep_locale.rb:19:in `plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/world.rb:30:in `execute' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:441:in `block (2 levels) in execute_plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan.rb:234:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan.rb:234:in `switch_flow' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:347:in `concurrence' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:440:in `block in execute_plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `block in with_error_handling' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `catch' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `with_error_handling' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:439:in `execute_plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:262:in `execute' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/plan_step.rb:46:in `block in execute' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/plan_step.rb:45:in `execute' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:375:in `plan_action' - /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.73/app/lib/actions/katello/repository/capsule_generate_and_sync.rb:23:in `plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:442:in `block (3 levels) in execute_plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:29:in `plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.73/app/lib/actions/middleware/keep_locale.rb:19:in `plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/world.rb:30:in `execute' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:441:in `block (2 levels) in execute_plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan.rb:234:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan.rb:234:in `switch_flow' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:347:in `concurrence' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:440:in `block in execute_plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `block in with_error_handling' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `catch' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `with_error_handling' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:439:in `execute_plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:262:in `execute' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/plan_step.rb:46:in `block in execute' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/plan_step.rb:45:in `execute' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan.rb:166:in `block (3 levels) in plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan.rb:234:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan.rb:234:in `switch_flow' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan.rb:224:in `with_planning_scope' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan.rb:165:in `block (2 levels) in plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:37:in `plan_phase' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:37:in `plan_phase' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/world.rb:30:in `execute' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan.rb:164:in `block in plan' - /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction' - /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:208:in `transaction' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/transaction_adapters/active_record.rb:5:in `transaction' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan.rb:163:in `plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/world.rb:129:in `block in plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/world.rb:127:in `tap' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/world.rb:127:in `plan' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/world.rb:100:in `trigger' - /opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.15.7/lib/foreman_tasks.rb:17:in `trigger' - /opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.15.7/lib/foreman_tasks.rb:23:in `trigger_task' - /opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.15.7/lib/foreman_tasks.rb:35:in `async_task' - /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.73/app/lib/actions/katello/repository/sync.rb:56:in `run' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:487:in `block (3 levels) in execute_run' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.15.7/app/lib/actions/middleware/keep_current_user.rb:23:in `block in run' - /opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.15.7/app/lib/actions/middleware/keep_current_user.rb:34:in `restore_curent_user' - /opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.15.7/app/lib/actions/middleware/keep_current_user.rb:23:in `run' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/progress.rb:30:in `with_progress_calculation' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/progress.rb:16:in `run' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass' - /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.73/app/lib/actions/middleware/keep_locale.rb:23:in `block in run' - /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.73/app/lib/actions/middleware/keep_locale.rb:34:in `with_locale' - /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.73/app/lib/actions/middleware/keep_locale.rb:23:in `run' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/world.rb:30:in `execute' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:486:in `block (2 levels) in execute_run' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:485:in `catch' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:485:in `block in execute_run' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `block in with_error_handling' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `catch' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `with_error_handling' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:480:in `execute_run' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:262:in `execute' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/executors/parallel/worker.rb:20:in `block in on_message' - /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:859:in `block in assigns' - /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `tap' - /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `assigns' - /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:138:in `match_value' - /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:116:in `block in match' - /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `each' - /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `match' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/executors/parallel/worker.rb:17:in `on_message' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:82:in `on_envelope' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:72:in `receive' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `block (2 levels) in run' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `loop' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `block in run' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `catch' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `run' - /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:13:in `block in initialize' - /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `call' - /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
This appears to be fixed in pulp 2.8, and probably earlier than that. I can chase down the exact fix if desired, or we can just call it fixed in 2.8.
Hi, Please, guide me here for the steps: 1.)Create a product 2.) Add a Repository Is it specific to (Red Hat OpenStack 6.0 for RHEL 7 RPMs x86_64 7Server) ? 3.) Sync it 4.) Stop the pulp_celerybeat.service 5.) Check /var/log/messages Thanks and Regards, Tazim
Verified ! @ Sat 6.2 Snap 9.2 After pulp worker dies the task has been canceled without throwing any errors as mentioned in Description. The proper info message of pulp workers shutdown is displayed in /var/log/messages as below: Apr 27 02:53:05 qeblade36 systemd: Stopping Pulp Celery Workers... Apr 27 02:53:05 qeblade36 systemd: Stopping Pulp Worker #4... Apr 27 02:53:05 qeblade36 systemd: Stopping Pulp Worker #2... Apr 27 02:53:05 qeblade36 systemd: Stopping Pulp Worker #6... Apr 27 02:53:05 qeblade36 systemd: Stopping Pulp Worker #5... Apr 27 02:53:05 qeblade36 systemd: Stopping Pulp Worker #3... Apr 27 02:53:05 qeblade36 systemd: Stopping Pulp Worker #1... Apr 27 02:53:05 qeblade36 systemd: Stopping Pulp Worker #7... Apr 27 02:53:05 qeblade36 systemd: Stopping Pulp Worker #0... Apr 27 02:53:05 qeblade36 celery: worker: Cold shutdown (MainProcess) Apr 27 02:53:05 qeblade36 celery: worker: Cold shutdown (MainProcess) Apr 27 02:53:05 qeblade36 celery: worker: Cold shutdown (MainProcess) Apr 27 02:53:05 qeblade36 celery: worker: Cold shutdown (MainProcess) Apr 27 02:53:05 qeblade36 celery: worker: Cold shutdown (MainProcess) Apr 27 02:53:05 qeblade36 celery: worker: Cold shutdown (MainProcess) Apr 27 02:53:05 qeblade36 celery: worker: Cold shutdown (MainProcess) Apr 27 02:53:05 qeblade36 celery: worker: Cold shutdown (MainProcess) Apr 27 02:53:05 qeblade36 pulp: pulp.server.async.worker_watcher:INFO: Worker 'reserved_resource_worker-1.lab.eng.bos.redhat.com' shutdown Apr 27 02:53:05 qeblade36 pulp: pulp.server.async.worker_watcher:INFO: Worker 'reserved_resource_worker-0.lab.eng.bos.redhat.com' shutdown Apr 27 02:53:05 qeblade36 pulp: pulp.server.async.worker_watcher:INFO: Worker 'reserved_resource_worker-7.lab.eng.bos.redhat.com' shutdown Apr 27 02:53:05 qeblade36 pulp: pulp.server.async.worker_watcher:INFO: Worker 'reserved_resource_worker-5.lab.eng.bos.redhat.com' shutdown Apr 27 02:53:05 qeblade36 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS Apr 27 02:53:05 qeblade36 pulp: pulp.server.async.tasks:INFO: Task canceled: 0fba90b2-eabb-446e-ade7-30da9f433dce. Apr 27 02:53:05 qeblade36 pulp: pulp.server.async.worker_watcher:INFO: Worker 'reserved_resource_worker-2.lab.eng.bos.redhat.com' shutdown Apr 27 02:53:05 qeblade36 pulp: pulp.server.async.worker_watcher:INFO: Worker 'reserved_resource_worker-4.lab.eng.bos.redhat.com' shutdown Apr 27 02:53:05 qeblade36 pulp: pulp.server.async.worker_watcher:INFO: Worker 'reserved_resource_worker-6.lab.eng.bos.redhat.com' shutdown Apr 27 02:53:05 qeblade36 pulp: pulp.server.async.worker_watcher:INFO: Worker 'reserved_resource_worker-3.lab.eng.bos.redhat.com' shutdown Apr 27 02:53:05 qeblade36 celery: reserved_resource_worker-4.lab.eng.bos.redhat.com ready. Apr 27 02:53:05 qeblade36 celery: reserved_resource_worker-6.lab.eng.bos.redhat.com ready. Apr 27 02:53:05 qeblade36 celery: reserved_resource_worker-3.lab.eng.bos.redhat.com ready. Apr 27 02:53:05 qeblade36 celery: reserved_resource_worker-1.lab.eng.bos.redhat.com ready. Apr 27 02:53:05 qeblade36 celery: reserved_resource_worker-7.lab.eng.bos.redhat.com ready. Apr 27 02:53:05 qeblade36 celery: reserved_resource_worker-0.lab.eng.bos.redhat.com ready. Apr 27 02:53:05 qeblade36 celery: reserved_resource_worker-2.lab.eng.bos.redhat.com ready. Apr 27 02:53:06 qeblade36 systemd: Stopped Pulp Worker #7. Apr 27 02:53:06 qeblade36 systemd: Stopped Pulp Worker #3. Apr 27 02:53:06 qeblade36 systemd: Stopped Pulp Worker #6. Apr 27 02:53:06 qeblade36 systemd: Stopped Pulp Worker #0. Apr 27 02:53:06 qeblade36 systemd: Stopped Pulp Worker #4. Apr 27 02:53:06 qeblade36 systemd: Stopped Pulp Worker #2. Apr 27 02:53:06 qeblade36 systemd: Stopped Pulp Worker #1. Apr 27 02:53:06 qeblade36 celery: reserved_resource_worker-5.lab.eng.bos.redhat.com ready. Apr 27 02:53:07 qeblade36 systemd: Stopped Pulp Worker #5. Apr 27 02:53:07 qeblade36 systemd: Stopped Pulp Celery Workers. So moving this Bz to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1501