Bug 878620
| Summary: | CLI is stuck on repo sync, yet task says it is Successful | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Retired] Pulp | Reporter: | John Matthews <jmatthew> | ||||
| Component: | async/tasks | Assignee: | Jason Connor <jconnor> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Preethi Thomas <pthomas> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | urgent | ||||||
| Version: | 2.0.6 | CC: | jason.dobies, mmccune, pthomas, skarmark | ||||
| Target Milestone: | --- | Keywords: | Triaged | ||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2013-01-09 17:04:14 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
*** Bug 876345 has been marked as a duplicate of this bug. *** Fixed duplicates and extraneous serialized call reports in the task groups REST API resource. This has only been merged into pulp-2.0 and will still need to be merged into master someday. build: 2.0.6-0.10.beta Failing this before Preethi has a chance to.
Using the .11 beta, I have a failed sync because NFS is improperly set up. The CLI never ends:
[jdob@prosperity pulp]$ pulp-admin rpm repo sync run --repo-id pulp-2-f17-64
+----------------------------------------------------------------------+
Synchronizing Repository [pulp-2-f17-64]
+----------------------------------------------------------------------+
This command may be exited by pressing ctrl+c without affecting the actual
operation on the server.
Downloading metadata...
[\]
... failed
^C
The reports coming back during the polling look like this:
2012-11-30 14:44:19,137 - INFO - GET request to /pulp/api/v2/tasks/85ee78b1-7e71-433b-9311-7f639de32cb0/ with parameters None
2012-11-30 14:44:19,137 - INFO - Response status : 200
2012-11-30 14:44:19,137 - INFO - Response body :
{
"task_group_id": "b01ef25a-9676-4227-8470-09651e853cc1",
"call_request_id": "85ee78b1-7e71-433b-9311-7f639de32cb0",
"exception": null,
"_href": "/pulp/api/v2/tasks/85ee78b1-7e71-433b-9311-7f639de32cb0/",
"task_id": "85ee78b1-7e71-433b-9311-7f639de32cb0",
"call_request_tags": [
"pulp:repository:pulp-2-f17-64",
"pulp:action:auto_publish",
"pulp:action:publish"
],
"reasons": [
{
"operation": "update",
"resource_type": "repository",
"resource_id": "pulp-2-f17-64"
}
],
"start_time": null,
"traceback": null,
"schedule_id": null,
"finish_time": "2012-11-30T19:18:11Z",
"state": "skipped",
"result": null,
"dependency_failures": {
"5f4f2eaf-e129-4446-8f5a-991d6619c790": {
"expected": [
"finished"
],
"actual": "error"
}
},
"call_request_group_id": "b01ef25a-9676-4227-8470-09651e853cc1",
"progress": {},
"principal_login": "admin",
"response": "postponed",
"tags": [
"pulp:repository:pulp-2-f17-64",
"pulp:action:auto_publish",
"pulp:action:publish"
]
}
45 minutes later, the tasks are still in the queue:
[jdob@prosperity .pulp]$ pulp-admin tasks list
+----------------------------------------------------------------------+
Tasks
+----------------------------------------------------------------------+
Operations: auto_publish, publish
Resources: pulp-2-f17-64 (repository)
State: Unknown
Start Time: Unstarted
Finish Time: Incomplete
Result: Incomplete
Task Id: 85ee78b1-7e71-433b-9311-7f639de32cb0
Operations: sync
Resources: pulp-2-f17-64 (repository)
State: Failed
Start Time: Unstarted
Finish Time: 2012-11-30T19:18:11Z
Result: Incomplete
Task Id: 5f4f2eaf-e129-4446-8f5a-991d6619c790
[jdob@prosperity .pulp]$ date
Fri Nov 30 15:02:26 EST 2012
Ok, so this is a separate issue where the cli hangs when the sync is in failure. The issue is that the cli parsing doesn't recognized 'skipped' as a finish state. Opening a new bug. Ok, so this is a separate issue where the cli hangs when the sync is in failure. The issue is that the cli parsing doesn't recognized 'skipped' as a finish state. Opening a new bug and moving this one back to modified. Fixed in the 0.12 beta. verified [root@preethi-el6-pulp ~]# rpm -qa|grep pulp python-pulp-rpm-common-2.0.6-0.14.beta.noarch pulp-puppet-plugins-2.0.6-0.14.beta.noarch pulp-selinux-2.0.6-0.14.beta.noarch pulp-rpm-admin-extensions-2.0.6-0.14.beta.noarch m2crypto-0.21.1.pulp-8.el6.x86_64 python-oauth2-1.5.170-3.pulp.el6.noarch mod_wsgi-3.3-4.pulp.el6.x86_64 pulp-server-2.0.6-0.14.beta.noarch python-pulp-puppet-common-2.0.6-0.14.beta.noarch pulp-rpm-plugins-2.0.6-0.14.beta.noarch python-pulp-rpm-extension-2.0.6-0.14.beta.noarch python-pulp-bindings-2.0.6-0.14.beta.noarch pulp-builtins-admin-extensions-2.0.6-0.14.beta.noarch pulp-puppet-admin-extensions-2.0.6-0.14.beta.noarch python-rhsm-1.8.0-1.pulp.el6.x86_64 python-pulp-common-2.0.6-0.14.beta.noarch python-pulp-client-lib-2.0.6-0.14.beta.noarch pulp-admin-client-2.0.6-0.14.beta.noarch python-isodate-0.5.0-1.pulp.el6.noarch [root@preethi-el6-pulp ~]# Pulp v2.0 released |
Created attachment 648749 [details] Logs from a stuck CLI Description of problem: Pulp CLI sometimes does not finish a repo sync. Server logs say the sync task has finished, yet the CLI remains in a loop waiting. Version-Release number of selected component (if applicable): # rpm -qa | grep pulp python-pulp-common-0.0.339-1.noarch python-pulp-bindings-0.0.339-1.noarch python-rhsm-1.1.4-0.pulp.el6.x86_64 pulp-rpm-handlers-0.0.339-1.noarch pulp-agent-0.0.339-1.noarch pulp-admin-client-0.0.339-1.noarch pulp-rpm-admin-extensions-0.0.339-1.noarch pulp-rpm-plugins-0.0.339-1.noarch pulp-rpm-agent-0.0.339-1.noarch python-pulp-rpm-common-0.0.339-1.noarch python-pulp-agent-lib-0.0.339-1.noarch m2crypto-0.21.1.pulp-8.el6.x86_64 python-oauth2-1.5.170-3.pulp.el6.noarch pulp-builtins-admin-extensions-0.0.339-1.noarch pulp-builtins-consumer-extensions-0.0.339-1.noarch pulp-rpm-consumer-extensions-0.0.339-1.noarch pulp-server-0.0.339-1.noarch pulp-rpm-server-0.0.339-1.noarch pulp-rpm-consumer-client-0.0.339-1.noarch python-pulp-rpm-extension-0.0.339-1.noarch python-pulp-client-lib-0.0.339-1.noarch pulp-consumer-client-0.0.339-1.noarch mod_wsgi-3.3-4.pulp.el6.x86_64 pulp-rpm-admin-client-0.0.339-1.noarch How reproducible: It seems to take an act of putting Pulp into a bad state. I've done this by syncing a large repo then canceling the task midway or restarting httpd. I think the key is to do something to cause a problem with a repo sync. # time pulp-admin rpm repo sync run --repo-id f17_x86_64 +----------------------------------------------------------------------+ Synchronizing Repository [f17_x86_64] +----------------------------------------------------------------------+ This command may be exited by pressing ctrl+c without affecting the actual operation on the server. Importing errata... [-] ... completed Importing package groups/categories... [|] ... completed Publishing packages... [==================================================] 100% Packages: 0/0 items ... completed Publishing distributions... ... failed Generating metadata [/] ... completed Publishing repository over HTTP [-] ... completed Publishing repository over HTTPS [-] ... skipped This is where it's hung. When I look at the pulp logs I see: 2012-11-19 09:07:27,303 18412:140092620347136: pulp.server.dispatch.task:INFO: task:148 SUCCESS: Task e1bf15c7-22fd-454a-9733-78748f8f8ba6: CallRequest: RepoPublishManager.publish(u'f17_x86_64', u'yum_distributor', distributor_instance=<yum_distributor.distributor.YumDistributor object at 0x7f69c3c2cb90>, distributor_config={}) I typically see a task remaining like the below # pulp-admin tasks list +----------------------------------------------------------------------+ Tasks +----------------------------------------------------------------------+ Operations: auto_publish, publish Resources: f17_x86_64 (repository) State: Successful Start Time: Unstarted Finish Time: 2012-11-19T14:07:27Z Result: Incomplete Task Id: e1bf15c7-22fd-454a-9733-78748f8f8ba6 # pulp-admin tasks cancel --task-id e1bf15c7-22fd-454a-9733-78748f8f8ba6 The following resource(s) could not be found: e1bf15c7-22fd-454a-9733-78748f8f8ba6 (resource_id) I have seen the CLI loop through syncs without incident, worked for 24+ hours of continuous syncs.