Similar to a recent problem discovered with rpm repo syncs, I noticed today that Puppet modules continue to be imported long after the task cancellation is issued. I created a mirror of Puppet forge and started the sync. I cancelled it after it had retrieved 269 modules. As of 5 or so minutes later, it has downloaded 502 modules and is still downloading, as can be seen by $ pulp-admin puppet repo list --details. The Celery worker still seems to be processing the task.
Of course, immediately after filing this bug the sync does appear to have stopped. It got to 515 modules. I do think we have an issue. It appears that cancelation takes a very long time (several minutes) to be effective.
https://github.com/pulp/pulp_puppet/pull/118
The fix for this issue is included in pulp-2.4.0-0.12.beta.
Task before cancellation: $ curl -H "Accept: application/json" -X GET -k -u admin:admin 'https://example.com/pulp/api/v2/tasks/1594a959-1c4e-4e12-a11b-379270e5c676/'|python -m json.tool % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1069 100 1069 0 0 1307 0 --:--:-- --:--:-- --:--:-- 1306 { "_href": "/pulp/api/v2/tasks/1594a959-1c4e-4e12-a11b-379270e5c676/", "_id": { "$oid": "5370c48b930c1f4a8635b4d9" }, "_ns": "task_status", "error": null, "exception": null, "finish_time": null, "id": "5370c48bd42884638748853e", "progress_report": { "puppet_importer": { "metadata": { "current_query": "http://forge.puppetlabs.com/modules.json", "error": "None", "error_message": null, "execution_time": 23, "query_finished_count": 1, "query_total_count": 1, "state": "success", "traceback": null }, "modules": { "error": "None", "error_count": 0, "error_message": null, "execution_time": null, "finished_count": 26, "individual_errors": [], "state": "running", "total_count": 2348, "traceback": null } } }, "queue": "reserved_resource_worker-0.compute.amazonaws.com", "result": null, "spawned_tasks": [], "start_time": "2014-05-12T12:54:35Z", "state": "running", "tags": [ "pulp:repository:m1", "pulp:action:sync" ], "task_id": "1594a959-1c4e-4e12-a11b-379270e5c676", "task_type": "pulp.server.tasks.repository.sync_with_auto_publish", "traceback": null } Task immediately after cancellation: $ curl -H "Accept: application/json" -X GET -k -u admin:admin 'https://example.com/pulp/api/v2/tasks/1594a959-1c4e-4e12-a11b-379270e5c676/'|python -m json.tool % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1562 100 1562 0 0 1963 0 --:--:-- --:--:-- --:--:-- 1962 { "_href": "/pulp/api/v2/tasks/1594a959-1c4e-4e12-a11b-379270e5c676/", "_id": { "$oid": "5370c48b930c1f4a8635b4d9" }, "_ns": "task_status", "error": null, "exception": null, "finish_time": "2014-05-12T12:55:49Z", "id": "5370c48bd42884638748853e", "progress_report": { "puppet_importer": { "metadata": { "current_query": "http://forge.puppetlabs.com/modules.json", "error": "None", "error_message": null, "execution_time": 23, "query_finished_count": 1, "query_total_count": 1, "state": "success", "traceback": null }, "modules": { "error": "None", "error_count": 0, "error_message": null, "execution_time": 50, "finished_count": 43, "individual_errors": [], "state": "success", "total_count": 2348, "traceback": null } } }, "queue": "reserved_resource_worker-0.compute.amazonaws.com", "result": { "_ns": "repo_sync_results", "added_count": 43, "completed": "2014-05-12T12:55:49Z", "details": { "error_count": 0, "finished_count": 43, "total_count": 2348 }, "error_message": null, "exception": null, "id": "5370c4d5d4288463e0ddd678", "importer_id": "puppet_importer", "importer_type_id": "puppet_importer", "removed_count": 0, "repo_id": "m1", "result": "success", "started": "2014-05-12T12:54:35Z", "summary": { "total_execution_time": 73 }, "traceback": null, "updated_count": 0 }, "spawned_tasks": [], "start_time": "2014-05-12T12:54:35Z", "state": "finished", "tags": [ "pulp:repository:m1", "pulp:action:sync" ], "task_id": "1594a959-1c4e-4e12-a11b-379270e5c676", "task_type": "pulp.server.tasks.repository.sync_with_auto_publish", "traceback": null } # pulp-admin -u admin -p admin puppet repo list +----------------------------------------------------------------------+ Puppet Repositories +----------------------------------------------------------------------+ Id: m1 Display Name: m1 Description: None Content Unit Counts: Puppet Module: 43 After task cancellation number of downloaded puppet modules did not increase. Tested in pulp-server-2.4.0-0.13.beta.fc20.noarch Moving to Verified.
Just one thing that I noticed - after task cancellation the state of task is 'finished', would not it be more correct 'cancelled'?
Yes, I believe that is also incorrect behavior. The importer must not be checking the task state before marking itself as finished. I believe this is a separate issue, so I've filed that here: https://bugzilla.redhat.com/show_bug.cgi?id=1096822
This has been fixed in Pulp 2.4.0-1.