Description of problem: Currently when syncing puppet forge-style repos, the progress report looks something like: On first sync: modules: traceback: execution_time: 1 total_count: 2136 error_message: finished_count: 2000 state: success error_count: 136 error: None On next sync: modules: traceback: execution_time: 1 total_count: 136 error_message: finished_count: 0 state: success error_count: 136 error: None This looks correct. But puppet repos that use a pulp manifest file are incorrect: On first sync: modules: traceback: execution_time: total_count: 4 error_message: finished_count: 0 state: running error_count: 0 error: None individual_errors: [] On next sync: modules: traceback: execution_time: total_count: 4 error_message: finished_count: 0 state: running error_count: 0 error: None individual_errors: [] There are 3 things wrong here: * State stays 'running'. Even though the task is finished * "finished_count" is 0 even though 4 modules were downloaded * The 2nd sync still shows a total of 4, even though all 4 modules were already synced the first time. Puppet forge syncing doesn't appear to show any of these issues Version-Release number of selected component (if applicable): pulp-server-2.4.0-0.29.beta.el6.noarch How reproducible: Always Steps to Reproduce: 1. Sync a pulp_manifest style puppet repo 2. Examine progress report Actual results: Progress report is not correct (3 issues above) Expected results: Progress report is correct Additional info:
We need to make both of them consistent.
https://github.com/pulp/pulp_puppet/pull/131 To verify: 1. Download https://forgeapi.puppetlabs.com/v3/files/puppetlabs-concat-1.0.4.tar.gz 2. Extract the above module to a directory <dir> 3. cd <dir> && pulp-puppet-module-builder 4. pulp-admin puppet repo create --repo-id dir_test --feed file:///path/to/<dir> 5. pulp-admin puppet repo sync run --repo-id dir_test 6. Check ~/.pulp/server_calls.log or curl the task to check the progress report. It should be in the state 'success', with a total count of 1 and a finished count of 1. 7. Repeat step 6. 8. Consult ~/.pulp/server_calls.log again. The total count should be 0, and the finished count is 0. You could also add some more modules are rerun pulp-puppet-module-builder to make sure the total count only includes new modules.
build: 2.5.0-0.6.beta
verified [root@cloud-qe-15 ~]# rpm -qa pulp-server pulp-server-2.5.0-0.6.beta.el7.noarch [root@cloud-qe-15 ~]# "progress_report": { "puppet_importer": { "modules": { "error_message": null, "execution_time": 0.007193088531494141, "total_count": 1, "traceback": null, "individual_errors": [], "state": "success", "error_count": 0, "error": "None", "finished_count": 1 }, "metadata": { "query_finished_count": 1, "traceback": null, "execution_time": 0.000804901123046875, "query_total_count": 1, "error_message": null, "state": "success", "error": "None", "current_query": null } } }, "queue": "reserved_resource_worker-1.lab.eng.bos.redhat.com.dq", "state": "finished", "result": { "result": "success", "importer_id": "puppet_importer", "exception": null, "repo_id": "puppet-builds", "traceback": null, "started": "2014-10-02T14:10:12Z", "_ns": "repo_sync_results", "completed": "2014-10-02T14:10:12Z", "importer_type_id": "puppet_importer", "error_message": null, "summary": { "total_execution_time": 0.007997989654541016 }, "added_count": 0, "removed_count": 0, "updated_count": 1, "id": "542d5cc4dba9cb738fd58ac8", "details": { "finished_count": 1, "total_count": 1, "error_count": 0 } }, "error": null, "_id": { "$oid": "542d5cc47ccf31edbac08bae" }, "id": "542d5cc4dba9cb6e3014ee69" } 2014-10-02 10:10:14,989 - INFO - GET request to /pulp/api/v2/tasks/6d9337f2-2015-442a-89c1-93619565fbd3/ with parameters None 2014-10-02 10:10:14,989 - INFO - Response status : 200 2014-10-02 10:10:14,990 - INFO - Response body : { "exception": null, "task_type": "pulp.server.managers.repo.publish.publish", "_href": "/pulp/api/v2/tasks/6d9337f2-2015-442a-89c1-93619565fbd3/", "task_id": "6d9337f2-2015-442a-89c1-93619565fbd3", "tags": [ "pulp:repository:puppet-builds", "pulp:action:publish" ], "finish_time": "2014-10-02T14:10:13Z", "_ns": "task_status", "start_time": "2014-10-02T14:10:13Z", "traceback": null, "spawned_tasks": [], "progress_report": { "puppet_distributor": { "modules": { "error_message": null, "execution_time": 0, "total_count": 1, "traceback": null, "individual_errors": null, "state": "success", "error_count": 0, "error": "None", "finished_count": 1 }, "publishing": { "http": "success", "https": "skipped" }, "metadata": { "execution_time": 0, "state": "success", "error_message": null, "error": "None", "traceback": null } } }, "queue": "reserved_resource_worker-1.lab.eng.bos.redhat.com.dq", "state": "finished", "result": { "result": "success", "exception": null, "repo_id": "puppet-builds", "started": "2014-10-02T14:10:13Z", "_ns": "repo_publish_results", "completed": "2014-10-02T14:10:13Z", "traceback": null, "distributor_type_id": "puppet_distributor", "summary": { "total_execution_time": 0 }, "error_message": null, "distributor_id": "puppet_distributor", "id": "542d5cc5dba9cb738fd58acb", "details": {} }, "error": null, "_id": { "$oid": "542d5cc57ccf31edbac08baf" }, "id": "542d5cc5dba9cb738fd58ac9" } Resync "progress_report": { "puppet_importer": { "modules": { "error_message": null, "execution_time": 0.004343986511230469, "total_count": 0, "traceback": null, "individual_errors": [], "state": "success", "error_count": 0, "error": "None", "finished_count": 0 }, "metadata": { "query_finished_count": 1, "traceback": null, "execution_time": 0.0008900165557861328, "query_total_count": 1, "error_message": null, "state": "success", "error": "None", "current_query": null } } }, "queue": "reserved_resource_worker-1.lab.eng.bos.redhat.com.dq", "state": "finished", "result": { "result": "success", "importer_id": "puppet_importer", "exception": null, "repo_id": "puppet-builds", "traceback": null, "started": "2014-10-02T14:11:25Z", "_ns": "repo_sync_results", "completed": "2014-10-02T14:11:25Z", "importer_type_id": "puppet_importer", "error_message": null, "summary": { "total_execution_time": 0.0052340030670166016 }, "added_count": 0, "removed_count": 0, "updated_count": 0, "id": "542d5d0ddba9cb738fd58acd", "details": { "finished_count": 0, "total_count": 0, "error_count": 0 } }, "error": null, "_id": { "$oid": "542d5d0d7ccf31edbac08bb0" }, "id": "542d5d0ddba9cb6e3014ee6b" } 2014-10-02 10:11:27,143 - INFO - GET request to /pulp/api/v2/tasks/846dbd34-a4ce-4667-a978-3793c7c86290/ with parameters None 2014-10-02 10:11:27,143 - INFO - Response status : 200 2014-10-02 10:11:27,143 - INFO - Response body : { "exception": null, "task_type": "pulp.server.managers.repo.publish.publish", "_href": "/pulp/api/v2/tasks/846dbd34-a4ce-4667-a978-3793c7c86290/", "task_id": "846dbd34-a4ce-4667-a978-3793c7c86290", "tags": [ "pulp:repository:puppet-builds", "pulp:action:publish" ], "finish_time": "2014-10-02T14:11:25Z", "_ns": "task_status", "start_time": "2014-10-02T14:11:25Z", "traceback": null, "spawned_tasks": [], "progress_report": { "puppet_distributor": { "modules": { "error_message": null, "execution_time": 0, "total_count": 1, "traceback": null, "individual_errors": null, "state": "success", "error_count": 0, "error": "None", "finished_count": 1 }, "publishing": { "http": "success", "https": "skipped" }, "metadata": { "execution_time": 0, "state": "success", "error_message": null, "error": "None", "traceback": null } } }, "queue": "reserved_resource_worker-1.lab.eng.bos.redhat.com.dq", "state": "finished", "result": { "result": "success", "exception": null, "repo_id": "puppet-builds", "started": "2014-10-02T14:11:25Z", "_ns": "repo_publish_results", "completed": "2014-10-02T14:11:25Z", "traceback": null, "distributor_type_id": "puppet_distributor", "summary": { "total_execution_time": 0 }, "error_message": null, "distributor_id": "puppet_distributor", "id": "542d5d0ddba9cb738fd58ad0", "details": {} }, "error": null, "_id": { "$oid": "542d5d0d7ccf31edbac08bb1" }, "id": "542d5d0ddba9cb738fd58ace" }