Bug 1128274 - Puppet module sync progress incorrect for PULP_MANIFEST style repos
Summary: Puppet module sync progress incorrect for PULP_MANIFEST style repos
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: API
Version: 6.0.0
Hardware: Unspecified
OS: Unspecified
medium
unspecified vote
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Katello QA List
URL:
Whiteboard:
Depends On:
Blocks: sat6-pulp-future
TreeView+ depends on / blocked
 
Reported: 2014-08-08 18:41 UTC by Justin Sherrill
Modified: 2017-07-26 19:44 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-11-24 21:33:42 UTC


Attachments (Terms of Use)

Description Justin Sherrill 2014-08-08 18:41:17 UTC
Description of problem:

Currently when syncing puppet forge-style repos, the progress report looks something like:

On first sync:
modules:
        traceback: 
        execution_time: 1
        total_count: 2136
        error_message: 
        finished_count: 2000
        state: success
        error_count: 136
        error: None

On next sync:

modules:
        traceback: 
        execution_time: 1
        total_count: 136
        error_message: 
        finished_count: 0
        state: success
        error_count: 136
        error: None


This looks correct.  But puppet repos that use a pulp manifest file are incorrect:

On first sync:

      modules:
        traceback: 
        execution_time: 
        total_count: 4
        error_message: 
        finished_count: 0
        state: running
        error_count: 0
        error: None
        individual_errors: []

On next sync:

      modules:
        traceback: 
        execution_time: 
        total_count: 4
        error_message: 
        finished_count: 0
        state: running
        error_count: 0
        error: None
        individual_errors: []

There are 3 things wrong here:

* State stays 'running'.  Even though the task is finished
* "finished_count" is 0 even though 4 modules were downloaded
* The 2nd sync still shows a total of 4, even though all 4 modules were already synced the first time.

Puppet forge syncing doesn't appear to show any of these issues 

Version-Release number of selected component (if applicable):
pulp-server-2.4.0-0.29.beta.el6.noarch

How reproducible:
Always

Steps to Reproduce:
1.  Sync a pulp_manifest style puppet repo
2.  Examine progress report


Actual results:
Progress report is not correct (3 issues above)

Expected results:
Progress report is correct

Additional info:

Comment 1 Sayli Karmarkar 2014-08-13 16:09:38 UTC
We need to make both of them consistent.

Comment 2 Jeremy Cline 2014-08-18 19:26:37 UTC
https://github.com/pulp/pulp_puppet/pull/131

To verify:

1. Download https://forgeapi.puppetlabs.com/v3/files/puppetlabs-concat-1.0.4.tar.gz
2. Extract the above module to a directory <dir>
3. cd <dir> && pulp-puppet-module-builder
4. pulp-admin puppet repo create --repo-id dir_test --feed file:///path/to/<dir>
5. pulp-admin puppet repo sync run --repo-id dir_test
6. Check ~/.pulp/server_calls.log or curl the task to check the progress report. It should be in the state 'success', with a total count of 1 and a finished count of 1.
7. Repeat step 6.
8. Consult ~/.pulp/server_calls.log again. The total count should be 0, and the finished count is 0. You could also add some more modules are rerun pulp-puppet-module-builder to make sure the total count only includes new modules.

Comment 3 Chris Duryee 2014-09-30 13:52:23 UTC
build: 2.5.0-0.6.beta

Comment 4 Preethi Thomas 2014-10-02 14:16:01 UTC
verified

[root@cloud-qe-15 ~]# rpm -qa pulp-server
pulp-server-2.5.0-0.6.beta.el7.noarch
[root@cloud-qe-15 ~]# 



  "progress_report": {
    "puppet_importer": {
      "modules": {
        "error_message": null, 
        "execution_time": 0.007193088531494141, 
        "total_count": 1, 
        "traceback": null, 
        "individual_errors": [], 
        "state": "success", 
        "error_count": 0, 
        "error": "None", 
        "finished_count": 1
      }, 
      "metadata": {
        "query_finished_count": 1, 
        "traceback": null, 
        "execution_time": 0.000804901123046875, 
        "query_total_count": 1, 
        "error_message": null, 
        "state": "success", 
        "error": "None", 
        "current_query": null
      }
    }
  }, 
  "queue": "reserved_resource_worker-1@cloud-qe-15.idmqe.lab.eng.bos.redhat.com.dq", 
  "state": "finished", 
  "result": {
    "result": "success", 
    "importer_id": "puppet_importer", 
    "exception": null, 
    "repo_id": "puppet-builds", 
    "traceback": null, 
    "started": "2014-10-02T14:10:12Z", 
    "_ns": "repo_sync_results", 
    "completed": "2014-10-02T14:10:12Z", 
    "importer_type_id": "puppet_importer", 
    "error_message": null, 
    "summary": {
      "total_execution_time": 0.007997989654541016
    }, 
    "added_count": 0, 
    "removed_count": 0, 
    "updated_count": 1, 
    "id": "542d5cc4dba9cb738fd58ac8", 
    "details": {
      "finished_count": 1, 
      "total_count": 1, 
      "error_count": 0
    }
  }, 
  "error": null, 
  "_id": {
    "$oid": "542d5cc47ccf31edbac08bae"
  }, 
  "id": "542d5cc4dba9cb6e3014ee69"
}

2014-10-02 10:10:14,989 - INFO - GET request to /pulp/api/v2/tasks/6d9337f2-2015-442a-89c1-93619565fbd3/ with parameters None
2014-10-02 10:10:14,989 - INFO - Response status : 200 

2014-10-02 10:10:14,990 - INFO - Response body :
 {
  "exception": null, 
  "task_type": "pulp.server.managers.repo.publish.publish", 
  "_href": "/pulp/api/v2/tasks/6d9337f2-2015-442a-89c1-93619565fbd3/", 
  "task_id": "6d9337f2-2015-442a-89c1-93619565fbd3", 
  "tags": [
    "pulp:repository:puppet-builds", 
    "pulp:action:publish"
  ], 
  "finish_time": "2014-10-02T14:10:13Z", 
  "_ns": "task_status", 
  "start_time": "2014-10-02T14:10:13Z", 
  "traceback": null, 
  "spawned_tasks": [], 
  "progress_report": {
    "puppet_distributor": {
      "modules": {
        "error_message": null, 
        "execution_time": 0, 
        "total_count": 1, 
        "traceback": null, 
        "individual_errors": null, 
        "state": "success", 
        "error_count": 0, 
        "error": "None", 
        "finished_count": 1
      }, 
      "publishing": {
        "http": "success", 
        "https": "skipped"
      }, 
      "metadata": {
        "execution_time": 0, 
        "state": "success", 
        "error_message": null, 
        "error": "None", 
        "traceback": null
      }
    }
  }, 
  "queue": "reserved_resource_worker-1@cloud-qe-15.idmqe.lab.eng.bos.redhat.com.dq", 
  "state": "finished", 
  "result": {
    "result": "success", 
    "exception": null, 
    "repo_id": "puppet-builds", 
    "started": "2014-10-02T14:10:13Z", 
    "_ns": "repo_publish_results", 
    "completed": "2014-10-02T14:10:13Z", 
    "traceback": null, 
    "distributor_type_id": "puppet_distributor", 
    "summary": {
      "total_execution_time": 0
    }, 
    "error_message": null, 
    "distributor_id": "puppet_distributor", 
    "id": "542d5cc5dba9cb738fd58acb", 
    "details": {}
  }, 
  "error": null, 
  "_id": {
    "$oid": "542d5cc57ccf31edbac08baf"
  }, 
  "id": "542d5cc5dba9cb738fd58ac9"
}




Resync 



  "progress_report": {
    "puppet_importer": {
      "modules": {
        "error_message": null, 
        "execution_time": 0.004343986511230469, 
        "total_count": 0, 
        "traceback": null, 
        "individual_errors": [], 
        "state": "success", 
        "error_count": 0, 
        "error": "None", 
        "finished_count": 0
      }, 
      "metadata": {
        "query_finished_count": 1, 
        "traceback": null, 
        "execution_time": 0.0008900165557861328, 
        "query_total_count": 1, 
        "error_message": null, 
        "state": "success", 
        "error": "None", 
        "current_query": null
      }
    }
  }, 
  "queue": "reserved_resource_worker-1@cloud-qe-15.idmqe.lab.eng.bos.redhat.com.dq", 
  "state": "finished", 
  "result": {
    "result": "success", 
    "importer_id": "puppet_importer", 
    "exception": null, 
    "repo_id": "puppet-builds", 
    "traceback": null, 
    "started": "2014-10-02T14:11:25Z", 
    "_ns": "repo_sync_results", 
    "completed": "2014-10-02T14:11:25Z", 
    "importer_type_id": "puppet_importer", 
    "error_message": null, 
    "summary": {
      "total_execution_time": 0.0052340030670166016
    }, 
    "added_count": 0, 
    "removed_count": 0, 
    "updated_count": 0, 
    "id": "542d5d0ddba9cb738fd58acd", 
    "details": {
      "finished_count": 0, 
      "total_count": 0, 
      "error_count": 0
    }
  }, 
  "error": null, 
  "_id": {
    "$oid": "542d5d0d7ccf31edbac08bb0"
  }, 
  "id": "542d5d0ddba9cb6e3014ee6b"
}

2014-10-02 10:11:27,143 - INFO - GET request to /pulp/api/v2/tasks/846dbd34-a4ce-4667-a978-3793c7c86290/ with parameters None
2014-10-02 10:11:27,143 - INFO - Response status : 200 

2014-10-02 10:11:27,143 - INFO - Response body :
 {
  "exception": null, 
  "task_type": "pulp.server.managers.repo.publish.publish", 
  "_href": "/pulp/api/v2/tasks/846dbd34-a4ce-4667-a978-3793c7c86290/", 
  "task_id": "846dbd34-a4ce-4667-a978-3793c7c86290", 
  "tags": [
    "pulp:repository:puppet-builds", 
    "pulp:action:publish"
  ], 
  "finish_time": "2014-10-02T14:11:25Z", 
  "_ns": "task_status", 
  "start_time": "2014-10-02T14:11:25Z", 
  "traceback": null, 
  "spawned_tasks": [], 
  "progress_report": {
    "puppet_distributor": {
      "modules": {
        "error_message": null, 
        "execution_time": 0, 
        "total_count": 1, 
        "traceback": null, 
        "individual_errors": null, 
        "state": "success", 
        "error_count": 0, 
        "error": "None", 
        "finished_count": 1
      }, 
      "publishing": {
        "http": "success", 
        "https": "skipped"
      }, 
      "metadata": {
        "execution_time": 0, 
        "state": "success", 
        "error_message": null, 
        "error": "None", 
        "traceback": null
      }
    }
  }, 
  "queue": "reserved_resource_worker-1@cloud-qe-15.idmqe.lab.eng.bos.redhat.com.dq", 
  "state": "finished", 
  "result": {
    "result": "success", 
    "exception": null, 
    "repo_id": "puppet-builds", 
    "started": "2014-10-02T14:11:25Z", 
    "_ns": "repo_publish_results", 
    "completed": "2014-10-02T14:11:25Z", 
    "traceback": null, 
    "distributor_type_id": "puppet_distributor", 
    "summary": {
      "total_execution_time": 0
    }, 
    "error_message": null, 
    "distributor_id": "puppet_distributor", 
    "id": "542d5d0ddba9cb738fd58ad0", 
    "details": {}
  }, 
  "error": null, 
  "_id": {
    "$oid": "542d5d0d7ccf31edbac08bb1"
  }, 
  "id": "542d5d0ddba9cb738fd58ace"
}


Note You need to log in before you can comment on or make changes to this bug.