Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1128274 - Puppet module sync progress incorrect for PULP_MANIFEST style repos
Summary: Puppet module sync progress incorrect for PULP_MANIFEST style repos
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: API
Version: 6.0.0
Hardware: Unspecified
OS: Unspecified
medium
unspecified
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Katello QA List
URL:
Whiteboard:
Depends On:
Blocks: sat6-pulp-future
TreeView+ depends on / blocked
 
Reported: 2014-08-08 18:41 UTC by Justin Sherrill
Modified: 2019-09-26 13:49 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-11-24 21:33:42 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Justin Sherrill 2014-08-08 18:41:17 UTC 
Description of problem:

Currently when syncing puppet forge-style repos, the progress report looks something like:

On first sync:
modules:
        traceback: 
        execution_time: 1
        total_count: 2136
        error_message: 
        finished_count: 2000
        state: success
        error_count: 136
        error: None

On next sync:

modules:
        traceback: 
        execution_time: 1
        total_count: 136
        error_message: 
        finished_count: 0
        state: success
        error_count: 136
        error: None


This looks correct.  But puppet repos that use a pulp manifest file are incorrect:

On first sync:

      modules:
        traceback: 
        execution_time: 
        total_count: 4
        error_message: 
        finished_count: 0
        state: running
        error_count: 0
        error: None
        individual_errors: []

On next sync:

      modules:
        traceback: 
        execution_time: 
        total_count: 4
        error_message: 
        finished_count: 0
        state: running
        error_count: 0
        error: None
        individual_errors: []

There are 3 things wrong here:

* State stays 'running'.  Even though the task is finished
* "finished_count" is 0 even though 4 modules were downloaded
* The 2nd sync still shows a total of 4, even though all 4 modules were already synced the first time.

Puppet forge syncing doesn't appear to show any of these issues 

Version-Release number of selected component (if applicable):
pulp-server-2.4.0-0.29.beta.el6.noarch

How reproducible:
Always

Steps to Reproduce:
1.  Sync a pulp_manifest style puppet repo
2.  Examine progress report


Actual results:
Progress report is not correct (3 issues above)

Expected results:
Progress report is correct

Additional info:
Comment 1 Sayli Karmarkar 2014-08-13 16:09:38 UTC 
We need to make both of them consistent.
Comment 2 Jeremy Cline 2014-08-18 19:26:37 UTC 
https://github.com/pulp/pulp_puppet/pull/131

To verify:

1. Download https://forgeapi.puppetlabs.com/v3/files/puppetlabs-concat-1.0.4.tar.gz
2. Extract the above module to a directory <dir>
3. cd <dir> && pulp-puppet-module-builder
4. pulp-admin puppet repo create --repo-id dir_test --feed file:///path/to/<dir>
5. pulp-admin puppet repo sync run --repo-id dir_test
6. Check ~/.pulp/server_calls.log or curl the task to check the progress report. It should be in the state 'success', with a total count of 1 and a finished count of 1.
7. Repeat step 6.
8. Consult ~/.pulp/server_calls.log again. The total count should be 0, and the finished count is 0. You could also add some more modules are rerun pulp-puppet-module-builder to make sure the total count only includes new modules.
Comment 3 Chris Duryee 2014-09-30 13:52:23 UTC 
build: 2.5.0-0.6.beta
Comment 4 Preethi Thomas 2014-10-02 14:16:01 UTC 
verified

[root@cloud-qe-15 ~]# rpm -qa pulp-server
pulp-server-2.5.0-0.6.beta.el7.noarch
[root@cloud-qe-15 ~]# 



  "progress_report": {
    "puppet_importer": {
      "modules": {
        "error_message": null, 
        "execution_time": 0.007193088531494141, 
        "total_count": 1, 
        "traceback": null, 
        "individual_errors": [], 
        "state": "success", 
        "error_count": 0, 
        "error": "None", 
        "finished_count": 1
      }, 
      "metadata": {
        "query_finished_count": 1, 
        "traceback": null, 
        "execution_time": 0.000804901123046875, 
        "query_total_count": 1, 
        "error_message": null, 
        "state": "success", 
        "error": "None", 
        "current_query": null
      }
    }
  }, 
  "queue": "reserved_resource_worker-1.lab.eng.bos.redhat.com.dq", 
  "state": "finished", 
  "result": {
    "result": "success", 
    "importer_id": "puppet_importer", 
    "exception": null, 
    "repo_id": "puppet-builds", 
    "traceback": null, 
    "started": "2014-10-02T14:10:12Z", 
    "_ns": "repo_sync_results", 
    "completed": "2014-10-02T14:10:12Z", 
    "importer_type_id": "puppet_importer", 
    "error_message": null, 
    "summary": {
      "total_execution_time": 0.007997989654541016
    }, 
    "added_count": 0, 
    "removed_count": 0, 
    "updated_count": 1, 
    "id": "542d5cc4dba9cb738fd58ac8", 
    "details": {
      "finished_count": 1, 
      "total_count": 1, 
      "error_count": 0
    }
  }, 
  "error": null, 
  "_id": {
    "$oid": "542d5cc47ccf31edbac08bae"
  }, 
  "id": "542d5cc4dba9cb6e3014ee69"
}

2014-10-02 10:10:14,989 - INFO - GET request to /pulp/api/v2/tasks/6d9337f2-2015-442a-89c1-93619565fbd3/ with parameters None
2014-10-02 10:10:14,989 - INFO - Response status : 200 

2014-10-02 10:10:14,990 - INFO - Response body :
 {
  "exception": null, 
  "task_type": "pulp.server.managers.repo.publish.publish", 
  "_href": "/pulp/api/v2/tasks/6d9337f2-2015-442a-89c1-93619565fbd3/", 
  "task_id": "6d9337f2-2015-442a-89c1-93619565fbd3", 
  "tags": [
    "pulp:repository:puppet-builds", 
    "pulp:action:publish"
  ], 
  "finish_time": "2014-10-02T14:10:13Z", 
  "_ns": "task_status", 
  "start_time": "2014-10-02T14:10:13Z", 
  "traceback": null, 
  "spawned_tasks": [], 
  "progress_report": {
    "puppet_distributor": {
      "modules": {
        "error_message": null, 
        "execution_time": 0, 
        "total_count": 1, 
        "traceback": null, 
        "individual_errors": null, 
        "state": "success", 
        "error_count": 0, 
        "error": "None", 
        "finished_count": 1
      }, 
      "publishing": {
        "http": "success", 
        "https": "skipped"
      }, 
      "metadata": {
        "execution_time": 0, 
        "state": "success", 
        "error_message": null, 
        "error": "None", 
        "traceback": null
      }
    }
  }, 
  "queue": "reserved_resource_worker-1.lab.eng.bos.redhat.com.dq", 
  "state": "finished", 
  "result": {
    "result": "success", 
    "exception": null, 
    "repo_id": "puppet-builds", 
    "started": "2014-10-02T14:10:13Z", 
    "_ns": "repo_publish_results", 
    "completed": "2014-10-02T14:10:13Z", 
    "traceback": null, 
    "distributor_type_id": "puppet_distributor", 
    "summary": {
      "total_execution_time": 0
    }, 
    "error_message": null, 
    "distributor_id": "puppet_distributor", 
    "id": "542d5cc5dba9cb738fd58acb", 
    "details": {}
  }, 
  "error": null, 
  "_id": {
    "$oid": "542d5cc57ccf31edbac08baf"
  }, 
  "id": "542d5cc5dba9cb738fd58ac9"
}




Resync 



  "progress_report": {
    "puppet_importer": {
      "modules": {
        "error_message": null, 
        "execution_time": 0.004343986511230469, 
        "total_count": 0, 
        "traceback": null, 
        "individual_errors": [], 
        "state": "success", 
        "error_count": 0, 
        "error": "None", 
        "finished_count": 0
      }, 
      "metadata": {
        "query_finished_count": 1, 
        "traceback": null, 
        "execution_time": 0.0008900165557861328, 
        "query_total_count": 1, 
        "error_message": null, 
        "state": "success", 
        "error": "None", 
        "current_query": null
      }
    }
  }, 
  "queue": "reserved_resource_worker-1.lab.eng.bos.redhat.com.dq", 
  "state": "finished", 
  "result": {
    "result": "success", 
    "importer_id": "puppet_importer", 
    "exception": null, 
    "repo_id": "puppet-builds", 
    "traceback": null, 
    "started": "2014-10-02T14:11:25Z", 
    "_ns": "repo_sync_results", 
    "completed": "2014-10-02T14:11:25Z", 
    "importer_type_id": "puppet_importer", 
    "error_message": null, 
    "summary": {
      "total_execution_time": 0.0052340030670166016
    }, 
    "added_count": 0, 
    "removed_count": 0, 
    "updated_count": 0, 
    "id": "542d5d0ddba9cb738fd58acd", 
    "details": {
      "finished_count": 0, 
      "total_count": 0, 
      "error_count": 0
    }
  }, 
  "error": null, 
  "_id": {
    "$oid": "542d5d0d7ccf31edbac08bb0"
  }, 
  "id": "542d5d0ddba9cb6e3014ee6b"
}

2014-10-02 10:11:27,143 - INFO - GET request to /pulp/api/v2/tasks/846dbd34-a4ce-4667-a978-3793c7c86290/ with parameters None
2014-10-02 10:11:27,143 - INFO - Response status : 200 

2014-10-02 10:11:27,143 - INFO - Response body :
 {
  "exception": null, 
  "task_type": "pulp.server.managers.repo.publish.publish", 
  "_href": "/pulp/api/v2/tasks/846dbd34-a4ce-4667-a978-3793c7c86290/", 
  "task_id": "846dbd34-a4ce-4667-a978-3793c7c86290", 
  "tags": [
    "pulp:repository:puppet-builds", 
    "pulp:action:publish"
  ], 
  "finish_time": "2014-10-02T14:11:25Z", 
  "_ns": "task_status", 
  "start_time": "2014-10-02T14:11:25Z", 
  "traceback": null, 
  "spawned_tasks": [], 
  "progress_report": {
    "puppet_distributor": {
      "modules": {
        "error_message": null, 
        "execution_time": 0, 
        "total_count": 1, 
        "traceback": null, 
        "individual_errors": null, 
        "state": "success", 
        "error_count": 0, 
        "error": "None", 
        "finished_count": 1
      }, 
      "publishing": {
        "http": "success", 
        "https": "skipped"
      }, 
      "metadata": {
        "execution_time": 0, 
        "state": "success", 
        "error_message": null, 
        "error": "None", 
        "traceback": null
      }
    }
  }, 
  "queue": "reserved_resource_worker-1.lab.eng.bos.redhat.com.dq", 
  "state": "finished", 
  "result": {
    "result": "success", 
    "exception": null, 
    "repo_id": "puppet-builds", 
    "started": "2014-10-02T14:11:25Z", 
    "_ns": "repo_publish_results", 
    "completed": "2014-10-02T14:11:25Z", 
    "traceback": null, 
    "distributor_type_id": "puppet_distributor", 
    "summary": {
      "total_execution_time": 0
    }, 
    "error_message": null, 
    "distributor_id": "puppet_distributor", 
    "id": "542d5d0ddba9cb738fd58ad0", 
    "details": {}
  }, 
  "error": null, 
  "_id": {
    "$oid": "542d5d0d7ccf31edbac08bb1"
  }, 
  "id": "542d5d0ddba9cb738fd58ace"
}
Note You need to log in before you can comment on or make changes to this bug.