Bug 1103311

Summary: [RFE] Have pulp parse the unit_key data from the module metadata if it's not passed in for uploaded puppet modules
Product: [Retired] Pulp Reporter: David Davis <daviddavis>
Component: puppet-supportAssignee: David Davis <daviddavis>
Status: CLOSED CURRENTRELEASE QA Contact: Preethi Thomas <pthomas>
Severity: unspecified Docs Contact:
Priority: medium    
Version: 2.4 BetaCC: daviddavis, mhrivnak, mmccune, pthomas
Target Milestone: ---Keywords: Triaged
Target Release: 2.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-08-09 06:56:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 950743, 1097209    

Description David Davis 2014-05-30 17:18:57 UTC
Katello (the webserver part) basically passes the upload bits calls from the CLI straight through to Pulp so we never see the full file. We could also store a copy of the file but it would probably just be easier to have pulp use the metadata from the module to populate the unit_key data. This is what we're doing in Katello already. Having this small code change would simplify content uploads in Katello a lot.

Comment 1 David Davis 2014-05-30 17:19:24 UTC
Here's the patch I am proposing:

https://github.com/daviddavis/pulp_puppet/commit/f06d1186c88df3f6ea9e27965827467b432293e0

Comment 2 David Davis 2014-05-30 17:25:02 UTC
https://github.com/pulp/pulp_puppet/pull/126

Comment 3 Michael Hrivnak 2014-06-02 13:10:02 UTC
This is worth doing in pulp. Having glanced at the existing code, the workflow is a bit odd. I think this RFE can be accomplished with relatively little change, as the above PR demonstrates.

The only reservation I have about the above PR is that it allows instantiation of a model object with no data. I think we can re-work the process slightly to avoid that.

Comment 4 David Davis 2014-06-05 12:12:29 UTC
https://github.com/pulp/pulp_puppet/pull/126

Comment 5 David Davis 2014-06-05 12:13:33 UTC
Commit info:

58c075b0009eda30ab285e66a4fa34e095ff8e65

1103311 - Extract any unit_key fields from metadata
If no unit_key metadata is passed in for uploaded puppet modules, just use the
module metadata to populate name, author, and version.

https://bugzilla.redhat.com/show_bug.cgi?id=1103311

Comment 6 Michael Hrivnak 2014-06-05 13:03:04 UTC
https://github.com/pulp/pulp_puppet/pull/126

Comment 7 Jeff Ortel 2014-06-11 01:33:07 UTC
build: 2.4.0-0.20.beta

Comment 8 Preethi Thomas 2014-06-30 17:49:06 UTC
could you please include the steps on how to verify this bug

Comment 9 David Davis 2014-07-10 18:37:32 UTC
This might be really hard to test without scripting something as it requires:

1) Directly hitting the API
2) Reading in a file and sending its contents

That said, Katello is currently using this feature and it's been tested as part of https://bugzilla.redhat.com/show_bug.cgi?id=1097209.

Comment 10 Preethi Thomas 2014-07-11 15:11:12 UTC
verified

[root@yttrium ~]# rpm -qa pulp-server
pulp-server-2.4.0-0.23.beta.el6.noarch

[root@yttrium ~]#  curl -H "Accept: application/json" -X PUT -k -u admin:admin  --data-binary @puppetlabs-java-1.1.1.tar.gz  'https://yttrium.idm.lab.bos.redhat.com/pulp/api/v2/content/uploads/41a2ecd6-ca45-4ba5-97a1-3f287e7b751a/0/'
null[root@yttrium ~]# 


[root@yttrium ~]# curl -H "Accept: application/json" -X POST -k -u admin:admin -d '{"unit_type_id": "puppet_module", "upload_id": "41a2ecd6-ca45-4ba5-97a1-3f287e7b751a/", "unit_key": {"version": "1.1.1", "name": "java", "author": "puppetlabs"}, "unit_metadata": {}}' 'https://yttrium.idm.lab.bos.redhat.com/pulp/api/v2/repositories/repo/actions/import_upload/'
{"spawned_tasks": [{"_href": "/pulp/api/v2/tasks/0cb1bf50-4c2b-4f5e-881f-fb3e5c11cf4d/", "task_id": "0cb1bf50-4c2b-4f5e-881f-fb3e5c11cf4d"}], "result": null, "error": null}[root@yttrium ~]# 
[root@yttrium ~]# 


[root@yttrium ~]# curl -H "Accept: application/json" -X GET -k -u admin:admin -d '{"upload_id":"41a2ecd6-ca45-4ba5-97a1-3f287e7b751a","unit_type_id":"puppet_module","unit_key":{"author":"puppetlabs","name":"java","version":"1.1.1"}}' 'https://yttrium.idm.lab.bos.redhat.com/pulp/api/v2/tasks/0cb1bf50-4c2b-4f5e-881f-fb3e5c11cf4d/' |python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
128   749  107   749    0   150   5714   1144 --:--:-- --:--:-- --:--:--  7216
{
    "_href": "/pulp/api/v2/tasks/0cb1bf50-4c2b-4f5e-881f-fb3e5c11cf4d/", 
    "_id": {
        "$oid": "53bff3d35e162a0c2626fab1"
    }, 
    "_ns": "task_status", 
    "error": null, 
    "exception": null, 
    "finish_time": "2014-07-11T14:25:23Z", 
    "id": "53bff3d37bcdb34d0750c025", 
    "progress_report": {}, 
    "queue": "reserved_resource_worker-3.lab.bos.redhat.com.dq", 
    "result": {
        "details": {}, 
        "success_flag": false, 
        "summary": "/var/lib/pulp/uploads/41a2ecd6-ca45-4ba5-97a1-3f287e7b751a/"
    }, 
    "spawned_tasks": [], 
    "start_time": "2014-07-11T14:25:23Z", 
    "state": "finished", 
    "tags": [
        "pulp:repository:repo", 
        "pulp:action:import_upload"
    ], 
    "task_id": "0cb1bf50-4c2b-4f5e-881f-fb3e5c11cf4d", 
    "task_type": "pulp.server.managers.content.upload.import_uploaded_unit", 
    "traceback": null
}

Comment 11 Randy Barlow 2014-08-09 06:56:37 UTC
This has been fixed in Pulp 2.4.0-1.