Bug 1123446

Summary:	Puppet Module sync incorrectly calculates module name
Product:	[Retired] Pulp	Reporter:	Justin Sherrill <jsherril>
Component:	puppet-support	Assignee:	Jeremy Cline <jcline>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Preethi Thomas <pthomas>
Severity:	unspecified	Docs Contact:
Priority:	medium
Version:	2.4 Beta	CC:	jcline, katello-bugs, katello-qa-list, mmccune, peter.vreman, pthomas, rbarlow, skarmark
Target Milestone:	---	Keywords:	Triaged
Target Release:	2.4.1
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1123445	Environment:
Last Closed:	2014-09-23 17:54:09 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1123445
Bug Blocks:	1122832, 1131719

Description Justin Sherrill 2014-07-25 16:26:18 UTC

+++ This bug was initially created as a clone of Bug #1123445 +++

Description of problem:

When using puppet module sync on a puppet repo created using pulp-puppet-module-builder, puppet modules are imported with a name of 'author-name' instead of just 'name'

The same issue does not occur when using hammer or the webUI to upload the modules 


Version-Release number of selected component (if applicable):

pulp-server-2.4.0-0.23.beta.el6sat.noarch

How reproducible:
Always

Steps to Reproduce:
1.  Download a puppet module such as puppetlabs-concat from puppet forge
2.  Extract the module in a directory /modules
3.  cd /modules && pulp-puppet-module-builder
4.  chmod 755 /modules
5.  Create a puppet repo with url of  file:///modules
6.  Sync the repo

Actual results:
The puppet module is imported with a name of 'puppetlabs-concat'.  The module is viewable via content search or the 'content view puppet modules' page

Expected results:
The puppet module is imported with a name of 'concat'


Additional info:

--- Additional comment from Justin Sherrill on 2014-07-25 12:22:43 EDT ---

patch from a customer to fix the issue.  (Note it was sent in email, so spacing is likely off)

diff -urN /usr/lib/python2.6/site-packages/pulp_puppet/plugins/importers.old/directory.py /usr/lib/python2.6/site-packages/pulp_puppet/plugins/importers/directory.py
--- /usr/lib/python2.6/site-packages/pulp_puppet/plugins/importers.old/directory.py 2014-06-16 13:00:58.706453008 +0000
+++ /usr/lib/python2.6/site-packages/pulp_puppet/plugins/importers/directory.py 2014-06-16 13:01:10.871179975 +0000
@@ -271,6 +271,7 @@
if self.canceled:
return []
puppet_manifest = self._extract_metadata(module_path)
+ puppet_manifest["name"]=puppet_manifest["name"].replace(puppet_manifest["author"]+"-","")
module = Module.from_dict(puppet_manifest)
if inventory.already_associated(module):
continue

Comment 1 Peter Vreman 2014-07-28 13:22:59 UTC

See also this commit in pulp_puppet that fixes this for manual uploaded modules. But the directory.py importer is not updated to use the new function:

https://github.com/pulp/pulp_puppet/commit/58c075b0009eda30ab285e66a4fa34e095ff8e65

Comment 2 Peter Vreman 2014-07-28 14:12:15 UTC

The following patch looks like to work also:

--- directory.py.old    2014-07-28 13:15:58.205905361 +0000
+++ directory.py        2014-07-28 14:09:41.284902425 +0000
@@ -271,7 +271,7 @@
             if self.canceled:
                 return []
             puppet_manifest = self._extract_metadata(module_path)
-            module = Module.from_dict(puppet_manifest)
+            module = Module.from_json(puppet_manifest)
             if inventory.already_associated(module):
                 continue
             _LOG.info(IMPORT_MODULE % dict(mod=module_path))

Comment 3 Jeremy Cline 2014-07-31 13:03:53 UTC

https://github.com/pulp/pulp_puppet/pull/127

Comment 4 Jeremy Cline 2014-08-11 14:43:51 UTC

The above pull request also fixes the inability to import modules using the old Puppet naming style (authorname/modulename rather than authorname-modulename), which is still legal.

To verify:
1. Download the following puppet modules. The first uses the new name style, and the second uses the old naming style:

https://forgeapi.puppetlabs.com/v3/files/puppetlabs-concat-1.0.4.tar.gz
https://forgeapi.puppetlabs.com/v3/files/jcline-test-0.1.0.tar.gz

2. Extract the modules into a directory <module-dir>
3. cd <module-dir> && pulp-puppet-module-builder
4. chmod 755 <module-dir>
5. Create a puppet repo with url of file://<absolute-path-to-module-dir>
6. Sync the repo
7. List the Puppet modules in the repo using 'pulp-admin puppet repo modules --repo-id <repo-name>'
8. The first module should have the name 'concat' and the author 'puppetlabs'
9. The second module should have the name 'test' and the author 'jcline'

Comment 5 Randy Barlow 2014-08-13 19:59:06 UTC

This was fixed in the 2.4.1-0.1.alpha build.

Comment 6 Preethi Thomas 2014-08-18 13:52:26 UTC

verified
[root@cloud-qe-1-vm-2 module]# rpm -qa pulp-server
pulp-server-2.4.1-0.1.alpha.el7.noarch
[root@cloud-qe-1-vm-2 module]# 

[root@cloud-qe-1-vm-2 ~]# mkdir /tmp/module/
[root@cloud-qe-1-vm-2 ~]# 
[root@cloud-qe-1-vm-2 ~]# 
[root@cloud-qe-1-vm-2 ~]# 
[root@cloud-qe-1-vm-2 ~]# cd /tmp/module/
[root@cloud-qe-1-vm-2 module]# wget https://forgeapi.puppetlabs.com/v3/files/puppetlabs-concat-1.0.4.tar.gz
--2014-08-18 09:49:27--  https://forgeapi.puppetlabs.com/v3/files/puppetlabs-concat-1.0.4.tar.gz
Resolving forgeapi.puppetlabs.com (forgeapi.puppetlabs.com)... 23.236.57.230
Connecting to forgeapi.puppetlabs.com (forgeapi.puppetlabs.com)|23.236.57.230|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 17588 (17K) [application/octet-stream]
Saving to: ‘puppetlabs-concat-1.0.4.tar.gz’

100%[======================================>] 17,588      --.-K/s   in 0s      

2014-08-18 09:49:27 (124 MB/s) - ‘puppetlabs-concat-1.0.4.tar.gz’ saved [17588/17588]

[root@cloud-qe-1-vm-2 module]# wget https://forgeapi.puppetlabs.com/v3/files/jcline-test-0.1.0.tar.gz
--2014-08-18 09:49:35--  https://forgeapi.puppetlabs.com/v3/files/jcline-test-0.1.0.tar.gz
Resolving forgeapi.puppetlabs.com (forgeapi.puppetlabs.com)... 23.236.57.230
Connecting to forgeapi.puppetlabs.com (forgeapi.puppetlabs.com)|23.236.57.230|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1869 (1.8K) [application/octet-stream]
Saving to: ‘jcline-test-0.1.0.tar.gz’

100%[======================================>] 1,869       --.-K/s   in 0s      

2014-08-18 09:49:36 (295 MB/s) - ‘jcline-test-0.1.0.tar.gz’ saved [1869/1869]

[root@cloud-qe-1-vm-2 module]# 
[root@cloud-qe-1-vm-2 module]# 
[root@cloud-qe-1-vm-2 module]# pulp-puppet-module-builder
cd /tmp/module
git status
find . -name init.pp
cd /tmp/module
cd /tmp/module
cd /tmp/module
[root@cloud-qe-1-vm-2 module]# pulp-admin puppet repo create --repo-id puppet --feed file://tmp/module/
Successfully created repository [puppet]

[root@cloud-qe-1-vm-2 module]# 
[root@cloud-qe-1-vm-2 module]# 
[root@cloud-qe-1-vm-2 module]# pulp-admin puppet repo sync run --repo-id puppet
+----------------------------------------------------------------------+
                   Synchronizing Repository [puppet]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Downloading metadata...
[==================================================] 100%
Metadata Query: 1/1 items
... completed

Downloading new modules...
[                                                  ] 0%
Module: 0/2 items

Task Succeeded



Publishing modules...
[==================================================] 100%
Module: 2/2 items
... completed

Generating repository metadata...
[-]
... completed

Publishing repository over HTTP...
... completed

Publishing repository over HTTPS...
... skipped

Task Succeeded


[root@cloud-qe-1-vm-2 module]# pulp-admin puppet repo modules --repo-id puppet
Name:          concat
Version:       1.0.4
Author:        puppetlabs
Checksum:      None
Checksum Type: sha256
Dependencies:  
Description:   Concat module
License:       Apache 2.0
Project Page:  http://github.com/puppetlabs/puppetlabs-concat
Source:        git://github.com/puppetlabs/puppetlabs-concat.git
Summary:       Concat module
Tag List:      None

Name:          test
Version:       0.1.0
Author:        jcline
Checksum:      None
Checksum Type: sha256
Dependencies:  
Description:   UNKNOWN
License:       Apache License, Version 2.0
Project Page:  UNKNOWN
Source:        UNKNOWN
Summary:       UNKNOWN
Tag List:      None


[root@cloud-qe-1-vm-2 module]#

Comment 7 Randy Barlow 2014-08-20 03:31:03 UTC

We are rebasing pulp-2.4.1, and so all of these bugs can no longer be considered VERIFIED. We will need to recheck them against the rebased build to make sure the cherry picking strategy did not introduce any regressions.

Comment 8 Randy Barlow 2014-08-20 18:29:47 UTC

I have cherry picked the commit that fixed this bug[0] into the 2.4.1-dev branch. The commit applied cleanly, and all unit tests pass.

[0] https://github.com/pulp/pulp_puppet/commit/891709e5898a26d9f4f02828ff122f7600f6e35c#diff-d41d8cd98f00b204e9800998ecf8427e

Comment 9 Randy Barlow 2014-08-20 22:16:43 UTC

This was fixed in 2.4.1-0.2.alpha, available in Pulp's testing repository.

Comment 10 Preethi Thomas 2014-08-21 02:06:52 UTC

verified
root@qe-blade-08 module]# rpm -qa pulp-server
pulp-server-2.4.1-0.2.alpha.el6.noarch

[root@qe-blade-13 module]# rpm -qa pulp-server
pulp-server-2.4.1-0.2.alpha.el7.noarch
[root@qe-blade-13 module]# 

[root@qe-blade-08 module]# 

[root@qe-blade-08 module]# wget https://forgeapi.puppetlabs.com/v3/files/puppetlabs-concat-1.0.4.tar.gz --no-check-certificate
--2014-08-20 22:00:46--  https://forgeapi.puppetlabs.com/v3/files/puppetlabs-concat-1.0.4.tar.gz
Resolving forgeapi.puppetlabs.com... 23.236.57.230
Connecting to forgeapi.puppetlabs.com|23.236.57.230|:443... connected.
WARNING: certificate common name “forge.puppetlabs.com” doesn’t match requested host name “forgeapi.puppetlabs.com”.
HTTP request sent, awaiting response... 200 OK
Length: 17588 (17K) [application/octet-stream]
Saving to: “puppetlabs-concat-1.0.4.tar.gz”

100%[======================================>] 17,588      --.-K/s   in 0.07s   

2014-08-20 22:00:47 (255 KB/s) - “puppetlabs-concat-1.0.4.tar.gz” saved [17588/17588]

[root@qe-blade-08 module]# wget https://forgeapi.puppetlabs.com/v3/files/jcline-test-0.1.0.tar.gz --no-check-certificate
--2014-08-20 22:00:55--  https://forgeapi.puppetlabs.com/v3/files/jcline-test-0.1.0.tar.gz
Resolving forgeapi.puppetlabs.com... 23.236.57.230
Connecting to forgeapi.puppetlabs.com|23.236.57.230|:443... connected.
WARNING: certificate common name “forge.puppetlabs.com” doesn’t match requested host name “forgeapi.puppetlabs.com”.
HTTP request sent, awaiting response... 200 OK
Length: 1869 (1.8K) [application/octet-stream]
Saving to: “jcline-test-0.1.0.tar.gz”

100%[======================================>] 1,869       --.-K/s   in 0s      

2014-08-20 22:00:55 (327 MB/s) - “jcline-test-0.1.0.tar.gz” saved [1869/1869]

[root@qe-blade-08 module]# ls
jcline-test-0.1.0.tar.gz  puppetlabs-concat-1.0.4.tar.gz
[root@qe-blade-08 module]# yum install pulp-puppet-tools
Loaded plugins: product-id, pulp-profile-update, security, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package pulp-puppet-tools.noarch 0:2.4.1-0.2.alpha.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package              Arch      Version                Repository          Size
================================================================================
Installing:
 pulp-puppet-tools    noarch    2.4.1-0.2.alpha.el6    pulp-v2-testing     31 k

Transaction Summary
================================================================================
Install       1 Package(s)

Total download size: 31 k
Installed size: 42 k
Is this ok [y/N]: y
Downloading Packages:
pulp-puppet-tools-2.4.1-0.2.alpha.el6.noarch.rpm         |  31 kB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : pulp-puppet-tools-2.4.1-0.2.alpha.el6.noarch                 1/1 
  Verifying  : pulp-puppet-tools-2.4.1-0.2.alpha.el6.noarch                 1/1 

Installed:
  pulp-puppet-tools.noarch 0:2.4.1-0.2.alpha.el6                                

Complete!
[root@qe-blade-08 module]# 
[root@qe-blade-08 module]# 
[root@qe-blade-08 module]# ls
jcline-test-0.1.0.tar.gz  puppetlabs-concat-1.0.4.tar.gz
[root@qe-blade-08 module]# pulp-puppet-module-builder
cd /tmp/module
git status
find . -name init.pp
cd /tmp/module
cd /tmp/module
cd /tmp/module
[root@qe-blade-08 module]# 
[root@qe-blade-08 module]# pulp-admin puppet repo create --repo-id puppet --feed file://tmp/module/
Successfully created repository [puppet]

[root@qe-blade-08 module]# pulp-admin puppet repo sync run --repo-id puppet
+----------------------------------------------------------------------+
                   Synchronizing Repository [puppet]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Downloading metadata...
[==================================================] 100%
Metadata Query: 1/1 items
... completed

Downloading new modules...
[                                                  ] 0%
Module: 0/2 items

Task Succeeded



Publishing modules...
[==================================================] 100%
Module: 2/2 items
... completed

Generating repository metadata...
[-]
... completed

Publishing repository over HTTP...
... completed

Publishing repository over HTTPS...
... skipped

Task Succeeded


[root@qe-blade-08 module]#  pulp-admin puppet repo modules --repo-id puppet
Name:          concat
Version:       1.0.4
Author:        puppetlabs
Checksum:      None
Checksum Type: sha256
Dependencies:  
Description:   Concat module
License:       Apache 2.0
Project Page:  http://github.com/puppetlabs/puppetlabs-concat
Source:        git://github.com/puppetlabs/puppetlabs-concat.git
Summary:       Concat module
Tag List:      None

Name:          test
Version:       0.1.0
Author:        jcline
Checksum:      None
Checksum Type: sha256
Dependencies:  
Description:   UNKNOWN
License:       Apache License, Version 2.0
Project Page:  UNKNOWN
Source:        UNKNOWN
Summary:       UNKNOWN
Tag List:      None


[root@qe-blade-08 module]#

Comment 11 Peter Vreman 2014-09-22 09:40:14 UTC

Bug can be closed, confirmed that it works in Sat6 GA - 6.0.4

Comment 12 Randy Barlow 2014-09-23 17:54:09 UTC

This is fixed in Pulp-2.4.1-1.