Bug 1886005 - pulp-puppet-module-builder uses depreciated pulp command and is unable to build any puppet module [NEEDINFO]
Summary: pulp-puppet-module-builder uses depreciated pulp command and is unable to bui...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.8.0
Hardware: Unspecified
OS: Unspecified
high
high vote
Target Milestone: 6.9.2
Assignee: satellite6-bugs
QA Contact: Lai
URL:
Whiteboard:
: 1869382 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-10-07 12:49 UTC by Mirek Długosz
Modified: 2021-08-13 03:04 UTC (History)
15 users (show)

Fixed In Version: pulp-puppet-2.21.5.1-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-05-20 18:05:30 UTC
Target Upstream Version:
kurathod: needinfo? (satellite6-bugs)
sfroemer: needinfo? (satellite6-bugs)


Attachments (Terms of Use)
patch for comment#7 (650 bytes, patch)
2020-12-11 17:02 UTC, Steffen Froemer
no flags Details | Diff
pulp-puppet-tools-2.21.3-3.HFRHBZ1886005.el7sat.noarch.rpm (29.95 KB, application/x-rpm)
2021-04-08 21:20 UTC, Mike McCune
no flags Details
pulp-puppet-tools-2.21.3-4.HFRHBZ1886005.el7sat.noarch.rpm (29.95 KB, application/x-rpm)
2021-04-30 15:32 UTC, Mike McCune
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2021:2074 0 None None None 2021-05-20 18:05:40 UTC

Description Mirek Długosz 2020-10-07 12:49:45 UTC
According to Satellite docs[0] supported way of creating puppet module from git repository is by using pulp-puppet-module-builder tool. However, this tool no longer works on Satellite 6.8:

#v+
# pulp-puppet-module-builder --output-dir=/var/www/puppet-modules --url=https://github.com/redhataccess/puppet-access_insights_client.git --branch=master
cd /root
git clone --recursive https://github.com/redhataccess/puppet-access_insights_client.git
cd puppet-access_insights_client
git status
git remote show -n origin
git fetch
git fetch --tags
git checkout master
git pull
find . -name Modulefile -o -name metadata.json
puppet module build .
Error: This action has been replaced by Puppet Development Kit. For more information visit https://puppet.com/docs/pdk/latest/pdk.html.
Error: Try 'puppet help module build' for usage
#v-

After running pulp-puppet-module-builder, `/var/www/puppet-modules` directory remains empty.

Running `puppet module build --verbose .` or `puppet module build --debug .` results in same message, without any additional information. Looks like entire command has been replaced by thin wrapper that prints depreciation message and exits.


Found on:
Satellite 6.8 snap 18
satellite-6.8.0-1.el7sat.noarch
katello-3.16.0-1.el7sat.noarch
foreman-2.1.2.19-1.el7sat.noarch
pulp-server-2.21.3-1.el7sat.noarch
pulp-puppet-tools-2.21.3-2.el7sat.noarch


[0] https://access.redhat.com/documentation/en-us/red_hat_satellite/6.8-beta/html/puppet_guide/chap-red_hat_satellite-puppet_guide-adding_puppet_modules_to_red_hat_satellite_6#sect-Red_Hat_Satellite-Puppet_Guide-Adding_Puppet_Modules_to_Red_Hat_Satellite_6-Adding_Puppet_Modules_from_a_Git_Repository

Comment 4 Stefan Nemeth 2020-12-01 16:02:42 UTC
workaround:

download puppet development kit

#wget --content-disposition 'https://pm.puppet.com/cgi-bin/pdk_download.cgi?dist=el&rel=6&arch=x86_64&ver=latest'

install pdk-1.18.1.0-1.el7.x86_64.rpm  



change the file /usr/lib/python2.7/site-packages/pulp_puppet/tools/puppet_module_builder.py:


shell line 275 in file /usr/lib/python2.7/site-packages/pulp_puppet/tools/puppet_module_builder.py  to 

    for path in find_modules():
        shell('pdk build --force %s' % path)
        pkg_dir = os.path.join(path, PKG_DIR)
        publish_module(pkg_dir, options.output_dir, options.force)

Comment 5 Melanie Falz 2020-12-11 08:49:26 UTC
it turns out the workaround fixes the problem in general, but in case you start pdk build in uplevel directory the error is seen again:

[root@labsystem-sat65 test]# pdk build --force ./check                                          │    
pdk (INFO): `pdk build` can only be run from inside a valid module with a metadata.json.

[root@labsystem-sat65 check]# pdk build --force .                                                                                                                                                                                                   
pdk (INFO): Building check-test version 0.0.1                                                                                                                      
pdk (INFO): Build of check-test has completed successfully. Built package can be found here: /root/test/check/pkg/check-test-0.0.1.tar.gz                                                                                                                                


with add chdir(path)

in /usr/lib/python2.7/site-packages/pulp_puppet/tools/puppet_module_builder.py (again)


Build puppet modules found during the search and publish                
 (copy) them to the user specified output directory.                     
                                                                         
 :param options: The command line options.                               
 :type options: optparse.Options                                         
 """                                                                     
 for path in find_modules():                                             
     chdir(path)                                                         
     shell('pdk build --force .')                                        
     chdir("..")                                                         
     pkg_dir = os.path.join(path, PKG_DIR)                               
     publish_module(pkg_dir, options.output_dir, options.force)

this problem is solved.

Comment 7 Steffen Froemer 2020-12-11 14:06:21 UTC
With following change, even the generic use-case is possible.


def build_puppet_modules(options):
    """
    Build puppet modules found during the search and publish
    (copy) them to the user specified output directory.

    :param options: The command line options.
    :type options: optparse.Options
    """
    for path in find_modules():
        basedir = os.getcwd()
        chdir(path)
        shell('pdk build --force .')
        chdir(basedir)
        pkg_dir = os.path.join(basedir, path, PKG_DIR)
        publish_module(pkg_dir, options.output_dir, options.force)



The test-cases are splitted into devel and master-branch

master = special use-case by customer
devel = generic use-case

[root@host repo]# git branch
* master
[root@host repo]# tree .
.
└── subdir
    ├── manifests
    │   ├── appserver.pp
    │   ├── base.pp
    │   ├── init.pp
    │   ├── params.pp
    │   └── servers.pp
    ├── metadata.json       <<<=== metadata.json at subdirectoy
    └── README.md

2 directories, 7 files
[root@host repo]# git checkout devel
Branch devel set up to track remote branch devel from origin.
Switched to a new branch 'devel'
[root@host repo]# tree .
.
├── subdir
│   ├── manifests
│   │   ├── appserver.pp
│   │   ├── base.pp
│   │   ├── init.pp
│   │   ├── params.pp
│   │   └── servers.pp
│   └── README.md
└── metadata.json       <<<====== metadata.json at toplevel 

2 directories, 7 files




/// Running the tests ///


[root@host ~]# /usr/bin/pulp-puppet-module-builder --output-dir=/modules --url=git@git.example.com:project/repo.git --branch=devel                                              
cd /root
git clone --recursive git@git.example.com:project/repo.git
cd repo
git status
git remote show -n origin
git fetch
git fetch --tags
git checkout devel
git pull
find . -name Modulefile -o -name metadata.json
cd .
pdk build --force .
cd /root/repo
mkdir -p /modules
cp /root/repo/./pkg/subdir-test-0.0.1.tar.gz /modules
cd /modules
cd /root/repo
cd /root


[root@host ~]# /usr/bin/pulp-puppet-module-builder --output-dir=/modules --url=git@git.example.com:project/repo.git --branch=master
cd /root
git clone --recursive git@git.example.com:project/repo.git
cd repo
git status
git remote show -n origin
git fetch
git fetch --tags
git checkout master
git pull
find . -name Modulefile -o -name metadata.json
cd ./subdir
pdk build --force .
cd /root/repo
mkdir -p /modules
cp /root/repo/./subdir/pkg/subdir-test-0.0.1.tar.gz /modules
cd /modules
cd /root/repo
cd /root

Comment 8 Steffen Froemer 2020-12-11 17:02:38 UTC
Created attachment 1738465 [details]
patch for comment#7

Comment 11 Mike McCune 2021-04-06 17:53:02 UTC
*** Bug 1869382 has been marked as a duplicate of this bug. ***

Comment 12 Mike McCune 2021-04-06 17:54:34 UTC
As noted in this BZ, the puppet version we ship in Satellite 6.8 and later does not support the 'puppet module build' command as well as this functionality being deprecated and removed in Satellite 7 coming up mid-2022.

If customers still rely on this functionaliy they can perform the workarounds outlined in this BZ and remain supported but we have no plans to resolve this in the Satellite 6 product line directly.

Comment 16 Mike McCune 2021-04-08 21:20:56 UTC
Created attachment 1770453 [details]
pulp-puppet-tools-2.21.3-3.HFRHBZ1886005.el7sat.noarch.rpm

Comment 17 Jason Holland 2021-04-15 17:24:53 UTC
The hotfix pulp-puppet-tools-2.21.3-3.HFRHBZ1886005.el7sat.noarch.rpm appears to be incomplete.  There is a line missing that changes back to the basedir.  Without that line, this fails to traverse the modules directory correctly.  Can you update the hotfix please?

--- puppet_module_builder.py.orig       2021-04-15 12:21:29.255299251 -0500
+++ puppet_module_builder.py    2021-04-15 12:21:37.116416970 -0500
@@ -277,6 +277,7 @@
         basedir = os.getcwd()
         chdir(path)
         shell('pdk build --force .')
+        chdir(basedir)
         pkg_dir = os.path.join(basedir, path, PKG_DIR)
         publish_module(pkg_dir, options.output_dir, options.force)

Thanks

Comment 19 Mike McCune 2021-04-30 15:31:47 UTC
*** Satellite 6.8.6 Hotfix Available ***

1) Download pulp-puppet-tools-2.21.3-4.HFRHBZ1886005.el7sat.noarch.rpm from this bugzilla to your Satellite

2) Download Puppetlabs tools package and install on your Satellite:

satellite-maintain packages unlock
sudo rpm -Uvh https://yum.puppet.com/puppet-tools-release-el-7.noarch.rpm
yum -y install pdk
satellite-maintain packages lock

3) Install hotfix:

rpm -Uvh pulp-puppet-tools-2.21.3-4.HFRHBZ1886005.el7sat.noarch.rpm 

4) resume operations

Comment 20 Mike McCune 2021-04-30 15:32:25 UTC
Created attachment 1777835 [details]
pulp-puppet-tools-2.21.3-4.HFRHBZ1886005.el7sat.noarch.rpm

Comment 21 Mike McCune 2021-04-30 15:32:52 UTC
Jason, apologies for the missed line, see the updated RPM attached.

Comment 22 Jason Holland 2021-04-30 15:39:04 UTC
Thank you very much Mike, the updated RPM appears to fix that issue.  Tested and everything works as expected.

Comment 24 Lai 2021-05-11 19:52:46 UTC
Steps to test.

1. Apply patch from command 19, but on step 2 use `foremain-maintain packages install pdk`.
2. Run pulp-puppet-module-builder
3. Check output for usage of pdk from 2.
4. (optional), follow commands from 2 and run `pdk build --force .` and check result.

Expected:
3) Should see that `pdk build --force .` is used in output.
4) Should see that the packages are built.

Actual:
3) # pulp-puppet-module-builder 
cd /root
git status
find . -name Modulefile -o -name metadata.json
cd ./puppet-access_insights_client
pdk build --force .
cd /root
mkdir -p /root
Skipping lphiri-access_insights_client-1.0.1.tar.gz as the file exists
cd /root
cd /root
cd /root

4) # pdk build --force .
pdk (INFO): Building lphiri-access_insights_client version 1.0.1
pdk (INFO): Build of lphiri-access_insights_client has completed successfully. Built package can be found here: /root/puppet-access_insights_client/pkg/lphiri-access_insights_client-1.0.1.tar.gz

Verified on 6.9.2 snap 2
python-pulp-puppet-common-2.21.5.1-1.el7sat.noarch
pulp-puppet-plugins-2.21.5.1-1.el7sat.noarc

Comment 29 errata-xmlrpc 2021-05-20 18:05:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite 6.9.2 Async Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2074


Note You need to log in before you can comment on or make changes to this bug.