Red Hat Bugzilla – Bug 1247364
pulp-puppet-module-builder DELETEs working_dir (and is severely broken in other ways...)
Last modified: 2018-09-19 11:19:07 EDT
Created attachment 1056740 [details] Patch for /usr/lib/python2.7/site-packages/pulp_puppet/tools/puppet_module_builder.py Description of problem: * malformed URLs can even make pulp-puppet-module-builder _DELETE_ the whole working_dir (dont try --url=/srv/git/puppet-modules.git/ or anything ending with a slash) * it does not work if a git repo contains more than one puppet module * it does not work for URLs with relative path names like --url=git@git.example.com:puppet-modules.git * --clean option does not work for any repo where the remote puppet-modules.git is cloned to a local puppet-modules/ Version-Release number of selected component (if applicable): pulp-puppet-tools-2.6.0.14-1 python-pulp-puppet-common-2.6.0.14-1 How reproducible: Just try anything of the above. Actual results: Nothing useful at best or in case of malformed URL complete loss of data in working_dir
Let's discuss each issue separately: - The behavior of --clean when the URL has a trailing '/' is definitely a bug and needs to be fixed. - The issue of: "it does not work if a git repo contains more than one puppet module". Can you include the URL for a git repository containing more than one module? Tested with: git@github.com:puppetlabs/puppetlabs-strings.git/ and it built (2) modules: username-test-0.0.1.tar.gz, puppetlabs-strings-0.2.0.tar.gz. - The issue of: it "does not work for URLs with relative path names like --url=git@git.example.com:puppet-modules.git". Can be more specific about what does not work? The issue of: "--clean option does not work for any repo where the remote puppet-modules.git is cloned to a local puppet-modules". is definitely a bug and needs to be fixed.
The puppetlabs-strings case described in Comment 2 contains only one puppet module and the according test example. The latter is most likely not what we want to manage with Satellite, do we? I would even say creating the test module for upload is a bug! Even with the example from Comment 2 the --clean option does not work. Just try 'pulp-puppet-module-builder --url=https://github.com/puppetlabs/puppetlabs-strings.git --clean' The puppetlabs-strings dir remains and subsequent calls fail with message 'fatal: destination path 'puppetlabs-strings' already exists and is not an empty directory.' Use https://github.com/shetze/puppet-modules.git for testing. This repo contains two puppet modules. Expected behaviour is that both of them get built. (This is at least how I understand "Search the working directory and build all puppet modules found.")
Jeff, can you get upstream bug reports filed and linked?
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
Hello, I've noticed that this patch is set for release in z stream for more than a month. has this been added to 6.1.4 or is it scheduled to go into 6.1.5 or 6.1.6?
@jnikolak Hello, I've tried to build more than one puppet module in release 6.1.6 but with no success. pulp-puppet-tools-2.6.0.19-1.el7sat.noarch
I fixed the --clean behavior but still not able to reproduce any issues related to finding and building the puppet modules. Ran using the suggested git repository and it seemed to find all of the modules. Am I missing something? ----------------------------- [jortel@f23d ~]$ pulp-puppet-module-builder -w /tmp/puppet/ -cu https://github.com/shetze/puppet-modules.git/ rm -rf /tmp/puppet/puppet-modules cd /tmp/puppet/ git clone --recursive https://github.com/shetze/puppet-modules.git cd puppet-modules git status git remote show -n origin git fetch git fetch --tags find . -name Modulefile -o -name metadata.json puppet module build ./module-git mkdir -p /tmp/puppet/ cp ./module-git/pkg/LunetIX-git-0.6.5.tar.gz /tmp/puppet/ puppet module build ./profile-buildhost mkdir -p /tmp/puppet/ cp ./profile-buildhost/pkg/LunetIX-buildhost-0.4.5.tar.gz /tmp/puppet/ puppet module build ./profile-dockerhost mkdir -p /tmp/puppet/ cp ./profile-dockerhost/pkg/LunetIX-dockerhost-0.3.0.tar.gz /tmp/puppet/ cp ./profile-dockerhost/pkg/LunetIX-dockerhost-0.1.0.tar.gz /tmp/puppet/ cd /tmp/puppet/ cd /tmp/puppet/puppet-modules cd /home/jortel rm -rf /tmp/puppet/puppet-modules [jortel@f23d ~]$ ll /tmp/puppet total 48 -rw-rw-r-- 1 jortel jortel 27334 Mar 7 12:36 LunetIX-buildhost-0.4.5.tar.gz -rw-rw-r-- 1 jortel jortel 3799 Mar 7 12:36 LunetIX-dockerhost-0.1.0.tar.gz -rw-rw-r-- 1 jortel jortel 3757 Mar 7 12:36 LunetIX-dockerhost-0.3.0.tar.gz -rw-rw-r-- 1 jortel jortel 5931 Mar 7 12:36 LunetIX-git-0.6.5.tar.gz -rw-rw-r-- 1 jortel jortel 401 Mar 7 12:36 PULP_MANIFEST [jortel@f23d ~]$ cat /tmp/puppet/PULP_MANIFEST LunetIX-dockerhost-0.1.0.tar.gz,f66dede1e6be15bbc9b6bbe6397b75d24fd1b7c931a66276ed8959bee7621716,3799 LunetIX-dockerhost-0.3.0.tar.gz,5650f8e3a4e87c116445275b53d216e54494f41123600208e8edf08e84f75316,3757 LunetIX-buildhost-0.4.5.tar.gz,7d0a22531211c5fe8c49a6a26f92a548e67fdb9dd2940800e5f11fbd1dc0784a,27334 LunetIX-git-0.6.5.tar.gz,bc933e9e7449267fd1df8c719fd5235719dc58ba08d17a2ad9a52e6070c1b958,5931
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
I've attached the reproducer below, this was working on older version. He was given a patch to fix the issue. So everytime we upgrade a satellite version, 6.1.4, 6.1.5,6.1.6 He has to manually apply the patch. >>>>>>>>> Start Reproducer >>>>>>>>>>>>>>>>>>>>>>> # pulp-puppet-module-builder -u essgit@essgit.qut.edu.au:/opt/git/puppet -c -w /opt/puppet/ -o /opt/puppet/modules/ rm -rf /opt/puppet/puppet cd /opt/puppet/ git clone --recursive essgit@essgit.qut.edu.au:/opt/git/puppet cd puppet git status git remote show -n origin git fetch git fetch --tags find . -name init.pp puppet module build ./QUT-qutsh mkdir -p /opt/puppet/modules/ cp ./QUT-qutsh/pkg/QUT-qutsh-0.2.0.tar.gz /opt/puppet/modules/ puppet module build ./QUT-test mkdir -p /opt/puppet/modules/ cp ./QUT-test/pkg/QUT-test-0.1.0.tar.gz /opt/puppet/modules/ puppet module build ./QUT-sssd mkdir -p /opt/puppet/modules/ cp ./QUT-sssd/pkg/QUT-sssd-0.1.6.tar.gz /opt/puppet/modules/ cd /opt/puppet/modules/ cd /opt/puppet/puppet cd /opt rm -rf /opt/puppet/puppet [root@satellite02-poc opt]# ls -l /opt/puppet/modules/ total 24 -rw-r--r--. 1 root root 277 Aug 19 13:30 PULP_MANIFEST -rw-r--r--. 1 root root 7556 Aug 19 13:30 QUT-qutsh-0.2.0.tar.gz -rw-r--r--. 1 root root 6026 Aug 19 13:30 QUT-sssd-0.1.6.tar.gz -rw-r--r--. 1 root root 3467 Aug 19 13:30 QUT-test-0.1.0.tar.gz <<<<<<<<<<<<< End Reproducer <<<<<<<<<<<<<<<<<<<<<<< Please see his comments below Hi Jon, Nothing has changed so it's still not fixed. The problem is still the same - the script is looking for a puppet module root in the root of the git repo and not looking into any sub folders which is where our module are (see the attachments for the directory layout). The patch in the bugzilla fixes it and it did work in 6.0, I don't know why it's been so hard to reproduce and fix (I just read the comments on the bz). This is quite frustrating as whenever a new release of pulp-puppet-tools comes out it breaks the script again and I have to go back and fix it each time (with the patch Redhat supplied!). Regards Greg
(In reply to Jeff Ortel from comment #15) > I fixed the --clean behavior but still not able to reproduce any issues > related to finding and building the puppet modules. Ran using the suggested > git repository and it seemed to find all of the modules. Am I missing > something? Actually, I have no idea what you are testing. I have: yum info pulp-puppet-tools-2.6.0.20-1.el7sat.noarch Name : pulp-puppet-tools Arch : noarch Version : 2.6.0.20 Release : 1.el7sat Size : 43 k Repo : installed From repo : rhel-7-server-satellite-6.1-rpms Summary : Pulp puppet tools URL : https://fedorahosted.org/pulp/ License : GPLv2 Description : A collection of tools used to manage puppet modules. With this version I get: >>>>>>>>> Start Reproducer >>>>>>>>>>>>>>>>>>>>>>> pulp-puppet-module-builder -w /tmp/puppet/ -cu https://github.com/shetze/puppet-modules.git/ rm -rf /tmp/puppet/ cd /tmp/puppet/ Traceback (most recent call last): File "/usr/bin/pulp-puppet-module-builder", line 9, in <module> load_entry_point('pulp-puppet-tools==2.6.0', 'console_scripts', 'pulp-puppet-module-builder')() File "/usr/lib/python2.7/site-packages/pulp_puppet/tools/puppet_module_builder.py", line 332, in main chdir(options.working_dir) File "/usr/lib/python2.7/site-packages/pulp_puppet/tools/puppet_module_builder.py", line 80, in chdir os.chdir(path) OSError: [Errno 2] No such file or directory: '/tmp/puppet/' <<<<<<<<<<<<< End Reproducer <<<<<<<<<<<<<<<<<<<<<<< OK, so you probably tried your fixed version that now understands how to deal with the clean option in that case. I don't have that, so I can not tell if that works with the "more than one puppet module issue" or the "relative pathname issue". If I use the version I have and we ship to our customers (and I avoid the broken clean option), I get: >>>>>>>>> Start Reproducer >>>>>>>>>>>>>>>>>>>>>>> pulp-puppet-module-builder -w /tmp/puppet/ -u https://github.com/shetze/puppet-modules.git/ cd /tmp/puppet/ git clone --recursive https://github.com/shetze/puppet-modules.git/ git status find . -name Modulefile find . -name metadata.json puppet module build Error: Unable to find module root at /tmp/puppet or parent directories Error: Try 'puppet help module build' for usage <<<<<<<<< End Reproducer <<<<<<<<<<<<<<<<<<<<<<<<< Maybe you can provide me your version of the puppet_module_builder.py so I can advise you what else need to be changed.
Just one additional comment: The LunetIX-git module from the example above manages a git server that uses a post-receive hook not only to create the puppet modules on the fly, but also syncs them into Satellite, publishes the CV and promotes to an environment based on the branch you push your commit into. This is CI for a Puppet development workflow without need of Jenkins and alike. https://github.com/shetze/puppet-modules/blob/master/module-git/templates/puppet-post-receive.erb This hook does not use pulp-puppet-module-builder and works today ;-)
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Customer confirmed that 6.2 has resolved this issue.
Verified in Satellite 6.2 Snap 19.0. * Directory with multiple puppet modules builds all of them * `pulp-puppet-module-builder --url=https://github.com/puppetlabs/puppetlabs-strings.git --clean` successfully cleans up the git checkouts * Multiple modules are imported from pulp-puppet-module-builder --url=https://github.com/shetze/puppet-modules.git * Malformed URL's don't delete the working directory
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1501