We are getting Invalid properties / MissingModuleFile when Pulp is extracting some PuppetForge files. We extracted these and it looks like they have been compressed on MacOS (there are some extra files and folders) which might confuse Pulp. # Extract the entire module try: tgz = tarfile.open(name=filename) tgz.extractall(path=extraction_dir) tgz.close() except Exception: raise InvalidTarball(filename), None, sys.exc_info()[2] try: # Attempt to find the metadata in the Puppet module's main directory # It is expected the .tar.gz file will contain exactly one Puppet module try: module_dir = os.listdir(extraction_dir)[0] except IndexError: raise MissingModuleFile(filename) metadata_filename = constants.MODULE_METADATA_FILENAME metadata_full_path = os.path.join(extraction_dir, module_dir, metadata_filename) if not os.path.isfile(metadata_full_path): raise MissingModuleFile(filename) return _read_contents(metadata_full_path) finally: # Delete the entire extraction directory shutil.rmtree(extraction_dir)
Steps to reproduce, install Satellite 6.2, sync Puppet Forge. I can get you repro box in labs if you want.
The Pulp upstream bug status is at CLOSED - WORKSFORME. Updating the external tracker on this bug.
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
It is indeed the MacOS files. I looked at the extracted file lists for these modules and they're all of the format: -rwxr-xr-x 1 501 user 311 Mar 27 2015 ._BillWeiss-spamassassin-1.0.5 drwxr-xr-x 7 501 user 4.0K Mar 27 2015 BillWeiss-spamassassin-1.0.5 The Pulp code is simply picking the first entry. We could change it to be smarter and pick the first non-file entry. Also worth noting is that I couldn't reproduce this in my pulp dev environment. I had to use a Satellite 6.2 box for whatever reason. I'm guessing maybe the ordering of the files is nondeterministic.
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Noticed this when uploading puppetlabs-denyhosts-0.1.0.tar.gz from Puppet Forge - when I deleted all files starting with "." from archive, I was able to import (after skip & resume on the task which failed). And I agree this is most probably not reproducible, I'm pretty sure I have imported exactly this file some time ago into different Satellite.
Forgot to mention I'm on pulp-server-2.8.7.14-1.el7sat.noarch
Requesting needsinfo from upstream developer daviddavis because the 'FailedQA' flag is set.
In case 01998333 version of pulp-server is pulp-server-2.8.7.17-1.el7sat.noarch.
Here's the failing puppet modules and why they are failing: jowy-druid-0.2.1.tar.gz - files are not contained in a <MODULE NAME> dir rbansal-svn-1.6.11-7.el6.x86_64.tar.gz - I get a 400 error when I try to download this maxadamo-glusterfs-0.0.5.tar.gz - has checksums in a field called 'checksum' instead of 'checksums' keeleysam-vmwaretools-2014.04.22.tar.gz - 400 error when I try to download this felixcarmona-test-1.0.0.tar.gz - files are not contained in a <MODULE NAME> dir puppetlabs-strings-999.999.999.tar.gz - files are not contained in a <MODULE NAME> dir For the puppet modules that report a 400 error, it looks like their filenames are invalid and Puppet Forge won't serve them (e.g. https://forge.puppet.com/v3/files/rbansal-svn-1.6.11-7.el6.x86_64.tar.gz). Note that it looks like the sync does import the valid puppet modules. I think that reporting an error is the correct behavior here since Puppet Forge is serving invalid puppet modules. Will double check with the other Pulp puppet developers.
I talked with the Pulp puppet team and they said that this is the intended behavior. Syncs from Puppet Forge are going to return errors as it is serving invalid puppet modules. Setting back to ON_QA.
Is this really a material for QE? Was there some actual change? Or should it be CLOSED(WONTFIX)?
Yes, the original bug was that Pulp did not properly parse puppet modules if there were extraneous files present in the puppet module. I made a change in Pulp/Satellite that fixed that. This change needs to be verified. Looking at the result from Perry Gagne's test run, the fix was successful.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1126