Bug 1445625 - Pulp throws MissingModuleFile for some Puppet Forge files
Summary: Pulp throws MissingModuleFile for some Puppet Forge files
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Pulp
Version: 6.2.8
Hardware: Unspecified
OS: Unspecified
medium
medium vote
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Perry Gagne
URL: https://pulp.plan.io/issues/1846
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-04-26 07:31 UTC by Lukas Zapletal
Modified: 2019-06-13 21:26 UTC (History)
16 users (show)

Fixed In Version: pulp-2.13.1
Doc Type: Known Issue
Doc Text:
*Known Issue* On Puppet Forge, some Puppet modules are invalid and cannot sync with Satellite. These invalid Puppet modules cause error messages such as `Invalid properties` or `MissingModulePile`. Despite receiving a report of a sync failure, the valid Puppet modules sync from Puppet Forge into Satellite.
Clone Of:
Environment:
Last Closed: 2018-04-13 13:29:48 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:1126 None None None 2018-04-13 13:31:33 UTC
Pulp Redmine 1846 High CLOSED - CURRENTRELEASE pulp does not sync puppetforge correctly 2017-05-24 18:37:25 UTC

Description Lukas Zapletal 2017-04-26 07:31:51 UTC
We are getting Invalid properties / MissingModuleFile when Pulp is extracting some PuppetForge files. We extracted these and it looks like they have been compressed on MacOS (there are some extra files and folders) which might confuse Pulp.

    # Extract the entire module
    try:
        tgz = tarfile.open(name=filename)
        tgz.extractall(path=extraction_dir)
        tgz.close()
    except Exception:
        raise InvalidTarball(filename), None, sys.exc_info()[2]

    try:
        # Attempt to find the metadata in the Puppet module's main directory
        # It is expected the .tar.gz file will contain exactly one Puppet module
        try:
            module_dir = os.listdir(extraction_dir)[0]
        except IndexError:
            raise MissingModuleFile(filename)
        metadata_filename = constants.MODULE_METADATA_FILENAME
        metadata_full_path = os.path.join(extraction_dir, module_dir, metadata_filename)
        if not os.path.isfile(metadata_full_path):
            raise MissingModuleFile(filename)

        return _read_contents(metadata_full_path)
    finally:
        # Delete the entire extraction directory
        shutil.rmtree(extraction_dir)

Comment 1 Lukas Zapletal 2017-04-26 07:33:37 UTC
Steps to reproduce, install Satellite 6.2, sync Puppet Forge.

I can get you repro box in labs if you want.

Comment 4 pulp-infra@redhat.com 2017-04-26 12:05:50 UTC
The Pulp upstream bug status is at CLOSED - WORKSFORME. Updating the external tracker on this bug.

Comment 5 pulp-infra@redhat.com 2017-04-26 12:05:53 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 6 David Davis 2017-04-26 16:53:44 UTC
It is indeed the MacOS files. I looked at the extracted file lists for these modules and they're all of the format:

-rwxr-xr-x   1  501 user  311 Mar 27  2015 ._BillWeiss-spamassassin-1.0.5
drwxr-xr-x   7  501 user 4.0K Mar 27  2015 BillWeiss-spamassassin-1.0.5

The Pulp code is simply picking the first entry. We could change it to be smarter and pick the first non-file entry.

Also worth noting is that I couldn't reproduce this in my pulp dev environment. I had to use a Satellite 6.2 box for whatever reason. I'm guessing maybe the ordering of the files is nondeterministic.

Comment 7 pulp-infra@redhat.com 2017-04-26 17:05:35 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 8 pulp-infra@redhat.com 2017-04-26 17:35:52 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 9 pulp-infra@redhat.com 2017-04-27 15:06:11 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 10 pulp-infra@redhat.com 2017-04-27 22:35:41 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 11 pulp-infra@redhat.com 2017-04-27 23:05:32 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 12 pulp-infra@redhat.com 2017-05-17 13:37:19 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 14 pulp-infra@redhat.com 2017-05-24 18:37:26 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 17 Jan Hutař 2017-09-16 19:09:30 UTC
Noticed this when uploading puppetlabs-denyhosts-0.1.0.tar.gz from Puppet Forge - when I deleted all files starting with "." from archive, I was able to import (after skip & resume on the task which failed). And I agree this is most probably not reproducible, I'm pretty sure I have imported exactly this file some time ago into different Satellite.

Comment 18 Jan Hutař 2017-09-16 19:10:36 UTC
Forgot to mention I'm on pulp-server-2.8.7.14-1.el7sat.noarch

Comment 22 pulp-infra@redhat.com 2018-01-29 20:05:26 UTC
Requesting needsinfo from upstream developer daviddavis@redhat.com because the 'FailedQA' flag is set.

Comment 23 jjansky 2018-01-30 13:26:29 UTC
In case 01998333 version of pulp-server is pulp-server-2.8.7.17-1.el7sat.noarch.

Comment 25 pulp-infra@redhat.com 2018-02-05 23:06:53 UTC
Requesting needsinfo from upstream developer daviddavis@redhat.com because the 'FailedQA' flag is set.

Comment 26 David Davis 2018-02-05 23:08:53 UTC
Here's the failing puppet modules and why they are failing:

jowy-druid-0.2.1.tar.gz - files are not contained in a <MODULE NAME> dir
rbansal-svn-1.6.11-7.el6.x86_64.tar.gz - I get a 400 error when I try to download this
maxadamo-glusterfs-0.0.5.tar.gz - has checksums in a field called 'checksum' instead of 'checksums'
keeleysam-vmwaretools-2014.04.22.tar.gz - 400 error when I try to download this
felixcarmona-test-1.0.0.tar.gz - files are not contained in a <MODULE NAME> dir
puppetlabs-strings-999.999.999.tar.gz - files are not contained in a <MODULE NAME> dir

For the puppet modules that report a 400 error, it looks like their filenames are invalid and Puppet Forge won't serve them (e.g. https://forge.puppet.com/v3/files/rbansal-svn-1.6.11-7.el6.x86_64.tar.gz).

Note that it looks like the sync does import the valid puppet modules. I think that reporting an error is the correct behavior here since Puppet Forge is serving invalid puppet modules. Will double check with the other Pulp puppet developers.

Comment 27 David Davis 2018-02-07 20:47:44 UTC
I talked with the Pulp puppet team and they said that this is the intended behavior. Syncs from Puppet Forge are going to return errors as it is serving invalid puppet modules. Setting back to ON_QA.

Comment 28 Jan Hutař 2018-02-08 07:02:56 UTC
Is this really a material for QE? Was there some actual change?

Or should it be CLOSED(WONTFIX)?

Comment 29 David Davis 2018-02-08 22:20:31 UTC
Yes, the original bug was that Pulp did not properly parse puppet modules if there were extraneous files present in the puppet module. I made a change in Pulp/Satellite that fixed that. This change needs to be verified. Looking at the result from Perry Gagne's test run, the fix was successful.

Comment 33 errata-xmlrpc 2018-04-13 13:29:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1126


Note You need to log in before you can comment on or make changes to this bug.