Description of problem: Following http://www.katello.org/docs/2.1/user_guide/puppet_integration/index.html to generate a PULP_MANIFEST style repo and sync it works fine unless you use a file:// url to perform the sync. If you do, the sync errors with: "FileRetrievalException: [Errno 2] No such file or directory: u'///var/www/html/puppetsync/modules.json'", Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task R = retval = fun(*args, **kwargs) File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 328, in __call__ return super(Task, self).__call__(*args, **kwargs) File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__ return self.run(*args, **kwargs) File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/sync.py", line 114, in sync raise PulpExecutionException(_('Importer indicated a failed response')) Version-Release number of selected component (if applicable): pulp-server-2.6.0.1-1.beta.1.el6_6sat.noarch How reproducible: Always Steps to Reproduce: 1. Use pulp-puppet-module-builder to generate a PULP_MANIFEST style puppet repo 2. Attempt to sync the repo using a file:// url Actual results: Failure Expected results: Success Additional info:
needinfo'ing the pulp team, as this seems to be a pulp issue
The pulp docs recommend a whopping 3 slashes, as in file:///home/me/mystuff/ https://pulp-puppet.readthedocs.org/en/latest/user-guide/recipes.html#building-and-importing-modules Did you try that?
Michael, yes, file urls use 3 slashes, two for the protocol, one for the actual path, notice in the log: No such file or directory: u'///var/www/html/puppetsync/modules.json'", even shows the 3 slashes. The actual url i used was: file:///var/www/html/puppetsync/ -Justin
Jeff, I think you have knowledge of this feature. Can you take a look and file an upstream bug if necessary?
Justin, Can you attach a tarball containing the contents of your: /var/www/html/puppetsync/ directory?
I don't have that data any more, but it is trivial to reproduce. If you have any trouble or really need the data I used let me know
I'm running into this issue as well and it is still valid with the upstream Katello 2.2. It's easy to reproduce. Just use pulp-puppet-module-builder, add a product+repo for it in Katello and perform a sync. The sync job will always fail with the message that a file named modules.json could not be found in the directory generated by pulp-puppet-module-builder. The directory generated by pulp-puppet-module-builder contains several tarballs (one for each Puppet module) and a file named 'PULP_MANIFEST'. There's no file named 'modules.json'. Full job details can be found at http://fpaste.org/241285/36362367/ The most interesting pieces of this job details are: progress_report: puppet_importer: <<snip>> metadata: query_finished_count: 0 error_message: Error downloading metadata execution_time: 0 query_total_count: 1 traceback: - - /usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/forge.py - 124 - _parse_metadata - metadata_json_docs = downloader.retrieve_metadata(self.progress_report) - - /usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/downloaders/local.py - 70 - retrieve_metadata - raise FileRetrievalException(report.error_msg) state: failed error: ! 'FileRetrievalException: [Errno 2] No such file or directory: u''//modules/modules.json''' current_query: modules/modules.json
I verified that this functionality works correctly with upstream pulp 2.6.4. Can you attach log output that was generated during the attempted sync? I suspect a filesystem permission or selinux issue is preventing pulp from accessing PULP_MANIFEST. In the log, there may be a statement such as: ERROR: Fetch URL: <your url>/PULP_MANIFEST failed: [Errno 13] Permission denied: ... The importer can sync either from a PULP_MANIFEST style repo, or a forge style. If the former fails, it tries the latter, which is why you're seeing the error about modules.json. Look a little further back in the log, and hopefully you'll see an error explaining why PULP_MANIFEST is not accessible.
I'm actually no longer able to reproduce this, it seems like it was fixed in Satellite 6.1 sometime before 6.1.5. Leo, I tested with your customer's modules as well and it seemed to work fine. Is the customer still seeing this on a fully updated 6.1.5?
Hello Justin, That case got closed after we provided him the workaround. So I am not sure whether the issue still persist or not with the latest version. However, at that time with the customer's module we were able to reproduce the issue. If it can't be reproduced with the latest Satellite version the issue might have fixed.
Verified with sat6.2 beta snap8.1 I copied the modules on filesystem under /modules and changed the dir permissions to '755' and synced the modules by setting url file:/// and I'm able to reproduce the issue: ==> /var/log/messages <== Apr 14 08:30:35 cloud-qe-3 pulp: pulp_puppet.plugins.importers.directory:ERROR: Fetch URL: file:///modules/PULP_MANIFEST failed: [Errno 13] Permission denied: u'///modules/PULP_MANIFEST' Apr 14 08:30:35 cloud-qe-3 pulp: pulp_puppet.plugins.importers.forge:INFO: Beginning sync for repository <Default_Organization-puppet-puppetgit> Apr 14 08:30:35 cloud-qe-3 pulp: pulp_puppet.plugins.importers.forge:INFO: Beginning metadata retrieval for repository <Default_Organization-puppet-puppetgit> Apr 14 08:30:35 cloud-qe-3 pulp: pulp_puppet.plugins.importers.forge:ERROR: (7753-16960) Exception while retrieving metadata for repository <Default_Organization-puppet-puppetgit> Apr 14 08:30:35 cloud-qe-3 pulp: pulp_puppet.plugins.importers.forge:ERROR: (7753-16960) Traceback (most recent call last): Apr 14 08:30:35 cloud-qe-3 pulp: pulp_puppet.plugins.importers.forge:ERROR: (7753-16960) File "/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/forge.py", line 113, in _parse_metadata Apr 14 08:30:35 cloud-qe-3 pulp: pulp_puppet.plugins.importers.forge:ERROR: (7753-16960) metadata_json_docs = downloader.retrieve_metadata(self.progress_report) Apr 14 08:30:35 cloud-qe-3 pulp: pulp_puppet.plugins.importers.forge:ERROR: (7753-16960) File "/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/downloaders/local.py", line 58, in retrieve_metadata Apr 14 08:30:35 cloud-qe-3 pulp: pulp_puppet.plugins.importers.forge:ERROR: (7753-16960) raise FileRetrievalException(report.error_msg) Apr 14 08:30:35 cloud-qe-3 pulp: pulp_puppet.plugins.importers.forge:ERROR: (7753-16960) FileRetrievalException: FileRetrievalException: [Errno 2] No such file or directory: u'///modules/modules.json' Looks like selinux issue: type=AVC msg=audit(1460637035.939:5311): avc: denied { read } for pid=7753 comm="celery" name="PULP_MANIFEST" dev="dm-0" ino=162121353 scontext=system_u:system_r:celery_t:s0 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file type=SYSCALL msg=audit(1460637035.939:5311): arch=c000003e syscall=2 success=no exit=-13 a0=3977ee0 a1=0 a2=1b6 a3=24 items=0 ppid=7498 pid=7753 auid=4294967295 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=4294967295 comm="celery" exe="/usr/bin/python2.7" subj=system_u:system_r:celery_t:s0 key=(null)
Created attachment 1147231 [details] complete logs with exception
Created redmine issue http://projects.theforeman.org/issues/15812 from this bug
Opened an issue against pulp since I was able to reproduce with just pulp-admin: https://pulp.plan.io/issues/2167
We need to update the docs text. See: https://pulp.plan.io/issues/1560
Upstream bug assigned to daviddavis
Upstream bug component is Repositories
The Pulp upstream bug status is at CLOSED - NOTABUG. Updating the external tracker on this bug.
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
We'll need some documentation around how to set SELinux labels on files. Here's an example: "If you have SELinux enabled, in order to sync from the file system, you'll need to apply a label to the files in order for Satellite 6 to access them. Two options are httpd_sys_r_content_t or pulp_tmp_t. Note: if you choose httpd_sys_r_content_t then the webserver can also read the files so that may or may not be good. One way to apply these labels would be to use chcon. See https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Security-Enhanced_Linux/sect-Security-Enhanced_Linux-Working_with_SELinux-SELinux_Contexts_Labeling_Files.html for more info." This applies to all content types (not just puppet).
Moving to 'NEW' while assigned to the default assignee.
Hello As per Bug 1301367 - pulp-puppet-module-builder and SELinux the guide was updated to use /var/www/puppet-modules See "Procedure 3.3. Publishing a Git Repository to a Local Directory" [1] [1] https://access.redhat.com/documentation/en/red-hat-satellite/6.2/paged/puppet-guide/35-adding-puppet-modules-from-a-git-repository
Hello I grep'd al guides and the use of "pulp_tmp_t" and "httpd_sys_r_content_t" are not documented. So I will add them to the doc mentioned in comment 37.
Assigning to Stephen for review.
Hello These changes are now live on the customer portal. https://access.redhat.com/documentation/en/red-hat-satellite/6.2/paged/puppet-guide/35-adding-puppet-modules-from-a-git-repository#proc-Red_Hat_Satellite-Puppet_Guide-Adding_Puppet_Modules_from_a_Git_Repository-Publishing_a_Git_Repository_to_a_Local_Directory Thank you