Bug 1215731
Summary: | Syncing a PULP_MANIFEST puppet repo over file:// fails with No such file or directory: u'///dir/modules.json' | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Satellite | Reporter: | Justin Sherrill <jsherril> | ||||
Component: | Docs Puppet Guide | Assignee: | Stephen Wadeley <swadeley> | ||||
Status: | CLOSED CURRENTRELEASE | QA Contact: | Lucie Jirakova <ljirakov> | ||||
Severity: | medium | Docs Contact: | |||||
Priority: | high | ||||||
Version: | 6.0.8 | CC: | adahms, bbuckingham, bkearney, bmbouter, cwelton, daviddavis, dkliban, egolov, erik-fedora, ggainey, ipanova, jortel, jsherril, kshirsal, mhrivnak, mmccune, nshaik, pcfe, pcreech, pmutha, rchan, sghai, snag, swadeley, ttereshc | ||||
Target Milestone: | Unspecified | Keywords: | Triaged | ||||
Target Release: | Unused | ||||||
Hardware: | All | ||||||
OS: | Linux | ||||||
Whiteboard: | |||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2016-11-18 08:06:08 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Attachments: |
|
Description
Justin Sherrill
2015-04-27 14:58:48 UTC
needinfo'ing the pulp team, as this seems to be a pulp issue The pulp docs recommend a whopping 3 slashes, as in file:///home/me/mystuff/ https://pulp-puppet.readthedocs.org/en/latest/user-guide/recipes.html#building-and-importing-modules Did you try that? Michael, yes, file urls use 3 slashes, two for the protocol, one for the actual path, notice in the log: No such file or directory: u'///var/www/html/puppetsync/modules.json'", even shows the 3 slashes. The actual url i used was: file:///var/www/html/puppetsync/ -Justin Jeff, I think you have knowledge of this feature. Can you take a look and file an upstream bug if necessary? Justin, Can you attach a tarball containing the contents of your: /var/www/html/puppetsync/ directory? I don't have that data any more, but it is trivial to reproduce. If you have any trouble or really need the data I used let me know I'm running into this issue as well and it is still valid with the upstream Katello 2.2. It's easy to reproduce. Just use pulp-puppet-module-builder, add a product+repo for it in Katello and perform a sync. The sync job will always fail with the message that a file named modules.json could not be found in the directory generated by pulp-puppet-module-builder. The directory generated by pulp-puppet-module-builder contains several tarballs (one for each Puppet module) and a file named 'PULP_MANIFEST'. There's no file named 'modules.json'. Full job details can be found at http://fpaste.org/241285/36362367/ The most interesting pieces of this job details are: progress_report: puppet_importer: <<snip>> metadata: query_finished_count: 0 error_message: Error downloading metadata execution_time: 0 query_total_count: 1 traceback: - - /usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/forge.py - 124 - _parse_metadata - metadata_json_docs = downloader.retrieve_metadata(self.progress_report) - - /usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/downloaders/local.py - 70 - retrieve_metadata - raise FileRetrievalException(report.error_msg) state: failed error: ! 'FileRetrievalException: [Errno 2] No such file or directory: u''//modules/modules.json''' current_query: modules/modules.json I verified that this functionality works correctly with upstream pulp 2.6.4. Can you attach log output that was generated during the attempted sync? I suspect a filesystem permission or selinux issue is preventing pulp from accessing PULP_MANIFEST. In the log, there may be a statement such as: ERROR: Fetch URL: <your url>/PULP_MANIFEST failed: [Errno 13] Permission denied: ... The importer can sync either from a PULP_MANIFEST style repo, or a forge style. If the former fails, it tries the latter, which is why you're seeing the error about modules.json. Look a little further back in the log, and hopefully you'll see an error explaining why PULP_MANIFEST is not accessible. I'm actually no longer able to reproduce this, it seems like it was fixed in Satellite 6.1 sometime before 6.1.5. Leo, I tested with your customer's modules as well and it seemed to work fine. Is the customer still seeing this on a fully updated 6.1.5? Hello Justin, That case got closed after we provided him the workaround. So I am not sure whether the issue still persist or not with the latest version. However, at that time with the customer's module we were able to reproduce the issue. If it can't be reproduced with the latest Satellite version the issue might have fixed. Verified with sat6.2 beta snap8.1 I copied the modules on filesystem under /modules and changed the dir permissions to '755' and synced the modules by setting url file:/// and I'm able to reproduce the issue: ==> /var/log/messages <== Apr 14 08:30:35 cloud-qe-3 pulp: pulp_puppet.plugins.importers.directory:ERROR: Fetch URL: file:///modules/PULP_MANIFEST failed: [Errno 13] Permission denied: u'///modules/PULP_MANIFEST' Apr 14 08:30:35 cloud-qe-3 pulp: pulp_puppet.plugins.importers.forge:INFO: Beginning sync for repository <Default_Organization-puppet-puppetgit> Apr 14 08:30:35 cloud-qe-3 pulp: pulp_puppet.plugins.importers.forge:INFO: Beginning metadata retrieval for repository <Default_Organization-puppet-puppetgit> Apr 14 08:30:35 cloud-qe-3 pulp: pulp_puppet.plugins.importers.forge:ERROR: (7753-16960) Exception while retrieving metadata for repository <Default_Organization-puppet-puppetgit> Apr 14 08:30:35 cloud-qe-3 pulp: pulp_puppet.plugins.importers.forge:ERROR: (7753-16960) Traceback (most recent call last): Apr 14 08:30:35 cloud-qe-3 pulp: pulp_puppet.plugins.importers.forge:ERROR: (7753-16960) File "/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/forge.py", line 113, in _parse_metadata Apr 14 08:30:35 cloud-qe-3 pulp: pulp_puppet.plugins.importers.forge:ERROR: (7753-16960) metadata_json_docs = downloader.retrieve_metadata(self.progress_report) Apr 14 08:30:35 cloud-qe-3 pulp: pulp_puppet.plugins.importers.forge:ERROR: (7753-16960) File "/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/downloaders/local.py", line 58, in retrieve_metadata Apr 14 08:30:35 cloud-qe-3 pulp: pulp_puppet.plugins.importers.forge:ERROR: (7753-16960) raise FileRetrievalException(report.error_msg) Apr 14 08:30:35 cloud-qe-3 pulp: pulp_puppet.plugins.importers.forge:ERROR: (7753-16960) FileRetrievalException: FileRetrievalException: [Errno 2] No such file or directory: u'///modules/modules.json' Looks like selinux issue: type=AVC msg=audit(1460637035.939:5311): avc: denied { read } for pid=7753 comm="celery" name="PULP_MANIFEST" dev="dm-0" ino=162121353 scontext=system_u:system_r:celery_t:s0 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file type=SYSCALL msg=audit(1460637035.939:5311): arch=c000003e syscall=2 success=no exit=-13 a0=3977ee0 a1=0 a2=1b6 a3=24 items=0 ppid=7498 pid=7753 auid=4294967295 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=4294967295 comm="celery" exe="/usr/bin/python2.7" subj=system_u:system_r:celery_t:s0 key=(null) Created attachment 1147231 [details]
complete logs with exception
Created redmine issue http://projects.theforeman.org/issues/15812 from this bug Opened an issue against pulp since I was able to reproduce with just pulp-admin: https://pulp.plan.io/issues/2167 We need to update the docs text. See: https://pulp.plan.io/issues/1560 Upstream bug assigned to daviddavis Upstream bug component is Repositories The Pulp upstream bug status is at CLOSED - NOTABUG. Updating the external tracker on this bug. The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug. We'll need some documentation around how to set SELinux labels on files. Here's an example: "If you have SELinux enabled, in order to sync from the file system, you'll need to apply a label to the files in order for Satellite 6 to access them. Two options are httpd_sys_r_content_t or pulp_tmp_t. Note: if you choose httpd_sys_r_content_t then the webserver can also read the files so that may or may not be good. One way to apply these labels would be to use chcon. See https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Security-Enhanced_Linux/sect-Security-Enhanced_Linux-Working_with_SELinux-SELinux_Contexts_Labeling_Files.html for more info." This applies to all content types (not just puppet). Upstream bug assigned to daviddavis Upstream bug component is Repositories Upstream bug assigned to daviddavis Moving to 'NEW' while assigned to the default assignee. Hello As per Bug 1301367 - pulp-puppet-module-builder and SELinux the guide was updated to use /var/www/puppet-modules See "Procedure 3.3. Publishing a Git Repository to a Local Directory" [1] [1] https://access.redhat.com/documentation/en/red-hat-satellite/6.2/paged/puppet-guide/35-adding-puppet-modules-from-a-git-repository Hello I grep'd al guides and the use of "pulp_tmp_t" and "httpd_sys_r_content_t" are not documented. So I will add them to the doc mentioned in comment 37. Assigning to Stephen for review. Hello These changes are now live on the customer portal. https://access.redhat.com/documentation/en/red-hat-satellite/6.2/paged/puppet-guide/35-adding-puppet-modules-from-a-git-repository#proc-Red_Hat_Satellite-Puppet_Guide-Adding_Puppet_Modules_from_a_Git_Repository-Publishing_a_Git_Repository_to_a_Local_Directory Thank you |