Bug 813347

Summary: IOError: [Errno 13] Permission denied: '/etc/pki/pulp/content/pulp-global-repo.ca'
Product: Red Hat Satellite Reporter: James Laska <jlaska>
Component: Content ManagementAssignee: Katello Bug Bin <katello-bugs>
Status: CLOSED NOTABUG QA Contact: Katello QA List <katello-qa-list>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.0.0CC: jturner, lzap
Target Milestone: Unspecified   
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-04-17 16:32:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description James Laska 2012-04-17 14:39:23 UTC
Description of problem:

Trying to debug why the client cert and key in my katello system templates results in a 403 Forbidden.  I noticed the cause in the httpd error log ...

> [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38] mod_wsgi (pid=23300): Exception occurred processing WSGI script '/srv/pulp/repo_auth.wsgi'.
> [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38] Traceback (most recent call last):
> [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38]   File "/srv/pulp/repo_auth.wsgi", line 34, in allow_access
> [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38]     authorized = _handle(environ)
> [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38]   File "/srv/pulp/repo_auth.wsgi", line 71, in _handle
> [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38]     result = f(environ)
> [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38]   File "/usr/lib/python2.6/site-packages/pulp/repo_auth/oid_validation.py", line 61, in authenticate
> [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38]     environ["wsgi.errors"].write)
> [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38]   File "/usr/lib/python2.6/site-packages/pulp/repo_auth/oid_validation.py", line 107, in is_valid
> [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38]     global_bundle = self.repo_cert_utils.read_global_cert_bundle(['ca'])
> [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38]   File "/usr/lib/python2.6/site-packages/pulp/repo_auth/repo_cert_utils.py", line 148, in read_global_cert_bundle
> [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38]     f = open(filename, 'r')
> [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38] IOError: [Errno 13] Permission denied: '/etc/pki/pulp/content/pulp-global-repo.ca'
> [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38] mod_wsgi (pid=23300): Client denied by server configuration: '/var/www/pub/repos/redhat/Dev/content/beta/rhel/server/5/5.8/i386/cf-tools/1.0/os/repodata/repomd.xml'.


Version-Release number of selected component (if applicable):
 * katello-0.1.309-1.el6.src.rpm
 * katello-candlepin-cert-key-pair-1.0-1.src.rpm
 * katello-certs-tools-1.0.4-1.el6.src.rpm
 * katello-cli-0.1.107-1.el6.src.rpm
 * katello-configure-0.1.107-1.el6.src.rpm
 * katello-qpid-broker-key-pair-1.0-1.src.rpm
 * katello-qpid-client-key-pair-1.0-1.src.rpm
 * katello-selinux-0.1.10-1.el6.src.rpm
 * pulp-1.0.4-1.el6.src.rpm


How reproducible:


Steps to Reproduce:
1. Generate valid system templates
2. Use them to build and deploy working images over a period of several days

Actual results:

All of the sudden, the client cert and key used in my templates is no longer valid.

> # curl --silent --cert /tmp/my.crt --key /tmp/my.key --insecure https://qeblade31.rhq.lab.eng.bos.redhat.com/pulp/repos/redhat/Dev/content/beta/rhel/server/5/5.8/i386/cf-tools/1.0/os/repodata/repomd.xml
> <snip>...
> <p>You don't have permission to access /pulp/repos/redhat/Dev/content/beta/rhel/server/5/5.8/i386/cf-tools/1.0/os/repodata/repomd.xml

Expected results:

The client cert and key should continue to work

Additional info:

Comment 1 James Laska 2012-04-17 14:40:16 UTC
The system templates I've used for all successful image builds are available at https://qeblade31.rhq.lab.eng.bos.redhat.com/templates/Dev/

Comment 2 Lukas Zapletal 2012-04-17 15:41:49 UTC
After our chat - something changed permissions of /etc/candlepin/certs/candlepin-ca.crt from 644 to 600.

Comment 3 James Laska 2012-04-17 16:32:12 UTC
Some{one,thing} changed the permissions.  I cannot determine what changed the permissions at this time.  With guidance from Lukas, I have setup a systemtap trap to catch if/when the file permissions change next time.

http://lukas.zapletalovi.com/2012/04/setup-systemtap-permission-change-trap.html

If it turns out that the cause of the permissions change is not a human error ... I will re-open this bug report.