Description of problem: Trying to debug why the client cert and key in my katello system templates results in a 403 Forbidden. I noticed the cause in the httpd error log ... > [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38] mod_wsgi (pid=23300): Exception occurred processing WSGI script '/srv/pulp/repo_auth.wsgi'. > [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38] Traceback (most recent call last): > [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38] File "/srv/pulp/repo_auth.wsgi", line 34, in allow_access > [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38] authorized = _handle(environ) > [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38] File "/srv/pulp/repo_auth.wsgi", line 71, in _handle > [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38] result = f(environ) > [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38] File "/usr/lib/python2.6/site-packages/pulp/repo_auth/oid_validation.py", line 61, in authenticate > [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38] environ["wsgi.errors"].write) > [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38] File "/usr/lib/python2.6/site-packages/pulp/repo_auth/oid_validation.py", line 107, in is_valid > [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38] global_bundle = self.repo_cert_utils.read_global_cert_bundle(['ca']) > [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38] File "/usr/lib/python2.6/site-packages/pulp/repo_auth/repo_cert_utils.py", line 148, in read_global_cert_bundle > [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38] f = open(filename, 'r') > [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38] IOError: [Errno 13] Permission denied: '/etc/pki/pulp/content/pulp-global-repo.ca' > [Tue Apr 17 10:15:34 2012] [error] [client 10.11.230.38] mod_wsgi (pid=23300): Client denied by server configuration: '/var/www/pub/repos/redhat/Dev/content/beta/rhel/server/5/5.8/i386/cf-tools/1.0/os/repodata/repomd.xml'. Version-Release number of selected component (if applicable): * katello-0.1.309-1.el6.src.rpm * katello-candlepin-cert-key-pair-1.0-1.src.rpm * katello-certs-tools-1.0.4-1.el6.src.rpm * katello-cli-0.1.107-1.el6.src.rpm * katello-configure-0.1.107-1.el6.src.rpm * katello-qpid-broker-key-pair-1.0-1.src.rpm * katello-qpid-client-key-pair-1.0-1.src.rpm * katello-selinux-0.1.10-1.el6.src.rpm * pulp-1.0.4-1.el6.src.rpm How reproducible: Steps to Reproduce: 1. Generate valid system templates 2. Use them to build and deploy working images over a period of several days Actual results: All of the sudden, the client cert and key used in my templates is no longer valid. > # curl --silent --cert /tmp/my.crt --key /tmp/my.key --insecure https://qeblade31.rhq.lab.eng.bos.redhat.com/pulp/repos/redhat/Dev/content/beta/rhel/server/5/5.8/i386/cf-tools/1.0/os/repodata/repomd.xml > <snip>... > <p>You don't have permission to access /pulp/repos/redhat/Dev/content/beta/rhel/server/5/5.8/i386/cf-tools/1.0/os/repodata/repomd.xml Expected results: The client cert and key should continue to work Additional info:
The system templates I've used for all successful image builds are available at https://qeblade31.rhq.lab.eng.bos.redhat.com/templates/Dev/
After our chat - something changed permissions of /etc/candlepin/certs/candlepin-ca.crt from 644 to 600.
Some{one,thing} changed the permissions. I cannot determine what changed the permissions at this time. With guidance from Lukas, I have setup a systemtap trap to catch if/when the file permissions change next time. http://lukas.zapletalovi.com/2012/04/setup-systemtap-permission-change-trap.html If it turns out that the cause of the permissions change is not a human error ... I will re-open this bug report.