Bug 1242310

Summary: published repositories give 403 forbidden when attempting to navigate to the https address
Product: Red Hat Satellite Reporter: Nagoor Shaik <nshaik>
Component: CapsuleAssignee: Alex Wood <awood>
Status: CLOSED ERRATA QA Contact: Katello QA List <katello-qa-list>
Severity: urgent Docs Contact:
Priority: high    
Version: 6.0.8CC: aathomas, achadha, adahms, aheslin, awood, bbuckingham, bkearney, brubisch, dlezzoum, ehelms, ftsiadim, fvzwieten, hartsjc, jkachuck, jsherril, katello-bugs, katello-qa-list, mmccune, mmello, rbobek, rplevka, sthirugn, suprabhu, xdmoon
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, published repositories returned a 403 Forbidden status. With this release, repositories that are published as SSL-protected can be accessed using the debug certificate for an organization.
Story Points: ---
Clone Of: 1128870
: 1320179 1351648 (view as bug list) Environment:
Last Closed: 2016-07-27 11:06:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1128870    
Bug Blocks: 1115190, 1320179, 1351648    
Attachments:
Description Flags
Cert Downloaded from Satellite 6.1.3 none

Description Nagoor Shaik 2015-07-13 02:13:24 UTC
+++ This bug was initially created as a clone of Bug #1128870 +++

Description of problem:

Published repositories give 403 forbidden when attempting to navigate to the https address


Version-Release number of selected component (if applicable):
Version 6.0.8

How reproducible:
Choose organization, then Content > Content Views, choose Content> repositories click the repository and try to navigate to 'Published At' link to receive.

Forbidden

You don't have permission to access /pulp/repos/ACME_Corporation/Library/content/dist/rhel/server/6/6Server/x86_64/os on this server.

Actual results:
Can browse repos via HTTP but on HTTPS port it fails with below error

Forbidden

You don't have permission to access /pulp/repos/RedHat/ on this server.

Expected results:
Rpos should be browsable via https repo

Additional information: 

Followed the steps provided in https://access.redhat.com/documentation/en-US/Red_Hat_Satellite/6.0/html-single/User_Guide/#Create_a_New_Organization_Debug_Certificate

Still the repos are not browsable via HTTPS port. Below are my findings 

 # cat /etc/httpd/conf.d/pulp_nodes.conf file

# -- HTTP Repositories ---------

Alias /pulp/nodes/http/repos /var/www/pulp/nodes/http/repos

<Directory /var/www/pulp/nodes/http/repos >
  Options FollowSymLinks Indexes
</Directory>

# -- HTTPS Repositories ---------

Alias /pulp/nodes/https/repos /var/www/pulp/nodes/https/repos

<Directory /var/www/pulp/nodes/https/repos >
  Options FollowSymLinks Indexes
  SSLRequireSSL
  SSLVerifyClient require
  SSLVerifyDepth  5
  SSLOptions +FakeBasicAuth
  SSLRequire %{SSL_CLIENT_S_DN_O} eq "PULP" and %{SSL_CLIENT_S_DN_OU} eq "NODES"   ------> This line is causing the trouble as its expecting Org to be PULP and OU to be NODES in the CA cert which is why its failing
</Directory>

Also we can see some traceback/errors in the /var/log/httpd/foreman-ssl_error_ssl.log when accessing content on HTTPS port.

[root@dhcp233-55 ~]# tail -f -n0 /var/log/httpd/foreman-ssl_error_ssl.log
[Mon Jul 13 07:40:09.924854 2015] [:error] [pid 31432] [client 10.65.223.44:55319] mod_wsgi (pid=31432): Exception occurred processing WSGI script '/srv/pulp/repo_auth.wsgi'.
[Mon Jul 13 07:40:09.924963 2015] [:error] [pid 31432] [client 10.65.223.44:55319] Traceback (most recent call last):
[Mon Jul 13 07:40:09.925010 2015] [:error] [pid 31432] [client 10.65.223.44:55319]   File "/srv/pulp/repo_auth.wsgi", line 34, in allow_access
[Mon Jul 13 07:40:09.925175 2015] [:error] [pid 31432] [client 10.65.223.44:55319]     authorized = _handle(environ)
[Mon Jul 13 07:40:09.925203 2015] [:error] [pid 31432] [client 10.65.223.44:55319]   File "/srv/pulp/repo_auth.wsgi", line 71, in _handle
[Mon Jul 13 07:40:09.925246 2015] [:error] [pid 31432] [client 10.65.223.44:55319]     result = f(environ)
[Mon Jul 13 07:40:09.925274 2015] [:error] [pid 31432] [client 10.65.223.44:55319]   File "/usr/lib/python2.7/site-packages/pulp_rpm/repo_auth/oid_validation.py", line 46, in authenticate
[Mon Jul 13 07:40:09.925735 2015] [:error] [pid 31432] [client 10.65.223.44:55319]     environ["wsgi.errors"].write)
[Mon Jul 13 07:40:09.925786 2015] [:error] [pid 31432] [client 10.65.223.44:55319]   File "/usr/lib/python2.7/site-packages/pulp_rpm/repo_auth/oid_validation.py", line 120, in is_valid
[Mon Jul 13 07:40:09.925837 2015] [:error] [pid 31432] [client 10.65.223.44:55319]     is_valid = self._check_extensions(cert_pem, dest, log_func)
[Mon Jul 13 07:40:09.925863 2015] [:error] [pid 31432] [client 10.65.223.44:55319]   File "/usr/lib/python2.7/site-packages/pulp_rpm/repo_auth/oid_validation.py", line 166, in _check_extensions
[Mon Jul 13 07:40:09.925904 2015] [:error] [pid 31432] [client 10.65.223.44:55319]     cert = certificate.create_from_pem(cert_pem)
[Mon Jul 13 07:40:09.925930 2015] [:error] [pid 31432] [client 10.65.223.44:55319]   File "/usr/lib64/python2.7/site-packages/rhsm/certificate.py", line 64, in create_from_pem
[Mon Jul 13 07:40:09.926441 2015] [:error] [pid 31432] [client 10.65.223.44:55319]     return _CertFactory().create_from_pem(pem)
[Mon Jul 13 07:40:09.926529 2015] [:error] [pid 31432] [client 10.65.223.44:55319]   File "/usr/lib64/python2.7/site-packages/rhsm/certificate2.py", line 74, in create_from_pem
[Mon Jul 13 07:40:09.927029 2015] [:error] [pid 31432] [client 10.65.223.44:55319]     return self._read_x509(_certificate.load(pem=pem), path, pem)
[Mon Jul 13 07:40:09.927160 2015] [:error] [pid 31432] [client 10.65.223.44:55319]   File "/usr/lib64/python2.7/site-packages/rhsm/certificate2.py", line 100, in _read_x509
[Mon Jul 13 07:40:09.927304 2015] [:error] [pid 31432] [client 10.65.223.44:55319]     raise CertificateException(str(e))
[Mon Jul 13 07:40:09.927491 2015] [:error] [pid 31432] [client 10.65.223.44:55319] CertificateException: Invalid date Bad time value: Invalid month: Bad
[Mon Jul 13 07:40:09.927717 2015] [:error] [pid 31432] [client 10.65.223.44:55319] mod_wsgi (pid=31432): Client denied by server configuration: '/var/www/pub/yum/https/repos/RedHat'.

[root@dhcp233-55 ~]# date
Mon Jul 13 07:40:32 IST 2015

Comment 1 RHEL Program Management 2015-07-13 02:24:20 UTC
Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

Comment 6 Bryan Kearney 2015-10-09 12:42:06 UTC
Created attachment 1081327 [details]
Cert Downloaded from Satellite 6.1.3

Comment 7 Joseph Kachuck 2015-11-09 19:34:06 UTC
May I allow IBM access to this BZ?

Thank You
Joe Kachuck

Comment 8 Mike McCune 2015-11-24 21:34:32 UTC
BZ is now public.

Comment 9 Justin Sherrill 2016-03-03 22:00:59 UTC
After chatting with the candlepin team, this seems like it may be a bug in m2crypto.  

I wrote a small python script https://gist.github.com/jlsherrill/ad4e9a7122c08e6a7436 to test a particular cert and the uber certs seem to fail.  

Al's analysis indicates that certs with an expiration date after 2050 fail to validate in m2crypto.

Comment 11 Alex Wood 2016-03-07 19:29:12 UTC
Filed upstream bug with M2Crypto at https://gitlab.com/m2crypto/m2crypto/issues/103

However, python-rhsm only uses M2Crypto to read version 1 entitlement certificates.  Version 3 entitlement certificates are read via a C extension.  The same problem exists in that extension as well (GeneralizedTimes being interpreted as UTCTimes).  See https://github.com/candlepin/python-rhsm/blob/d437f96f0aa3311f4b9b21a5fd128825ecc52f72/src/certificate.c#L447

The fastest fix is to path Candlepin to not issue certificates with an expiration date past 2050.

Comment 12 Bryan Kearney 2016-03-07 21:06:15 UTC
Upstream bug component is Capsule

Comment 13 Bryan Kearney 2016-03-10 19:50:04 UTC
Upstream bug assigned to jomitsch

Comment 14 Justin Sherrill 2016-03-10 19:56:45 UTC
Unlinked from upstream issue as the problem is being resolved in candlepin

Comment 16 Brad Buckingham 2016-05-02 19:13:00 UTC
Based on testing with Satellite 6.2 Beta Snap 9.x, this appears to have been solved.  I am going to move it ON_QA for verification.

Comment 17 Roman Plevka 2016-05-11 11:45:30 UTC
VERIFIED
on Sat 6.2.0 Beta Snap GA10.1

The repo listing now works just fine with using the Org certificate:


$ curl -s -k --cert ~/Downloads/Default\ Organization-key-cert.pem https://intel-canoepass-11.lab.bos.redhat.com/pulp/repos/Default_Organization/Library/content/dist/rhel/server/6/6.7/x86_64/os | head -n20

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Pulp Repository Index</title>
</head>
<body>
    <h1>Pulp Repository Content</h1>
    <a href="../">Parent Directory</a>
    <ul style='list-style: none outside none; font-family: monospace'>
            <li><a href="repodata/">repodata/</a></li>
            <li><a href="389-ds-base-1.2.10.2-15.el6.x86_64.rpm">389-ds-base-1.2.10.2-15.el6.x86_64.rpm</a></li>
            <li><a href="389-ds-base-1.2.10.2-18.el6_3.x86_64.rpm">389-ds-base-1.2.10.2-18.el6_3.x86_64.rpm</a></li>

Comment 18 James Hartsock 2016-06-15 18:30:31 UTC
Still failing after updating my Sat 6.1 to Sat 6.2 beta 2 release.

$ curl -s -k --cert ~/Downloads/PUB-key-cert.pem https://sat6.pub.com/pulp/repos/Org-Pub/Library/content/dist/rhel/server/7/7Server/x86_64/os | head -n 20

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Internal Server Error</title>
</head>
<body>
    <h1>HTTP 500: Internal server error.</h1>
    <p>An unexpected error occurred while handling your request.</p>
</body>
</html>


Same error when using web-browser too.

Comment 19 Roman Plevka 2016-07-18 13:28:58 UTC
(In reply to James Hartsock from comment #18)
> Still failing after updating my Sat 6.1 to Sat 6.2 beta 2 release.
> 
> $ curl -s -k --cert ~/Downloads/PUB-key-cert.pem
> https://sat6.pub.com/pulp/repos/Org-Pub/Library/content/dist/rhel/server/7/
> 7Server/x86_64/os | head -n 20
> 
> <!DOCTYPE html>
> <html lang="en">
> <head>
>     <meta charset="UTF-8">
>     <title>Internal Server Error</title>
> </head>
> <body>
>     <h1>HTTP 500: Internal server error.</h1>
>     <p>An unexpected error occurred while handling your request.</p>
> </body>
> </html>
> 
> 
> Same error when using web-browser too.

Hi James, that looks like a different issue as you're getting ISE (500) instead of Access Forbidden (403).
Could you check your server logs to find out what went wrong? (/var/log/foreman/production.log + httpd logs (ssl_error.log, etc.))

Comment 20 James Hartsock 2016-07-18 14:35:14 UTC
Roman,

There is a 403 for the request in var/log/httpd/foreman-ssl_access_ssl.log
~~~
172.31.0.40 - - [18/Jul/2016:09:32:47 -0500] "GET /pulp/repos/Org-Pub/Library/content/dist/rhel/server/7/7Server/x86_64/os HTTP/1.1" 403 303 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36"

172.31.0.40 - 2c9f809451549b3a015154a55d580009 [18/Jul/2016:09:32:47 -0500] "GET /pulp/repos/Org-Pub/Library/content/dist/rhel/server/7/7Server/x86_64/os HTTP/1.1" 500 254 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36"

172.31.0.20 - admin [18/Jul/2016:09:32:51 -0500] "GET /pulp/api/v2/tasks/5ff18de6-9e69-4d0b-865f-f2b16f3b7c31/ HTTP/1.1" 200 328 "-" "Ruby"
~~~

var/log/httpd/foreman-ssl_error_ssl.log
~~
[Mon Jul 18 09:32:47.893845 2016] [ssl:error] [pid 2488] [client 172.31.0.40:49056] AH02261: Re-negotiation handshake failed: Not accepted by client!?
~~~

Comment 21 Roman Plevka 2016-07-18 15:33:09 UTC
(In reply to James Hartsock from comment #20)
> Roman,
> 
> There is a 403 for the request in var/log/httpd/foreman-ssl_access_ssl.log
> ~~~
> 172.31.0.40 - - [18/Jul/2016:09:32:47 -0500] "GET
> /pulp/repos/Org-Pub/Library/content/dist/rhel/server/7/7Server/x86_64/os
> HTTP/1.1" 403 303 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36"
> 
> 172.31.0.40 - 2c9f809451549b3a015154a55d580009 [18/Jul/2016:09:32:47 -0500]
> "GET
> /pulp/repos/Org-Pub/Library/content/dist/rhel/server/7/7Server/x86_64/os
> HTTP/1.1" 500 254 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36"
> 
> 172.31.0.20 - admin [18/Jul/2016:09:32:51 -0500] "GET
> /pulp/api/v2/tasks/5ff18de6-9e69-4d0b-865f-f2b16f3b7c31/ HTTP/1.1" 200 328
> "-" "Ruby"
> ~~~
> 
> var/log/httpd/foreman-ssl_error_ssl.log
> ~~
> [Mon Jul 18 09:32:47.893845 2016] [ssl:error] [pid 2488] [client
> 172.31.0.40:49056] AH02261: Re-negotiation handshake failed: Not accepted by
> client!?
> ~~~

Sorry James,
got a satellite upgraded from 6.1.9 to recent beta snap and it works just fine.
(for the web browser, i needed to follow these steps in order to convert the certificate to supported format: http://www.katello.org/troubleshooting/debug_certificate.html)

If you have a reproducer steps, I'd suggest you to open a new BZ.

Comment 22 James Hartsock 2016-07-18 15:46:59 UTC
$ sed -n '/-----BEGIN RSA PRIVATE KEY-----/,/-----END RSA PRIVATE KEY-----/p' PUB-key-cert.pem  > key.pem


$ sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' PUB-key-cert.pem  > cert.pem


$ openssl pkcs12 -keypbe PBE-SHA1-3DES -certpbe PBE-SHA1-3DES -export -in cert.pem -inkey key.pem -out sat6.pfx v
openssl pkcs12 -keypbe PBE-SHA1-3DES -certpbe PBE-SHA1-3DES -export -in cert.pem -inkey key.pem -out sat6.pfx -name sat6.pub.com
Enter Export Password:
Verifying - Enter Export Password:


chrome://settings/certificates
import -> ~/Downloads/sat6.pfx

Still fails for me, but if working for others fine.

Comment 23 Bryan Kearney 2016-07-27 11:06:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501

Comment 24 Djebran Lezzoum 2017-12-13 08:48:18 UTC
qe_test_coverage PR: https://github.com/SatelliteQE/robottelo/pull/5672