Bug 1142289

Summary: "Empty reply from server"/httpd segfault when trying to consume content
Product: Red Hat Satellite Reporter: Mike McCune <mmccune>
Component: Content ManagementAssignee: Mike McCune <mmccune>
Status: CLOSED WORKSFORME QA Contact: Tazim Kolhar <tkolhar>
Severity: high Docs Contact:
Priority: urgent    
Version: 6.0.4CC: bbuckingham, bkearney, cduryee, cwelton, jortel, katello-qa-list, mhrivnak, mmccune, pthomas, rbarlow, sthirugn
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1135144 Environment:
Last Closed: 2015-08-07 16:21:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1135127, 1135144    
Bug Blocks:    

Description Mike McCune 2014-09-16 13:54:15 UTC
+++ This bug was initially created as a clone of Bug #1135144 +++

+++ This bug was initially created as a clone of Bug #1135127 +++

Description of problem:

When user tries to consume content from a sat, an error is thrown from yum, and checking the httpd logs on sat server, httpd is thrown a segfault every time client requests content.


Version-Release number of selected component (if applicable):

Satellite-6.0.4-RHEL-6-20140828.0 :(

How reproducible:


Steps to Reproduce:
1.  sync rhel content to a satellite
2.  register system to satellite
3.  attempt to consume content on client

Actual results:
[Errno 14] PYCURL ERROR 52 - "Empty reply from server"
Trying other mirror.
Error: Cannot retrieve repository metadata (repomd.xml) for repository: rhel-6-server-rpms. Please verify its path and try again

Every time content is requested, httpd throws an error akin to

[Thu Aug 28 22:10:38 2014] [notice] child pid 29705 exit signal Segmentation fault (11)



Expected results:

Content can be consumed.

Additional info:

--- Additional comment from RHEL Product and Program Management on 2014-08-28 17:21:06 EDT ---

This bug report previously had all acks and release flag approved.
However since at least one of its acks has been changed, the
release flag has been reset to ? by the bugbot (pm-rhel).  The
ack needs to become approved before the release flag can become
approved again.

--- Additional comment from RHEL Product and Program Management on 2014-08-28 17:34:14 EDT ---

Since this issue was entered in Red Hat Bugzilla, the pm_ack has been
set to + automatically for the next planned release

--- Additional comment from Michael Hrivnak on 2014-08-28 20:25:20 EDT ---

The code in 2.4.1-0.5.rc1 that verifies that a client's entitlement certificate is signed by a specified CA is broken. It actually causes apache processes to segfault. The problem is in new code that uses ctypes to call libcrypto directly from python. Fixing this code would (or will) take substantial investigation and testing.

For now, we are implementing a work-around to unblock sat6. The work-around is to disable client SSL cert verification in the wsgi access script, and instead let sat6 use mod_ssl to do that. This is a temporary fix that will need to be revisited before the next community release.

--- Additional comment from Jeff Ortel on 2014-08-28 20:29:51 EDT ---

https://github.com/pulp/pulp_rpm/pull/555

--- Additional comment from Jeff Ortel on 2014-08-28 20:30:33 EDT ---

Back to assigned for permanent fix.

--- Additional comment from Michael Hrivnak on 2014-08-28 20:48:58 EDT ---

This patch enables mod_ssl enforcement of client certificate CA verification. It assumes you want to use the CA installed by katello, so modify that path to enforce with some other CA.

--- Additional comment from Randy Barlow on 2014-09-11 17:05:30 EDT ---

https://github.com/pulp/pulp/pull/1152
https://github.com/pulp/pulp_rpm/pull/558

In order to verify the latest changes to this bug, there are four test cases to run:

1) Case #1 tests what will happen for users who upgrade and don't do anything. For them, there won't be a verify_ssl setting in /etc/pulp/repo_auth.conf, so make sure that isn't defined there. In this case, Pulp should assume you still want to do per-repo authorization. Test this case like you normally would for repo auth. It should succeed with valid entitlement certs and it should fail with invalid ones.

2) Case #2 tests what happens for users who explicitly set verify_ssl to true. The rest of this test case should work just like #1.

3) Case #3 tests what happens for users who set verify_ssl to false, but do not configure Apache to do the validation. In this case, all certificates that have the correct OIDs (the OIDs are still checked even when verify_ssl is false) should be allowed access, even if they are not signed by a valid CA. Make sure that certificates signed by invalid CAs are allowed access, as well as certificates that have valid CA signatures. Also, make sure that all certificates that don't have the correct OIDs for the given repos are still denied (this is very important!).

4) Case #4 tests what happens for users who set verify_ssl to false and also configure Apache to require validly signed certificates. This should again work just like Case #1, but Apache will be enforcing the signature checks instead of Pulp. Make sure the right certs are allowed and rejected, both due to signatures and due to correct/incorrect OIDs.

Note that the CA still needs to be uploaded to the repo in order to configure repo_auth for it, otherwise Pulp will not check the OIDs.

Please also verify the docs changes that went into the two pull requests.

--- Additional comment from Chris Duryee on 2014-09-12 11:11:52 EDT ---

I no-break tested this on a Sat 6 and it seemed OK. I did not test cases where requests should be denied. To do that, I think you'd need to set up a second sat6 and point a client registered to one to the other so you are using the same OID data but with different signing certs.

One thing to note, I broke my /usr/bin/openssl by mistake and got a 403 (correct) but did not see anything in /var/log/messages about not being able to shell out.

--- Additional comment from Randy Barlow on 2014-09-12 14:25:58 EDT ---

Fixed in pulp-2.4.1-0.8.beta.

--- Additional comment from Preethi Thomas on 2014-09-16 09:39:18 EDT ---

verified with different scenarios

Comment 1 Mike McCune 2014-09-16 13:57:26 UTC
NOTE:  Satellite 6 will need to include changes related to this bug if we get a new build of Pulp 2.4.1 in our ZStream release.

If during 6.0.5/6.0.6 we do not ship a new version of Pulp then we can close this bug out but we need to track this because it is included in 2.4.1

Comment 3 Mike McCune 2014-11-12 18:58:35 UTC
With 6.0.Z we are running with repository authentication still enabled which will have a performance impact, although small on repo authentication:

https://github.com/pulp/pulp_rpm/pull/558/files

For 6.1.0 we need to forcibly remove the specific Satellite build of m2crypto from our installed machines.  This will require katello-installer changes to downgrade this file and force it to run the rev that is in base RHEL.

Comment 4 Mike McCune 2015-08-07 16:21:04 UTC
This is a non-issue with 6.0 and 6.1 at this point. We don't ship any versions of m2crypto and had no issues during 6.0 -> 6.1 upgrades.

If a customer hits this for some reason they should just remove and re-install m2crypto from the base RHEL repositories.