Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1142289 - "Empty reply from server"/httpd segfault when trying to consume content
Summary: "Empty reply from server"/httpd segfault when trying to consume content
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Content Management
Version: 6.0.4
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: Unspecified
Assignee: Mike McCune
QA Contact: Tazim Kolhar
URL:
Whiteboard:
Depends On: 1135127 1135144
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-09-16 13:54 UTC by Mike McCune
Modified: 2017-02-23 20:57 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of: 1135144
Environment:
Last Closed: 2015-08-07 16:21:04 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Mike McCune 2014-09-16 13:54:15 UTC
+++ This bug was initially created as a clone of Bug #1135144 +++

+++ This bug was initially created as a clone of Bug #1135127 +++

Description of problem:

When user tries to consume content from a sat, an error is thrown from yum, and checking the httpd logs on sat server, httpd is thrown a segfault every time client requests content.


Version-Release number of selected component (if applicable):

Satellite-6.0.4-RHEL-6-20140828.0 :(

How reproducible:


Steps to Reproduce:
1.  sync rhel content to a satellite
2.  register system to satellite
3.  attempt to consume content on client

Actual results:
[Errno 14] PYCURL ERROR 52 - "Empty reply from server"
Trying other mirror.
Error: Cannot retrieve repository metadata (repomd.xml) for repository: rhel-6-server-rpms. Please verify its path and try again

Every time content is requested, httpd throws an error akin to

[Thu Aug 28 22:10:38 2014] [notice] child pid 29705 exit signal Segmentation fault (11)



Expected results:

Content can be consumed.

Additional info:

--- Additional comment from RHEL Product and Program Management on 2014-08-28 17:21:06 EDT ---

This bug report previously had all acks and release flag approved.
However since at least one of its acks has been changed, the
release flag has been reset to ? by the bugbot (pm-rhel).  The
ack needs to become approved before the release flag can become
approved again.

--- Additional comment from RHEL Product and Program Management on 2014-08-28 17:34:14 EDT ---

Since this issue was entered in Red Hat Bugzilla, the pm_ack has been
set to + automatically for the next planned release

--- Additional comment from Michael Hrivnak on 2014-08-28 20:25:20 EDT ---

The code in 2.4.1-0.5.rc1 that verifies that a client's entitlement certificate is signed by a specified CA is broken. It actually causes apache processes to segfault. The problem is in new code that uses ctypes to call libcrypto directly from python. Fixing this code would (or will) take substantial investigation and testing.

For now, we are implementing a work-around to unblock sat6. The work-around is to disable client SSL cert verification in the wsgi access script, and instead let sat6 use mod_ssl to do that. This is a temporary fix that will need to be revisited before the next community release.

--- Additional comment from Jeff Ortel on 2014-08-28 20:29:51 EDT ---

https://github.com/pulp/pulp_rpm/pull/555

--- Additional comment from Jeff Ortel on 2014-08-28 20:30:33 EDT ---

Back to assigned for permanent fix.

--- Additional comment from Michael Hrivnak on 2014-08-28 20:48:58 EDT ---

This patch enables mod_ssl enforcement of client certificate CA verification. It assumes you want to use the CA installed by katello, so modify that path to enforce with some other CA.

--- Additional comment from Randy Barlow on 2014-09-11 17:05:30 EDT ---

https://github.com/pulp/pulp/pull/1152
https://github.com/pulp/pulp_rpm/pull/558

In order to verify the latest changes to this bug, there are four test cases to run:

1) Case #1 tests what will happen for users who upgrade and don't do anything. For them, there won't be a verify_ssl setting in /etc/pulp/repo_auth.conf, so make sure that isn't defined there. In this case, Pulp should assume you still want to do per-repo authorization. Test this case like you normally would for repo auth. It should succeed with valid entitlement certs and it should fail with invalid ones.

2) Case #2 tests what happens for users who explicitly set verify_ssl to true. The rest of this test case should work just like #1.

3) Case #3 tests what happens for users who set verify_ssl to false, but do not configure Apache to do the validation. In this case, all certificates that have the correct OIDs (the OIDs are still checked even when verify_ssl is false) should be allowed access, even if they are not signed by a valid CA. Make sure that certificates signed by invalid CAs are allowed access, as well as certificates that have valid CA signatures. Also, make sure that all certificates that don't have the correct OIDs for the given repos are still denied (this is very important!).

4) Case #4 tests what happens for users who set verify_ssl to false and also configure Apache to require validly signed certificates. This should again work just like Case #1, but Apache will be enforcing the signature checks instead of Pulp. Make sure the right certs are allowed and rejected, both due to signatures and due to correct/incorrect OIDs.

Note that the CA still needs to be uploaded to the repo in order to configure repo_auth for it, otherwise Pulp will not check the OIDs.

Please also verify the docs changes that went into the two pull requests.

--- Additional comment from Chris Duryee on 2014-09-12 11:11:52 EDT ---

I no-break tested this on a Sat 6 and it seemed OK. I did not test cases where requests should be denied. To do that, I think you'd need to set up a second sat6 and point a client registered to one to the other so you are using the same OID data but with different signing certs.

One thing to note, I broke my /usr/bin/openssl by mistake and got a 403 (correct) but did not see anything in /var/log/messages about not being able to shell out.

--- Additional comment from Randy Barlow on 2014-09-12 14:25:58 EDT ---

Fixed in pulp-2.4.1-0.8.beta.

--- Additional comment from Preethi Thomas on 2014-09-16 09:39:18 EDT ---

verified with different scenarios

Comment 1 Mike McCune 2014-09-16 13:57:26 UTC
NOTE:  Satellite 6 will need to include changes related to this bug if we get a new build of Pulp 2.4.1 in our ZStream release.

If during 6.0.5/6.0.6 we do not ship a new version of Pulp then we can close this bug out but we need to track this because it is included in 2.4.1

Comment 3 Mike McCune 2014-11-12 18:58:35 UTC
With 6.0.Z we are running with repository authentication still enabled which will have a performance impact, although small on repo authentication:

https://github.com/pulp/pulp_rpm/pull/558/files

For 6.1.0 we need to forcibly remove the specific Satellite build of m2crypto from our installed machines.  This will require katello-installer changes to downgrade this file and force it to run the rev that is in base RHEL.

Comment 4 Mike McCune 2015-08-07 16:21:04 UTC
This is a non-issue with 6.0 and 6.1 at this point. We don't ship any versions of m2crypto and had no issues during 6.0 -> 6.1 upgrades.

If a customer hits this for some reason they should just remove and re-install m2crypto from the base RHEL repositories.


Note You need to log in before you can comment on or make changes to this bug.