Bug 1000586 - Connecting to proxy through /etc/hosts stoped working after upgrading to 1.7.12-11+
Connecting to proxy through /etc/hosts stoped working after upgrading to 1.7....
Status: CLOSED CURRENTRELEASE
Product: Spacewalk
Classification: Community
Component: Proxy Server (Show other bugs)
2.0
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Stephen Herr
Red Hat Satellite QA List
: Regression
Depends On: 962505
Blocks: space21
  Show dependency treegraph
 
Reported: 2013-08-23 14:44 EDT by Stephen Herr
Modified: 2014-03-04 08:08 EST (History)
8 users (show)

See Also:
Fixed In Version: spacewalk-proxy-2.1.6-1
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 962505
Environment:
Last Closed: 2014-03-04 08:06:56 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Stephen Herr 2013-08-23 14:44:48 EDT
+++ This bug was initially created as a clone of Bug #962505 +++

Description of problem:
Our proxies start responding to hosts checking in with 404 errors after updating the following packages to 1.7.12-11 or 1.7.12-14. 1.7.12-10 appears to be the last version that worked.

spacewalk-proxy-broker.noarch
spacewalk-proxy-common.noarch
spacewalk-proxy-management.noarch
spacewalk-proxy-package-manager.noarch
spacewalk-proxy-redirect.noarch  

Version-Release number of selected component (if applicable):
1.7.12-11 and 1.7.12-14

How reproducible:
Always

Steps to Reproduce:
1. Configure Satellite Proxies with a CNAME and an SSL Cert with appropriate subjectAltNames
2. Do a yum update and receive 1.7.12-14 updates
  
Actual results:
requests to the cname addresses stop working

Expected results:
They should continue to work.

Additional info:
Version 1.7.12-10 and prior work. The hosts are configured with cname entries and subjectaltnames for both the cname and a record addreses.

Updating to 1.7.12-11 or 1.7.12-14 breaks things.

Downgrading to 1.7.12-10 gets things working again.

--
On the client this looks like:
yum check-update
Loaded plugins: downloadonly, product-id, rhnplugin, security, subscription-
              : manager
Updating certificate-based repositories.
Unable to read consumer identity
Error: Cannot retrieve repository metadata (repomd.xml) for repository: rhel-x86_64-rhev-agent-6-server. Please verify its path and try again

--
On the server this looks like:
10.11.14.1 - - [13/May/2013:12:25:46 -0400] "POST /XMLRPC HTTP/1.1" 200 3097
10.11.14.1 - - [13/May/2013:12:25:46 -0400] "POST /XMLRPC HTTP/1.1" 200 12448
10.11.14.1 - - [13/May/2013:12:25:46 -0400] "GET /XMLRPC/GET-REQ/rhel-x86_64-server-6/repodata/repomd.xml HTTP/1.1" 404 -
10.11.14.1 - - [13/May/2013:12:25:46 -0400] "POST /XMLRPC HTTP/1.1" 200 3097
10.11.14.1 - - [13/May/2013:12:25:47 -0400] "GET /XMLRPC/GET-REQ/rhel-x86_64-server-6/repodata/repomd.xml HTTP/1.1" 404 -
10.11.14.1 - - [13/May/2013:12:25:47 -0400] "POST /XMLRPC HTTP/1.1" 200 3097
10.11.14.1 - - [13/May/2013:12:25:47 -0400] "GET /XMLRPC/GET-REQ/rhel-x86_64-server-6/repodata/repomd.xml HTTP/1.1" 404 -
10.11.14.1 - - [13/May/2013:12:25:47 -0400] "POST /XMLRPC HTTP/1.1" 200 3097
10.11.14.1 - - [13/May/2013:12:25:48 -0400] "GET /XMLRPC/GET-REQ/rhel-x86_64-server-6/repodata/repomd.xml HTTP/1.1" 404 -

--- Additional comment from Jason Montleon on 2013-05-13 13:29:17 EDT ---

I should probably at that if the actual hostname of the system is used downloads continue to work as expected.

--- Additional comment from Jason Montleon on 2013-05-13 14:53:03 EDT ---

This is less of an issue than we initially thought. We were testing with /etc/hosts entries before cutting over to a new server. It seems using an actual cname it works. Still kind of odd that it stopped working between -10 and -11.

--- Additional comment from Stephen Herr on 2013-08-21 15:00:34 EDT ---

Okay. Proxy bugs are always a nightmare to debug / fix, but in this case thanks to the very narrowly confined version that this was introduced in (thanks!) I think I know what is wrong here.

If you make a request from a client to the proxy and there's a proxy cache miss (either the actual file you're requesting or the client auth token validity) the process will look something like this.

1) client makes request
2) proxy receives request and processes it (checks auth tokens, etc)
3) proxy redirects request to self through squid to get cached content
3b) if squid cache miss, redirect request on up to Hosted / Satellite.
3c) populate cache with Hosted / Satellite response
4) respond to client with requested info

In step 3 the proxy needs to redirect the request back to itself. This used to be accomplished by redirecting to 127.0.0.1, but this was causing problems. See bug 872721. The thing that changed between proxy 1.7.12-10 and -11 is that instead of redirecting to to 127.0.0.1 we redirect to the proxy's hostname which we get from the HOST header of the incoming request.

In this case however (the client is using a hostname that only exists in it's local /etc/hosts) the HOST header is a lie, no one on the network knows about that hostname except the client. So when the proxy tries to redirect to itself it fails, and we end up passing a 404 down to the client.

I think the fix for this then is to not trust the client to know what our hostname is, instead look it up for ourselves every time.

--- Additional comment from Stephen Herr on 2013-08-23 11:53:08 EDT ---

Changing the title to reflect more accurately the situation. This has nothing to do with CNAMEs, but rather /etc/hosts entries (they are not the same thing).

CNAMEs continue to work fine.
Comment 1 Stephen Herr 2013-08-23 14:48:25 EDT
Committing to Spacewalk master:
4273986e4c1996c6c575a4cc4ca9d2c5587acf1c
Comment 2 Stephen Herr 2013-08-23 15:53:51 EDT
Fixing checkstyle errors:
683e4debba61b41caadd54cd78d7164b65984f8c
Comment 3 Stephen Herr 2013-08-23 16:25:48 EDT
Fixing checkstyle errors for real this time:
e3b3bf74c9cee069c75231ddfd9f9f27cb2a7858
Comment 4 Stephen Herr 2013-08-23 16:33:30 EDT
And the new one I just introduced with the long line :(
454960a2a3065b8b652ce06a12460c0dcbf4f975
Comment 5 Matej Kollar 2014-01-17 07:15:50 EST
Switching MODIFIED Spacewalk bugs to ON_QA before 2.1 release.
Comment 6 Matej Kollar 2014-03-04 08:06:56 EST
Spacewalk 2.1 has been released.
https://fedorahosted.org/spacewalk/wiki/ReleaseNotes21
Comment 7 Matej Kollar 2014-03-04 08:08:39 EST
Spacewalk 2.1 has been released.
https://fedorahosted.org/spacewalk/wiki/ReleaseNotes21

Note You need to log in before you can comment on or make changes to this bug.