Bug 1000586

+++ This bug was initially created as a clone of Bug #962505 +++

Description of problem:
Our proxies start responding to hosts checking in with 404 errors after updating the following packages to 1.7.12-11 or 1.7.12-14. 1.7.12-10 appears to be the last version that worked.

spacewalk-proxy-broker.noarch
spacewalk-proxy-common.noarch
spacewalk-proxy-management.noarch
spacewalk-proxy-package-manager.noarch
spacewalk-proxy-redirect.noarch  

Version-Release number of selected component (if applicable):
1.7.12-11 and 1.7.12-14

How reproducible:
Always

Steps to Reproduce:
1. Configure Satellite Proxies with a CNAME and an SSL Cert with appropriate subjectAltNames
2. Do a yum update and receive 1.7.12-14 updates
  
Actual results:
requests to the cname addresses stop working

Expected results:
They should continue to work.

Additional info:
Version 1.7.12-10 and prior work. The hosts are configured with cname entries and subjectaltnames for both the cname and a record addreses.

Updating to 1.7.12-11 or 1.7.12-14 breaks things.

Downgrading to 1.7.12-10 gets things working again.

--
On the client this looks like:
yum check-update
Loaded plugins: downloadonly, product-id, rhnplugin, security, subscription-
              : manager
Updating certificate-based repositories.
Unable to read consumer identity
Error: Cannot retrieve repository metadata (repomd.xml) for repository: rhel-x86_64-rhev-agent-6-server. Please verify its path and try again

--
On the server this looks like:
10.11.14.1 - - [13/May/2013:12:25:46 -0400] "POST /XMLRPC HTTP/1.1" 200 3097
10.11.14.1 - - [13/May/2013:12:25:46 -0400] "POST /XMLRPC HTTP/1.1" 200 12448
10.11.14.1 - - [13/May/2013:12:25:46 -0400] "GET /XMLRPC/GET-REQ/rhel-x86_64-server-6/repodata/repomd.xml HTTP/1.1" 404 -
10.11.14.1 - - [13/May/2013:12:25:46 -0400] "POST /XMLRPC HTTP/1.1" 200 3097
10.11.14.1 - - [13/May/2013:12:25:47 -0400] "GET /XMLRPC/GET-REQ/rhel-x86_64-server-6/repodata/repomd.xml HTTP/1.1" 404 -
10.11.14.1 - - [13/May/2013:12:25:47 -0400] "POST /XMLRPC HTTP/1.1" 200 3097
10.11.14.1 - - [13/May/2013:12:25:47 -0400] "GET /XMLRPC/GET-REQ/rhel-x86_64-server-6/repodata/repomd.xml HTTP/1.1" 404 -
10.11.14.1 - - [13/May/2013:12:25:47 -0400] "POST /XMLRPC HTTP/1.1" 200 3097
10.11.14.1 - - [13/May/2013:12:25:48 -0400] "GET /XMLRPC/GET-REQ/rhel-x86_64-server-6/repodata/repomd.xml HTTP/1.1" 404 -

--- Additional comment from Jason Montleon on 2013-05-13 13:29:17 EDT ---

I should probably at that if the actual hostname of the system is used downloads continue to work as expected.

--- Additional comment from Jason Montleon on 2013-05-13 14:53:03 EDT ---

This is less of an issue than we initially thought. We were testing with /etc/hosts entries before cutting over to a new server. It seems using an actual cname it works. Still kind of odd that it stopped working between -10 and -11.

--- Additional comment from Stephen Herr on 2013-08-21 15:00:34 EDT ---

Okay. Proxy bugs are always a nightmare to debug / fix, but in this case thanks to the very narrowly confined version that this was introduced in (thanks!) I think I know what is wrong here.

If you make a request from a client to the proxy and there's a proxy cache miss (either the actual file you're requesting or the client auth token validity) the process will look something like this.

1) client makes request
2) proxy receives request and processes it (checks auth tokens, etc)
3) proxy redirects request to self through squid to get cached content
3b) if squid cache miss, redirect request on up to Hosted / Satellite.
3c) populate cache with Hosted / Satellite response
4) respond to client with requested info

In step 3 the proxy needs to redirect the request back to itself. This used to be accomplished by redirecting to 127.0.0.1, but this was causing problems. See bug 872721. The thing that changed between proxy 1.7.12-10 and -11 is that instead of redirecting to to 127.0.0.1 we redirect to the proxy's hostname which we get from the HOST header of the incoming request.

In this case however (the client is using a hostname that only exists in it's local /etc/hosts) the HOST header is a lie, no one on the network knows about that hostname except the client. So when the proxy tries to redirect to itself it fails, and we end up passing a 404 down to the client.

I think the fix for this then is to not trust the client to know what our hostname is, instead look it up for ourselves every time.

--- Additional comment from Stephen Herr on 2013-08-23 11:53:08 EDT ---

Changing the title to reflect more accurately the situation. This has nothing to do with CNAMEs, but rather /etc/hosts entries (they are not the same thing).

CNAMEs continue to work fine.