Bug 872721

Summary: proxy sending incorrect auth token to upstream (hosted/satellite)
Product: [Community] Spacewalk Reporter: Stephen Herr <sherr>
Component: ServerAssignee: Stephen Herr <sherr>
Status: CLOSED CURRENTRELEASE QA Contact: Red Hat Satellite QA List <satellite-qa-list>
Severity: high Docs Contact:
Priority: high    
Version: 1.9CC: aperkins, bperkins, cperry, jmontleo, marcobillpeter, shughes, xdmoon
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: spacewalk-proxy-1.9.1-1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 867544 Environment:
Last Closed: 2013-03-06 13:33:56 EST Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Bug Depends On: 867544, 878096    
Bug Blocks: 917805    

Description Stephen Herr 2012-11-02 16:03:07 EDT
+++ This bug was initially created as a clone of Bug #867544 +++

Description of problem:

rhn proxy 5.5 is connected to rhn hosted. after the x-rhn-auth-expire-offset expires a 2nd token is created but the headers continue to send the older token and client receives 404 for yum updates. 

[root@proxy proxy-auth]# pwd
/var/cache/rhn/proxy-auth
[root@proxy proxy-auth]# ll
total 12
-rw-r--r--. 1 apache apache 313 Oct 17 16:32 1023707104
-rw-r--r--. 1 apache apache  75 Oct 11 15:11 p10237051934b84b15bff6ee5796152495a230e45e3d7e947d9
-rw-r--r--. 1 apache apache  86 Oct 17 11:32 p1023705193786cdd299f613e013b848b1f533206191f79cb03

[root@proxy proxy-auth]# cat p10237051934b84b15bff6ee5796152495a230e45e3d7e947d9 | cut -f5 -d:
VSfX5ykdIFpMzfnR42c+Dw==

[root@proxy proxy-auth]# cat p1023705193786cdd299f613e013b848b1f533206191f79cb03 | cut -f5 -d:
28dcyr8FNBzmnDGILz56sA==

proxy/rhnShared._proxy2server('Outgoing header', 'X-RHN-Proxy-Auth', '1023705193::1349968279.16:21600.0:VSfX5ykdIFpMzfnR42c+Dw==:127.0.0.1')

workaround is to delete older token in a cron job. customers are using cron jobs to wipe out /var/cache/rhn/proxy-auth

We have one case where this is affecting satellite but I have yet been able to duplicate in house. 

My current proxy-> hosted reproducer is on my home external network for both the proxy and client. 

Version-Release number of selected component (if applicable):


How reproducible:

everytime on external network going through hosted but sometimes i have to wait 3600 minutes for the expire to take effect

--- Additional comment from shughes@redhat.com on 2012-10-17 14:03:34 EDT ---

so looks like the two different tokens is that one is for the ipaddress and one is for the hostname of the proxy: 

[root@proxy logs]# grep "token for this" rhn_proxy_broker.log 
2012/10/17 17:40:08 -04:00 3862 192.168.1.22: broker/rhnBroker.handler('Auth token for this machine only! 1023705193::1350473526.3:21600.0:28dcyr8FNBzmnDGILz56sA==:proxy.hugheshoney.com',)
2012/10/17 17:40:09 -04:00 3861 192.168.1.22: broker/rhnBroker.handler('Auth token for this machine only! 1023705193::1350495611.41:21600.0:/Hm7xFxJ0268yx1xj2nLqQ==:proxy.hugheshoney.com',)
2012/10/17 17:40:10 -04:00 3855 192.168.1.22: broker/rhnBroker.handler('Auth token for this machine only! 1023705193::1350495611.41:21600.0:/Hm7xFxJ0268yx1xj2nLqQ==:proxy.hugheshoney.com',)
2012/10/17 17:40:13 -04:00 3860 127.0.0.1: broker/rhnBroker.handler('Auth token for this machine only! 1023705193::1349968279.16:21600.0:VSfX5ykdIFpMzfnR42c+Dw==:127.0.0.1',)
2012/10/17 17:40:13 -04:00 3856 192.168.1.22: broker/rhnBroker.handler('Auth token for this machine only! 1023705193::1350495611.41:21600.0:/Hm7xFxJ0268yx1xj2nLqQ==:proxy.hugheshoney.com',)
2012/10/17 17:40:14 -04:00 3861 127.0.0.1: broker/rhnBroker.handler('Auth token for this machine only! 1023705193::1349968279.16:21600.0:VSfX5ykdIFpMzfnR42c+Dw==:127.0.0.1',)


what is interesting in the above yum update transaction is that we have two separate hashes for the hostname: 

1023705193::1350473526.3:21600.0:28dcyr8FNBzmnDGILz56sA==:proxy.hugheshoney.com
1023705193::1350495611.41:21600.0:/Hm7xFxJ0268yx1xj2nLqQ==:proxy.hugheshoney.com
Comment 1 Stephen Herr 2012-11-02 16:12:28 EDT
I cannot for the life of me figure out what has changed recently that causes this to break. There are several things that are very similar, but in each case closer inspection shows that they are not responsible for this error.

However, we do have a fix. I believe that the commit soon to follow completely prevents this from happening.

Note for QA: The only known reliable way to reproduce this error is to set up a client that is talking to a Proxy 5.5 that is talking to RHN Hosted, do a yum install of something on the client, wait 3 days or so, and then try to install something else on the client. It has been reported when the proxy is talking to Satellite too, but that is not 100% reproducible.
Comment 2 Stephen Herr 2012-11-02 16:16:10 EDT
Committed to Spacewalk master: 5f00038675c58c5a401d0c4a5cbfa332c28a4405
Comment 3 Shannon Hughes 2012-11-05 08:37:47 EST
just a note I am testing the fix with kickstarts. initial kickstart failed but looking into the error more closely. 

sherr, this is the configuration: 

client.hugheshoney.com -> proxy.hugheshoney.com -> hosted
Comment 4 Clifford Perry 2012-11-14 04:49:00 EST
On the Client system, yum will often return a 404 error, similar to the one below. Adding this comment, so that anyone doing BZ searches for client side error will hopefully find this BZ. 

Error Downloading Packages:
  yum-rhn-plugin-1.8.8-1.el6.noarch: failed to retrieve getPackage/yum-rhn-plugin-1.8.8-1.el6.noarch.rpm from centos-6-x86_64-spacewalk-client
error was [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404"

From spacewalk-list thread:

"fix in the following commit:
http://git.fedorahosted.org/cgit/spacewalk.git/commit/?id=5f00038675c58c5a401d0c4a5cbfa332c28a4405

After applying the patch locally my proxy is working as expected again.
Yum update is finally working again.
"

Cliff
Comment 5 Stephen Herr 2013-03-01 12:06:35 EST
Marking bug as ON_QA since tonight's build of Spacewalk nightly is a release candidate for Spacewalk 1.9.
Comment 6 Stephen Herr 2013-03-06 13:33:56 EST
Spacewalk 1.9 has been released.

https://fedorahosted.org/spacewalk/wiki/ReleaseNotes19