Bug 1189184

Summary: RPM fetching via spacewalk proxy fails with 404 on 2.3/nightly
Product: [Community] Spacewalk Reporter: Patrick Hurrelmann <redhat>
Component: Proxy ServerAssignee: Stephen Herr <sherr>
Status: CLOSED CURRENTRELEASE QA Contact: Red Hat Satellite QA List <satqe-list>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 2.2   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: spacewalk-proxy-2.3.10-1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-04-14 19:03:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1207293    
Attachments:
Description Flags
rhn_proxy_broker.log none

Description Patrick Hurrelmann 2015-02-04 16:07:29 UTC
Created attachment 988165 [details]
rhn_proxy_broker.log

Description of problem:

Fetching RPMs (kickstart or yum install/update) fails with HTTP error 404, when system is registered to a spacewalk proxy. Spacewalk, proxy and client are all based on CentOS 7.0. Switching the client to use spacewalk fixes this.

Error downloading packages:
  systemd-libs-208-11.el7_0.6.x86_64: failed to retrieve getPackage/systemd-libs-208-11.el7_0.6.x86_64.rpm from centos7.0-updates-x86_64
error was [Errno 14] HTTPS Error 404 - Not Found
  kernel-3.10.0-123.20.1.el7.x86_64: failed to retrieve getPackage/kernel-3.10.0-123.20.1.el7.x86_64.rpm from centos7.0-updates-x86_64
error was [Errno 14] HTTPS Error 404 - Not Found
  1:openssl-libs-1.0.1e-34.el7_0.7.x86_64: failed to retrieve getPackage/openssl-libs-1.0.1e-34.el7_0.7.x86_64.rpm from centos7.0-updates-x86_64
error was [Errno 14] HTTPS Error 404 - Not Found
  glibc-2.17-55.el7_0.5.x86_64: failed to retrieve getPackage/glibc-2.17-55.el7_0.5.x86_64.rpm from centos7.0-updates-x86_64
error was [Errno 14] HTTPS Error 404 - Not Found
  jasper-libs-1.900.1-26.el7_0.3.x86_64: failed to retrieve getPackage/jasper-libs-1.900.1-26.el7_0.3.x86_64.rpm from centos7.0-updates-x86_64
error was [Errno 14] HTTPS Error 404 - Not Found
  systemd-208-11.el7_0.6.x86_64: failed to retrieve getPackage/systemd-208-11.el7_0.6.x86_64.rpm from centos7.0-updates-x86_64
error was [Errno 14] HTTPS Error 404 - Not Found
  glibc-common-2.17-55.el7_0.5.x86_64: failed to retrieve getPackage/glibc-common-2.17-55.el7_0.5.x86_64.rpm from centos7.0-updates-x86_64
error was [Errno 14] HTTPS Error 404 - Not Found
  libyaml-0.1.4-11.el7_0.x86_64: failed to retrieve getPackage/libyaml-0.1.4-11.el7_0.x86_64.rpm from centos7.0-updates-x86_64
error was [Errno 14] HTTPS Error 404 - Not Found
  kernel-tools-3.10.0-123.20.1.el7.x86_64: failed to retrieve getPackage/kernel-tools-3.10.0-123.20.1.el7.x86_64.rpm from centos7.0-updates-x86_64
error was [Errno 14] HTTPS Error 404 - Not Found
  httpd-2.4.6-19.el7.centos.x86_64: failed to retrieve getPackage/httpd-2.4.6-19.el7.centos.x86_64.rpm from centos7.0-updates-x86_64
error was [Errno 14] HTTPS Error 404 - Not Found
  kernel-tools-libs-3.10.0-123.20.1.el7.x86_64: failed to retrieve getPackage/kernel-tools-libs-3.10.0-123.20.1.el7.x86_64.rpm from centos7.0-updates-x86_64
error was [Errno 14] HTTPS Error 404 - Not Found
  1:openssl-1.0.1e-34.el7_0.7.x86_64: failed to retrieve getPackage/openssl-1.0.1e-34.el7_0.7.x86_64.rpm from centos7.0-updates-x86_64
error was [Errno 14] HTTPS Error 404 - Not Found
  1:mod_ssl-2.4.6-19.el7.centos.x86_64: failed to retrieve getPackage/mod_ssl-2.4.6-19.el7.centos.x86_64.rpm from centos7.0-updates-x86_64
error was [Errno 14] HTTPS Error 404 - Not Found
  1:mariadb-libs-5.5.40-2.el7_0.x86_64: failed to retrieve getPackage/mariadb-libs-5.5.40-2.el7_0.x86_64.rpm from centos7.0-updates-x86_64
error was [Errno 14] HTTPS Error 404 - Not Found
  systemd-sysv-208-11.el7_0.6.x86_64: failed to retrieve getPackage/systemd-sysv-208-11.el7_0.6.x86_64.rpm from centos7.0-updates-x86_64
error was [Errno 14] HTTPS Error 404 - Not Found
  httpd-tools-2.4.6-19.el7.centos.x86_64: failed to retrieve getPackage/httpd-tools-2.4.6-19.el7.centos.x86_64.rpm from centos7.0-updates-x86_64
error was [Errno 14] HTTPS Error 404 - Not Found
  libgudev1-208-11.el7_0.6.x86_64: failed to retrieve getPackage/libgudev1-208-11.el7_0.6.x86_64.rpm from centos7.0-updates-x86_64
error was [Errno 14] HTTPS Error 404 - Not Found


Version-Release number of selected component (if applicable):

Spacewalk 2.3/nightly on CentOS 7.0

How reproducible:

Always

Steps to Reproduce:
1. Install spacewalk
2. Create proxy
3. Register proxy to spacewalk
4. Change client to use proxy
5. Try to install or update packages via yum

Actual results:

RPMs can not be downloaded via proxy. Yum fails with 404 errors.

Expected results:

RPMs can be downloaded via proxy as done when connected directly to spacewalk

Additional info:

A full log file of rhn_proxy_broker when trying to update a bunch of packages is attached to this bug

Comment 1 Stephen Herr 2015-02-04 22:40:02 UTC
Squid 3.2 and later "helpfully" try to detect if a forwarding loop is in progress by looking at the 'via' header and comparing to server's hostname, and if a loop is detected it will 403 the request. A 403 for the channel-package list results in a 404 for the rpm get request from yum.

In Spacewalk Proxy's (weird) usage of squid, requests are /always/ going to be redirected to the same server that squid is running on if its a miss in squid's cache. The Redirect component of Proxy will pick up the request and forward it on to Satellite, with the appropriate header munging that indicates it's a valid request from an authenticated proxy.

So the upshot is that in Spacewalk Proxy we exclusively want what Squid detects and rejects as a forwarding loop. Since there isn't a config option to pass to squid to tell it not to do that, the only option is to just drop the 'via' header to mess with squid's loop detection.

This should be fine because none of the Spacewalk traffic cares about that header, and other uses of squid are unsupported on a Spacewalk Proxy server.

Committing to Spacewalk master:
b71f6742e4b6ca4dcb7f023602d66da231bbea00

Comment 2 Patrick Hurrelmann 2015-02-05 09:27:50 UTC
Thanks a lot Stephen. I applied the fix manually and can confirm that it works now. Can you please trigger new builds on koji and push them to the repo, too?

Comment 3 Stephen Herr 2015-02-05 13:44:52 UTC
Building now, should hit the nightly repos in an hour or two.

http://koji.spacewalkproject.org/koji/taskinfo?taskID=179916

Comment 4 Patrick Hurrelmann 2015-02-06 08:52:43 UTC
Build for el7 is still missing. It seems to have failed :/
http://koji.spacewalkproject.org/koji/buildinfo?buildID=39920

Comment 5 Grant Gainey 2015-03-23 16:59:11 UTC
Moving bugs to ON_QA as we move to release Spacewalk 2.3

Comment 6 Grant Gainey 2015-04-14 19:03:21 UTC
Spacewalk 2.3 has been released. See

https://fedorahosted.org/spacewalk/wiki/ReleaseNotes23