Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1172738

Summary:	"Package not in mapping" error if base channel contains no packages.
Product:	[Community] Spacewalk	Reporter:	Stephen Herr <sherr>
Component:	Proxy Server	Assignee:	Stephen Herr <sherr>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Red Hat Satellite QA List <satqe-list>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	2.2	CC:	redhat, rolf.linder
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	spacewalk-proxy-2.3.6-1	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-04-14 19:02:52 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1207293

Description Stephen Herr 2014-12-10 16:14:06 UTC

Description of problem:
Proxy assumes that packages referenced in kickstart distributions are available in the (base) channel they are associated with. If the base channel actually has no packages (or is missing a particular package) then the GET of the rpm from a Proxy will fail with "Package not in mapping" error.

Version-Release number of selected component (if applicable):
2.2

How reproducible:
always

Steps to Reproduce:
1. Create kickstart distribution, associate it with an empty base channel
2. Attempt to kickstart a machine

Actual results:
kicksarting directly against Spacewalk succeeds, kickstarting through a proxy fails.

Expected results:
Should work both ways.

Additional info:
This is related to the changes made to enable the proxy pre-caching feature in 2.2. In particular, in order for Proxy to correctly cache rpms permanently we need more information than is available in the GET request, specifically the checksum of the file, as multiple rpms with the same name / version / release but different checksums can coexist in different channels in Spacewalk. Proxy therefore has to maintain a mapping of package-channel to checksum so that it can retrieve the correct file. 

The problem here appears to be that Proxy always assumes that the package is available in the base channel that is associated with the kickstart distribution, where in some cases the package may actually be in a child channel instead.

--- Copied from spacewalk-list from Patrick Hurrelmann's email ---

I tried to debug this some more and I can confirm the findings of Rolf.
I too use a master channel like centos-6x or centos-7x as parents with
no packages assigned.

Kickstarting using this channels fails with the mentioned "package not
in mapping error". See log at http://ur1.ca/j1pu7

In this scenario the channels are setup the following:
centos6.x-base
 --> centos6.6-x86_64
 --> centos6.6-updates-x86_64

Kickstart tries to download the following rpm:

broker/rhnBroker._initConnectionVariables('set self.rhnParent:
http://proxy.example.com/ks/dist/centos6.6-x86_64/Packages/python-ethtool-0.6-5.el6.x86_64.rpm',)

and tries to fetch locally first (before going up in the chain):
broker/rhnBroker.__local_GET_handler('Retrieve from local repository.',)

and then it tries to cache the rpm using the parent channel:

broker/rhnRepository.getPackagePath('python-ethtool-0.6-5.el6.x86_64.rpm',)
broker/rhnRepository._cacheObj('package_mapping:centos6x-base:',
'20130308120245', ())
broker/rhnRepository._getPkgListDir('/var/spool/rhn-proxy/list',)
broker/rhnRepository.getPackagePath('ERROR', 'Package not in mapping:
python-ethtool-0.6-5.el6.x86_64.rpm')

Imho there is no fallback to the lower channel (which contains rpms).
The same is happening for centos7 kickstarts due to the same channel layout.

For testing I cloned my centos-6.6-x86_64 channel and made it a
standalone top-level channel and kickstarting is working again.

Comment 1 Stephen Herr 2014-12-10 16:32:05 UTC

Thanks to Rolf Linder and Michael Guidero also for confirming the cause of this problem

Comment 2 Stephen Herr 2014-12-10 21:27:59 UTC

Can you try a test for me?

In /usr/share/rhn/proxy/broker/rhnRepository.py line 100 (right below where we log the "Package not in mapping" error) should look like this:

raise rhnFault(17, _("Invalid RPM package requested: %s")
        % pkgFilename)

Please try changing that to:

raise NotLocalError(filePaths[0], pkgFilename)

and then "rhn-proxy restart" and try the kickstart again. This should cause us to just fail back to requesting the rpm from Spacewalk (which should still work). Proxy-pre-caching is not compatible with kickstarting with your weird channel setup, but this should still allow things to function like you're used to.

Comment 3 Patrick Hurrelmann 2014-12-10 23:28:09 UTC

Ok, tried that. But it didn't change much:

2014/12/11 00:01:29 +02:00 29855 1.2.3.4: proxy/apacheHandler.handler('METHOD', 'GET')
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: proxy/apacheHandler.handler('PATH_INFO', '/dist/centos6.6-x86_64/Packages/python-ethtool-0.6-5.el6.x86_64.rpm')
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: proxy/apacheHandler.handler('URI (full path info)', '/ks/dist/centos6.6-x86_64/Packages/python-ethtool-0.6-5.el6.x86_64.rpm')
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: proxy/apacheHandler.handler('Component', 'proxy.broker')
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: proxy/rhnProxyAuth.__init__
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnBroker._initConnectionVariables('set self.rhnParent:       http://proxy.example.com/ks/dist/centos6.6-x86_64/Packages/python-ethtool-0.6-5.el6.x86_64.rpm',)
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnBroker._initConnectionVariables('set self.rhnParentXMLRPC: http://proxy.example.com/XMLRPC',)
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnBroker._initConnectionVariables('using self.httpProxy:     127.0.0.1:8080 (non-authenticating)',)
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnBroker.handler
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: proxy/rhnShared._prepHandler({'outputtransportoptions': {'connection': 'close', 'x-rhn-proxy-version': '0', 'x-rhn-client-version': '0'}, 'auth_session_token': {}},)
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnBroker.handler('X-RHN-IP-Path is: None',)
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnBroker.handler('X-RHN-Proxy-Auth is not set',)
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnBroker.__local_GET_handler('request method: GET',)
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnBroker.__local_GET_handler('Retrieve from local repository.',)
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnBroker.__local_GET_handler('ks-dist', 'centos6.6-x86_64', 'getPackage', {'session': None, 'orgId': None, 'file': 'python-ethtool-0.6-5.el6.x86_64.rpm', 'child': None}
)
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnBroker.__callLocalRepository('ks-dist', 'centos6.6-x86_64', 'getPackage', {'session': None, 'orgId': None, 'file': 'python-ethtool-0.6-5.el6.x86_64.rpm', 'child': Non
e})
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnRepository.__init__('centos6.6-x86_64',)
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnRepository._getPkgListDir('/var/spool/rhn-proxy/list',)
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnRepository.__init__('centos6x-base',)
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: common/rhnRepository.__init__('centos6x-base',)
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnBroker.__callLocalRepository("Calling getPackage(['python-ethtool-0.6-5.el6.x86_64.rpm'])",)
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: common/rhnRepository.getPackage('python-ethtool-0.6-5.el6.x86_64.rpm',)
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnRepository.getPackagePath('python-ethtool-0.6-5.el6.x86_64.rpm',)
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnRepository._cacheObj('package_mapping:centos6x-base:', '20130308120245', ())
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnRepository._getPkgListDir('/var/spool/rhn-proxy/list',)
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnRepository.getPackagePath('ERROR', 'Package not in mapping: python-ethtool-0.6-5.el6.x86_64.rpm')
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: common/rhnApache.logHandler
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: proxy/apacheHandler.cleanupHandler
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: common/rhnApache.cleanupHandler
2014/12/11 00:01:29 +02:00 29855 1.2.3.4: common/rhnApache.timer('Request served in 0.10 sec',)


I'm happy to test more, but I won't be able to do this until next week.

Comment 4 Rolf Linder 2014-12-11 08:36:48 UTC

Same here, here's what we did:

[root@spacewalkproxy ~]# diff /usr/share/rhn/proxy/broker/rhnRepository.py /usr/share/rhn/proxy/broker/rhnRepository.py.orig
98c98,99
<             raise NotLocalError(filePaths[0], pkgFilename)
---
>             raise rhnFault(17, _("Invalid RPM package requested: %s")
>                                  % pkgFilename)
[root@spacewalkproxy ~]# diff /usr/share/rhn/proxy/broker/rhnRepository.py.orig /usr/share/rhn/proxy/broker/rhnRepository.py
98,99c98
<             raise rhnFault(17, _("Invalid RPM package requested: %s")
<                                  % pkgFilename)
---
>             raise NotLocalError(filePaths[0], pkgFilename)
[root@spacewalkproxy ~]# rpm -q $(rpm -qf /usr/share/rhn/proxy/broker/rhnRepository.py)
spacewalk-proxy-broker-2.2.10-1.el6.noarch
[root@spacewalkproxy ~]# rhn-proxy restart

and then we still see in httpd-access-log (on proxy):
127.0.0.1 - - [11/Dec/2014:09:33:31 +0100] "HEAD http://127.0.0.1:80/ty/hakJP1tN/Packages/iputils-20071127-17.el6_4.2.x86_64.rpm HTTP/1.1" 200 - "-" "CentOS (anaconda)/6.6"
10.0.2.253 - - [11/Dec/2014:09:33:31 +0100] "GET /ty/hakJP1tN/Packages/iputils-20071127-17.el6_4.2.x86_64.rpm HTTP/1.1" 500 627 "-" "CentOS (anaconda)/6.6"

which results in file is missing on the client (10.0.2.253) kickstarting...

Thanks Stephen for your investigations!

Comment 5 Stephen Herr 2014-12-11 20:11:54 UTC

Oops, my mistake. I expect that if you look in /var/log/httpd/error_log you will see that the problem has changed to:
UnboundLocalError: local variable 'filePaths' referenced before assignment

This would be because filePaths[0] is not defined when I referenced it on that changed line.

I have managed to set up a reproducer and tested myself, and the change now has the desired effect.

Committing to Spacewalk master:
279ca344936265a56c79384e188cdfd02d4dd698

The diff if you want to change this by hand on your existing proxies:
https://github.com/spacewalkproject/spacewalk/commit/279ca344936265a56c79384e188cdfd02d4dd698

Let me know if you have trouble.

Comment 6 Stephen Herr 2014-12-11 20:30:22 UTC

Another commit to make checkstyle happy:
f53ce5e5b2e3553fc46f6de45e934c157657140b

Comment 7 Patrick Hurrelmann 2014-12-15 08:59:07 UTC

I can confirm the fix. Kickstarting is back life.

Log excerpt from a running kickstart:

broker/rhnBroker.handler('X-RHN-IP-Path is: None',)
broker/rhnBroker.handler('X-RHN-Proxy-Auth is not set',)
broker/rhnBroker.__local_GET_handler('request method: GET',)
broker/rhnBroker.__local_GET_handler('Retrieve from local repository.',)
broker/rhnBroker.__local_GET_handler('ks-dist', 'centos6.6-x86_64', 'getPackage', {'session': None, 'orgId': None, 'file': 'perl-5.10.1-136.el6.x86_64.rpm', 'child': None})
broker/rhnBroker.__callLocalRepository('ks-dist', 'centos6.6-x86_64', 'getPackage', {'session': None, 'orgId': None, 'file': 'perl-5.10.1-136.el6.x86_64.rpm', 'child': None})
broker/rhnRepository.__init__('centos6.6-x86_64',)
broker/rhnRepository._getPkgListDir('/var/spool/rhn-proxy/list',)
broker/rhnRepository.__init__('centos6x-base',)
common/rhnRepository.__init__('centos6x-base',)
broker/rhnBroker.__callLocalRepository("Calling getPackage(['perl-5.10.1-136.el6.x86_64.rpm'])",)
common/rhnRepository.getPackage('perl-5.10.1-136.el6.x86_64.rpm',)
broker/rhnRepository.getPackagePath('perl-5.10.1-136.el6.x86_64.rpm',)
broker/rhnRepository._cacheObj('package_mapping:centos6x-base:', '20130308120245', ())
broker/rhnRepository._getPkgListDir('/var/spool/rhn-proxy/list',)
broker/rhnRepository.getPackagePath('Package not in mapping: perl-5.10.1-136.el6.x86_64.rpm',)

Comment 8 Grant Gainey 2015-03-23 16:58:53 UTC

Moving bugs to ON_QA as we move to release Spacewalk 2.3

Comment 9 Grant Gainey 2015-04-14 19:02:52 UTC

Spacewalk 2.3 has been released. See

https://fedorahosted.org/spacewalk/wiki/ReleaseNotes23