Description of problem: Proxy assumes that packages referenced in kickstart distributions are available in the (base) channel they are associated with. If the base channel actually has no packages (or is missing a particular package) then the GET of the rpm from a Proxy will fail with "Package not in mapping" error. Version-Release number of selected component (if applicable): 2.2 How reproducible: always Steps to Reproduce: 1. Create kickstart distribution, associate it with an empty base channel 2. Attempt to kickstart a machine Actual results: kicksarting directly against Spacewalk succeeds, kickstarting through a proxy fails. Expected results: Should work both ways. Additional info: This is related to the changes made to enable the proxy pre-caching feature in 2.2. In particular, in order for Proxy to correctly cache rpms permanently we need more information than is available in the GET request, specifically the checksum of the file, as multiple rpms with the same name / version / release but different checksums can coexist in different channels in Spacewalk. Proxy therefore has to maintain a mapping of package-channel to checksum so that it can retrieve the correct file. The problem here appears to be that Proxy always assumes that the package is available in the base channel that is associated with the kickstart distribution, where in some cases the package may actually be in a child channel instead. --- Copied from spacewalk-list from Patrick Hurrelmann's email --- I tried to debug this some more and I can confirm the findings of Rolf. I too use a master channel like centos-6x or centos-7x as parents with no packages assigned. Kickstarting using this channels fails with the mentioned "package not in mapping error". See log at http://ur1.ca/j1pu7 In this scenario the channels are setup the following: centos6.x-base --> centos6.6-x86_64 --> centos6.6-updates-x86_64 Kickstart tries to download the following rpm: broker/rhnBroker._initConnectionVariables('set self.rhnParent: http://proxy.example.com/ks/dist/centos6.6-x86_64/Packages/python-ethtool-0.6-5.el6.x86_64.rpm',) and tries to fetch locally first (before going up in the chain): broker/rhnBroker.__local_GET_handler('Retrieve from local repository.',) and then it tries to cache the rpm using the parent channel: broker/rhnRepository.getPackagePath('python-ethtool-0.6-5.el6.x86_64.rpm',) broker/rhnRepository._cacheObj('package_mapping:centos6x-base:', '20130308120245', ()) broker/rhnRepository._getPkgListDir('/var/spool/rhn-proxy/list',) broker/rhnRepository.getPackagePath('ERROR', 'Package not in mapping: python-ethtool-0.6-5.el6.x86_64.rpm') Imho there is no fallback to the lower channel (which contains rpms). The same is happening for centos7 kickstarts due to the same channel layout. For testing I cloned my centos-6.6-x86_64 channel and made it a standalone top-level channel and kickstarting is working again.
Thanks to Rolf Linder and Michael Guidero also for confirming the cause of this problem
Can you try a test for me? In /usr/share/rhn/proxy/broker/rhnRepository.py line 100 (right below where we log the "Package not in mapping" error) should look like this: raise rhnFault(17, _("Invalid RPM package requested: %s") % pkgFilename) Please try changing that to: raise NotLocalError(filePaths[0], pkgFilename) and then "rhn-proxy restart" and try the kickstart again. This should cause us to just fail back to requesting the rpm from Spacewalk (which should still work). Proxy-pre-caching is not compatible with kickstarting with your weird channel setup, but this should still allow things to function like you're used to.
Ok, tried that. But it didn't change much: 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: proxy/apacheHandler.handler('METHOD', 'GET') 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: proxy/apacheHandler.handler('PATH_INFO', '/dist/centos6.6-x86_64/Packages/python-ethtool-0.6-5.el6.x86_64.rpm') 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: proxy/apacheHandler.handler('URI (full path info)', '/ks/dist/centos6.6-x86_64/Packages/python-ethtool-0.6-5.el6.x86_64.rpm') 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: proxy/apacheHandler.handler('Component', 'proxy.broker') 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: proxy/rhnProxyAuth.__init__ 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnBroker._initConnectionVariables('set self.rhnParent: http://proxy.example.com/ks/dist/centos6.6-x86_64/Packages/python-ethtool-0.6-5.el6.x86_64.rpm',) 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnBroker._initConnectionVariables('set self.rhnParentXMLRPC: http://proxy.example.com/XMLRPC',) 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnBroker._initConnectionVariables('using self.httpProxy: 127.0.0.1:8080 (non-authenticating)',) 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnBroker.handler 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: proxy/rhnShared._prepHandler({'outputtransportoptions': {'connection': 'close', 'x-rhn-proxy-version': '0', 'x-rhn-client-version': '0'}, 'auth_session_token': {}},) 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnBroker.handler('X-RHN-IP-Path is: None',) 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnBroker.handler('X-RHN-Proxy-Auth is not set',) 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnBroker.__local_GET_handler('request method: GET',) 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnBroker.__local_GET_handler('Retrieve from local repository.',) 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnBroker.__local_GET_handler('ks-dist', 'centos6.6-x86_64', 'getPackage', {'session': None, 'orgId': None, 'file': 'python-ethtool-0.6-5.el6.x86_64.rpm', 'child': None} ) 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnBroker.__callLocalRepository('ks-dist', 'centos6.6-x86_64', 'getPackage', {'session': None, 'orgId': None, 'file': 'python-ethtool-0.6-5.el6.x86_64.rpm', 'child': Non e}) 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnRepository.__init__('centos6.6-x86_64',) 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnRepository._getPkgListDir('/var/spool/rhn-proxy/list',) 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnRepository.__init__('centos6x-base',) 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: common/rhnRepository.__init__('centos6x-base',) 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnBroker.__callLocalRepository("Calling getPackage(['python-ethtool-0.6-5.el6.x86_64.rpm'])",) 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: common/rhnRepository.getPackage('python-ethtool-0.6-5.el6.x86_64.rpm',) 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnRepository.getPackagePath('python-ethtool-0.6-5.el6.x86_64.rpm',) 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnRepository._cacheObj('package_mapping:centos6x-base:', '20130308120245', ()) 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnRepository._getPkgListDir('/var/spool/rhn-proxy/list',) 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: broker/rhnRepository.getPackagePath('ERROR', 'Package not in mapping: python-ethtool-0.6-5.el6.x86_64.rpm') 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: common/rhnApache.logHandler 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: proxy/apacheHandler.cleanupHandler 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: common/rhnApache.cleanupHandler 2014/12/11 00:01:29 +02:00 29855 1.2.3.4: common/rhnApache.timer('Request served in 0.10 sec',) I'm happy to test more, but I won't be able to do this until next week.
Same here, here's what we did: [root@spacewalkproxy ~]# diff /usr/share/rhn/proxy/broker/rhnRepository.py /usr/share/rhn/proxy/broker/rhnRepository.py.orig 98c98,99 < raise NotLocalError(filePaths[0], pkgFilename) --- > raise rhnFault(17, _("Invalid RPM package requested: %s") > % pkgFilename) [root@spacewalkproxy ~]# diff /usr/share/rhn/proxy/broker/rhnRepository.py.orig /usr/share/rhn/proxy/broker/rhnRepository.py 98,99c98 < raise rhnFault(17, _("Invalid RPM package requested: %s") < % pkgFilename) --- > raise NotLocalError(filePaths[0], pkgFilename) [root@spacewalkproxy ~]# rpm -q $(rpm -qf /usr/share/rhn/proxy/broker/rhnRepository.py) spacewalk-proxy-broker-2.2.10-1.el6.noarch [root@spacewalkproxy ~]# rhn-proxy restart and then we still see in httpd-access-log (on proxy): 127.0.0.1 - - [11/Dec/2014:09:33:31 +0100] "HEAD http://127.0.0.1:80/ty/hakJP1tN/Packages/iputils-20071127-17.el6_4.2.x86_64.rpm HTTP/1.1" 200 - "-" "CentOS (anaconda)/6.6" 10.0.2.253 - - [11/Dec/2014:09:33:31 +0100] "GET /ty/hakJP1tN/Packages/iputils-20071127-17.el6_4.2.x86_64.rpm HTTP/1.1" 500 627 "-" "CentOS (anaconda)/6.6" which results in file is missing on the client (10.0.2.253) kickstarting... Thanks Stephen for your investigations!
Oops, my mistake. I expect that if you look in /var/log/httpd/error_log you will see that the problem has changed to: UnboundLocalError: local variable 'filePaths' referenced before assignment This would be because filePaths[0] is not defined when I referenced it on that changed line. I have managed to set up a reproducer and tested myself, and the change now has the desired effect. Committing to Spacewalk master: 279ca344936265a56c79384e188cdfd02d4dd698 The diff if you want to change this by hand on your existing proxies: https://github.com/spacewalkproject/spacewalk/commit/279ca344936265a56c79384e188cdfd02d4dd698 Let me know if you have trouble.
Another commit to make checkstyle happy: f53ce5e5b2e3553fc46f6de45e934c157657140b
I can confirm the fix. Kickstarting is back life. Log excerpt from a running kickstart: broker/rhnBroker.handler('X-RHN-IP-Path is: None',) broker/rhnBroker.handler('X-RHN-Proxy-Auth is not set',) broker/rhnBroker.__local_GET_handler('request method: GET',) broker/rhnBroker.__local_GET_handler('Retrieve from local repository.',) broker/rhnBroker.__local_GET_handler('ks-dist', 'centos6.6-x86_64', 'getPackage', {'session': None, 'orgId': None, 'file': 'perl-5.10.1-136.el6.x86_64.rpm', 'child': None}) broker/rhnBroker.__callLocalRepository('ks-dist', 'centos6.6-x86_64', 'getPackage', {'session': None, 'orgId': None, 'file': 'perl-5.10.1-136.el6.x86_64.rpm', 'child': None}) broker/rhnRepository.__init__('centos6.6-x86_64',) broker/rhnRepository._getPkgListDir('/var/spool/rhn-proxy/list',) broker/rhnRepository.__init__('centos6x-base',) common/rhnRepository.__init__('centos6x-base',) broker/rhnBroker.__callLocalRepository("Calling getPackage(['perl-5.10.1-136.el6.x86_64.rpm'])",) common/rhnRepository.getPackage('perl-5.10.1-136.el6.x86_64.rpm',) broker/rhnRepository.getPackagePath('perl-5.10.1-136.el6.x86_64.rpm',) broker/rhnRepository._cacheObj('package_mapping:centos6x-base:', '20130308120245', ()) broker/rhnRepository._getPkgListDir('/var/spool/rhn-proxy/list',) broker/rhnRepository.getPackagePath('Package not in mapping: perl-5.10.1-136.el6.x86_64.rpm',)
Moving bugs to ON_QA as we move to release Spacewalk 2.3
Spacewalk 2.3 has been released. See https://fedorahosted.org/spacewalk/wiki/ReleaseNotes23