Description of problem: yum repolist throws traceback after subscription-manager subscribe Version-Release number of selected component (if applicable): [root@jsefler-5 ~]# subscription-manager version server type: Red Hat Subscription Management subscription management server: 0.8.5-1 subscription-manager: 1.8.6-1.git.63.fbdeec8.el5 python-rhsm: 1.8.10-1.git.0.3f8551e.el5 [root@jsefler-f14-candlepin candlepin]# git show-ref | grep master 7307cfe64d5d936ee184fd6817051a35b793802e refs/heads/master [root@jsefler-5 ~]# rpm -q yum yum-3.2.22-40.el5 How reproducible: Steps to Reproduce: [root@jsefler-5 ~]# subscription-manager config --server.insecure=1 [root@jsefler-5 ~]# subscription-manager register --username testuser1 --org admin --serverurl=jsefler-f14-candlepin.usersys.redhat.com:8443/candlepin Password: The system has been registered with ID: 2a729a3a-a47d-41a4-9b0f-4c0c97c8a361 [root@jsefler-5 ~]# subscription-manager list --avail | grep "Pool ID" -B1 | tail -11 SKU: non-stacked-multiattr Pool ID: 8a90f8313e85026f013e8503deb6083e -- SKU: non-stacked-8core4ram-multiattr Pool ID: 8a90f8313e85026f013e8503df020857 -- SKU: awesomeos-modifier Pool ID: 8a90f8313e85026f013e8503cec20369 -- SKU: awesomeos-modifier Pool ID: 8a90f8313e85026f013e8503cdf40362 [root@jsefler-5 ~]# yum repolist Loaded plugins: product-id, rhnplugin, security, subscription-manager This system is registered to Red Hat Subscription Management, but is not receiving updates. You can use subscription-manager to assign subscriptions. This system is not registered with RHN Classic or RHN Satellite. You can use rhn_register to register. RHN Satellite or RHN Classic support will be disabled. repolist: 0 [root@jsefler-5 ~]# subscription-manager subscribe --pool 8a90f8313e85026f013e8503deb6083e Successfully attached a subscription for: Multi-Attribute (non-stackable) (24 cores, 6 sockets, 8GB RAM) [root@jsefler-5 ~]# yum repolist Loaded plugins: product-id, rhnplugin, security, subscription-manager This system is receiving updates from Red Hat Subscription Management. This system is not registered with RHN Classic or RHN Satellite. You can use rhn_register to register. RHN Satellite or RHN Classic support will be disabled. Traceback (most recent call last): File "/usr/bin/yum", line 29, in ? yummain.user_main(sys.argv[1:], exit_code=True) File "/usr/share/yum-cli/yummain.py", line 309, in user_main errcode = main(args) File "/usr/share/yum-cli/yummain.py", line 178, in main result, resultmsgs = base.doCommands() File "/usr/share/yum-cli/cli.py", line 349, in doCommands return self.yum_cli_commands[self.basecmd].doCommand(self, self.basecmd, self.extcmds) File "/usr/share/yum-cli/yumcommands.py", line 788, in doCommand base.repos.populateSack() File "/usr/lib/python2.4/site-packages/yum/repos.py", line 260, in populateSack sack.populate(repo, mdtype, callback, cacheonly) File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 168, in populate if self._check_db_version(repo, mydbtype): File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 226, in _check_db_version return repo._check_db_version(mdtype) File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1226, in _check_db_version repoXML = self.repoXML File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1399, in <lambda> repoXML = property(fget=lambda self: self._getRepoXML(), File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1391, in _getRepoXML self._loadRepoXML(text=self) File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1381, in _loadRepoXML return self._groupLoadRepoXML(text, ["primary"]) File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1365, in _groupLoadRepoXML if self._commonLoadRepoXML(text): File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1201, in _commonLoadRepoXML result = self._getFileRepoXML(local, text) File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 974, in _getFileRepoXML cache=self.http_caching == 'all') File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 811, in _getFile http_headers=headers, File "/usr/lib/python2.4/site-packages/urlgrabber/mirror.py", line 412, in urlgrab return self._mirror_try(func, url, kw) File "/usr/lib/python2.4/site-packages/urlgrabber/mirror.py", line 398, in _mirror_try return func_ref( *(fullurl,), **kwargs ) File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 936, in urlgrab return self._retry(opts, retryfunc, url, filename) File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 854, in _retry r = apply(func, (opts,) + args, {}) File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 922, in retryfunc fo = URLGrabberFileObject(url, filename, opts) File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 1010, in __init__ self._do_open() File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 1093, in _do_open fo, hdr = self._make_request(req, opener) File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 1202, in _make_request fo = opener.open(req) File "/usr/lib64/python2.4/urllib2.py", line 358, in open response = self._open(req, data) File "/usr/lib64/python2.4/urllib2.py", line 376, in _open '_open', req) File "/usr/lib64/python2.4/urllib2.py", line 337, in _call_chain result = func(*args) File "/usr/lib64/python2.4/site-packages/M2Crypto/m2urllib2.py", line 82, in https_open h.request(req.get_method(), req.get_selector(), req.data, headers) File "/usr/lib64/python2.4/httplib.py", line 810, in request self._send_request(method, url, body, headers) File "/usr/lib64/python2.4/httplib.py", line 833, in _send_request self.endheaders() File "/usr/lib64/python2.4/httplib.py", line 804, in endheaders self._send_output() File "/usr/lib64/python2.4/httplib.py", line 685, in _send_output self.send(msg) File "/usr/lib64/python2.4/httplib.py", line 652, in send self.connect() File "/usr/lib64/python2.4/site-packages/M2Crypto/httpslib.py", line 55, in connect sock.connect((self.host, self.port)) File "/usr/lib64/python2.4/site-packages/M2Crypto/SSL/Connection.py", line 174, in connect ret = self.connect_ssl() File "/usr/lib64/python2.4/site-packages/M2Crypto/SSL/Connection.py", line 167, in connect_ssl return m2.ssl_connect(self.ssl, self._timeout) M2Crypto.SSL.SSLError: certificate verify failed Additional info: Nothing was appended to rhsm.log
The more I look at this, the more I start to think this is doing the right thing (well, not the crashing part, but that is poor ssl handling on old yum's part...) On the machines I've seen exhibiting this, they are setup to use either cdn.redhat.com or cdn.rcm-qe.redhat.com as the rhsm.baseurl (ie, content server) But, the systems are registrating to a standalone candlepin with test data, and are subscribing to 'awesomeos' products, that all have fake content (ie, 'always-enabled-content') The rhsm.conf for those machines also configures rhsm.repo_ca_cert to a non default value, usually the ca cert for the standalone candlepin server. So instead of: repo_ca_cert = %(ca_cert_dir)sredhat-uep.pem They have repo_ca_cert = %(ca_cert_dir)scandlepin-ca.pem Or similar. When subscription-manager generates the repo defs, it includes rhsm.repo_ca_cert value into the yum 'sslcacert' repo config value (as well as setting 'sslclientkey'/'sslclientcert' to the entitlement certs). BUT.. the 'baseurl' is something like: https://cdn.redhat.com/foo/path/always/$releasever When yum tries to get the repo data for that guy, it tries to ssl_connect to cdn.redhat.com with the configured 'sslcacert', which in these cases IS NOT THE RIGHT CA CERT. And the client throws an ssl error indicating such. On RHEL6, that is something like: https://cdn.redhat.com/foo/path/always/6Server/repodata/repomd.xml: [Errno 14] Peer cert cannot be verified or peer cert invalid On RHEL5, it's the stack trace in description (yum has pretty terrible error handling for this case, so BOOM). WHY DOES IT WORK SOMETIMES? The default rhsm.repo_ca_cert is redhat-uep.pem. So the repo's will get created with that for their 'sslcacert'. redhat-urp.pem IS A VALID CA CERT FOR cdn.redhat.com. So no 'certificate verify failure' (and no error or stack trace for rhel5), and then THE SERVER rejects the clients entitlement certificate, and you get the 403 error that yum handles more gracefully. So my guess is, at some point QA machines started getting configured to specify the rhsm.repo_ca_cert, and for the default CDN or the QA cdn, that is the wrong CA cert. tl;dr default repo_ca_cert works for CDN, but specifying it fails as above.
Some observational differences between rhel5, rhel6, and the configured cdn which lead to different connection errors... All yum transactions appear gracefully correct except on RHEL5 against cdn.rcm-qa.redhat.com __________________________________________________________ RHEL6 w/ rhsm.baseurl=https://cdn.redhat.com [root@rhsm-compat-rhel64 ~]# subscription-manager config --rhsm.baseurl=https://cdn.redhat.com [root@rhsm-compat-rhel64 ~]# yum repolist Loaded plugins: product-id, refresh-packagekit, security, subscription-manager This system is receiving updates from Red Hat Subscription Management. https://cdn.redhat.com/foo/path/always/6Server/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403 Forbidden" Trying other mirror. repo id repo name status always-enabled-content always-enabled-content 0 awesomeos awesomeos 0 repolist: 0 [root@rhsm-compat-rhel64 ~]# __________________________________________________________ RHEL6 w/ rhsm.baseurl=https://cdn.rcm-qa.redhat.com [root@rhsm-compat-rhel64 ~]# subscription-manager config --rhsm.baseurl=https://cdn.rcm-qa.redhat.com [root@rhsm-compat-rhel64 ~]# yum repolist Loaded plugins: product-id, refresh-packagekit, security, subscription-manager This system is receiving updates from Red Hat Subscription Management. https://cdn.rcm-qa.redhat.com/foo/path/always/6Server/repodata/repomd.xml: [Errno 14] problem making ssl connection Trying other mirror. repo id repo name status always-enabled-content always-enabled-content 0 awesomeos awesomeos 0 repolist: 0 [root@rhsm-compat-rhel64 ~]# __________________________________________________________ RHEL5 w/ rhsm.baseurl=https://cdn.redhat.com [root@rhsm-compat-rhel59 ~]# subscription-manager config --rhsm.baseurl=https://cdn.redhat.com [root@rhsm-compat-rhel59 ~]# yum repolist Loaded plugins: product-id, security, subscription-manager This system is receiving updates from Red Hat Subscription Management. https://cdn.redhat.com/foo/path/always/5Server/repodata/repomd.xml: [Errno 14] HTTP Error 403: Forbidden Trying other mirror. repo id repo name status always-enabled-content always-enabled-content 0 awesomeos awesomeos 0 repolist: 0 [root@rhsm-compat-rhel59 ~]# __________________________________________________________ RHEL5 w/ rhsm.baseurl=https://cdn.rcm-qa.redhat.com [root@rhsm-compat-rhel59 ~]# subscription-manager config --rhsm.baseurl=https://cdn.rcm-qa.redhat.com [root@rhsm-compat-rhel59 ~]# yum repolist Loaded plugins: product-id, security, subscription-manager This system is receiving updates from Red Hat Subscription Management. Traceback (most recent call last): File "/usr/bin/yum", line 29, in ? yummain.user_main(sys.argv[1:], exit_code=True) File "/usr/share/yum-cli/yummain.py", line 309, in user_main errcode = main(args) File "/usr/share/yum-cli/yummain.py", line 178, in main result, resultmsgs = base.doCommands() File "/usr/share/yum-cli/cli.py", line 349, in doCommands return self.yum_cli_commands[self.basecmd].doCommand(self, self.basecmd, self.extcmds) File "/usr/share/yum-cli/yumcommands.py", line 788, in doCommand base.repos.populateSack() File "/usr/lib/python2.4/site-packages/yum/repos.py", line 260, in populateSack sack.populate(repo, mdtype, callback, cacheonly) File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 168, in populate if self._check_db_version(repo, mydbtype): File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 226, in _check_db_version return repo._check_db_version(mdtype) File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1226, in _check_db_version repoXML = self.repoXML File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1399, in <lambda> repoXML = property(fget=lambda self: self._getRepoXML(), File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1391, in _getRepoXML self._loadRepoXML(text=self) File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1381, in _loadRepoXML return self._groupLoadRepoXML(text, ["primary"]) File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1365, in _groupLoadRepoXML if self._commonLoadRepoXML(text): File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1201, in _commonLoadRepoXML result = self._getFileRepoXML(local, text) File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 974, in _getFileRepoXML cache=self.http_caching == 'all') File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 811, in _getFile http_headers=headers, File "/usr/lib/python2.4/site-packages/urlgrabber/mirror.py", line 412, in urlgrab return self._mirror_try(func, url, kw) File "/usr/lib/python2.4/site-packages/urlgrabber/mirror.py", line 398, in _mirror_try return func_ref( *(fullurl,), **kwargs ) File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 936, in urlgrab return self._retry(opts, retryfunc, url, filename) File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 854, in _retry r = apply(func, (opts,) + args, {}) File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 922, in retryfunc fo = URLGrabberFileObject(url, filename, opts) File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 1010, in __init__ self._do_open() File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 1093, in _do_open fo, hdr = self._make_request(req, opener) File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 1202, in _make_request fo = opener.open(req) File "/usr/lib64/python2.4/urllib2.py", line 358, in open response = self._open(req, data) File "/usr/lib64/python2.4/urllib2.py", line 376, in _open '_open', req) File "/usr/lib64/python2.4/urllib2.py", line 337, in _call_chain result = func(*args) File "/usr/lib64/python2.4/site-packages/M2Crypto/m2urllib2.py", line 82, in https_open h.request(req.get_method(), req.get_selector(), req.data, headers) File "/usr/lib64/python2.4/httplib.py", line 810, in request self._send_request(method, url, body, headers) File "/usr/lib64/python2.4/httplib.py", line 833, in _send_request self.endheaders() File "/usr/lib64/python2.4/httplib.py", line 804, in endheaders self._send_output() File "/usr/lib64/python2.4/httplib.py", line 685, in _send_output self.send(msg) File "/usr/lib64/python2.4/httplib.py", line 652, in send self.connect() File "/usr/lib64/python2.4/site-packages/M2Crypto/httpslib.py", line 55, in connect sock.connect((self.host, self.port)) File "/usr/lib64/python2.4/site-packages/M2Crypto/SSL/Connection.py", line 174, in connect ret = self.connect_ssl() File "/usr/lib64/python2.4/site-packages/M2Crypto/SSL/Connection.py", line 167, in connect_ssl return m2.ssl_connect(self.ssl, self._timeout) M2Crypto.SSL.SSLError: tlsv1 alert unknown ca
for comment #2, what's rhsm_ca_cert configured to?
(In reply to comment #3) > for comment #2, what's rhsm_ca_cert configured to? __________________________________________________________ RHEL6 [root@rhsm-compat-rhel64 ~]# grep ca_cert /etc/rhsm/rhsm.conf ca_cert_dir = /etc/rhsm/ca/ repo_ca_cert = %(ca_cert_dir)sredhat-uep.pem [root@rhsm-compat-rhel64 ~]# ls /etc/rhsm/ca/ candlepin-compat-rhel64.pem candlepin-stage.pem redhat-uep.pem __________________________________________________________ RHEL5 [root@rhsm-compat-rhel59 ~]# grep ca_cert /etc/rhsm/rhsm.conf ca_cert_dir = /etc/rhsm/ca/ repo_ca_cert = %(ca_cert_dir)sredhat-uep.pem [root@rhsm-compat-rhel59 ~]# ls /etc/rhsm/ca/ candlepin-compat-rhel59.pem candlepin-stage.pem redhat-uep.pem
The traceback observed in this bug comes from yum on rhel5. Since this behavior has improved in newer yum in rhel6, I'm going to close this bug. It does not belong to component python-rhsm.