Bug 961152 - M2Crypto.SSL.SSLError: certificate verify failed
M2Crypto.SSL.SSLError: certificate verify failed
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: python-rhsm (Show other bugs)
5.10
Unspecified Unspecified
unspecified Severity urgent
: rc
: ---
Assigned To: candlepin-bugs
IDM QE LIST
:
Depends On:
Blocks: rhsm-rhel510
  Show dependency treegraph
 
Reported: 2013-05-08 18:52 EDT by John Sefler
Modified: 2014-12-02 15:47 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-05-22 10:11:04 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description John Sefler 2013-05-08 18:52:07 EDT
Description of problem:
yum repolist throws traceback after subscription-manager subscribe


Version-Release number of selected component (if applicable):
[root@jsefler-5 ~]# subscription-manager version
server type: Red Hat Subscription Management
subscription management server: 0.8.5-1
subscription-manager: 1.8.6-1.git.63.fbdeec8.el5
python-rhsm: 1.8.10-1.git.0.3f8551e.el5

[root@jsefler-f14-candlepin candlepin]# git show-ref | grep master
7307cfe64d5d936ee184fd6817051a35b793802e refs/heads/master

[root@jsefler-5 ~]# rpm -q yum
yum-3.2.22-40.el5


How reproducible:



Steps to Reproduce:
[root@jsefler-5 ~]# subscription-manager config --server.insecure=1

[root@jsefler-5 ~]# subscription-manager register --username testuser1 --org admin --serverurl=jsefler-f14-candlepin.usersys.redhat.com:8443/candlepin
Password: 
The system has been registered with ID: 2a729a3a-a47d-41a4-9b0f-4c0c97c8a361 

[root@jsefler-5 ~]# subscription-manager list --avail | grep "Pool ID" -B1 | tail -11
SKU:               non-stacked-multiattr
Pool ID:           8a90f8313e85026f013e8503deb6083e
--
SKU:               non-stacked-8core4ram-multiattr
Pool ID:           8a90f8313e85026f013e8503df020857
--
SKU:               awesomeos-modifier
Pool ID:           8a90f8313e85026f013e8503cec20369
--
SKU:               awesomeos-modifier
Pool ID:           8a90f8313e85026f013e8503cdf40362

[root@jsefler-5 ~]# yum repolist
Loaded plugins: product-id, rhnplugin, security, subscription-manager
This system is registered to Red Hat Subscription Management, but is not receiving updates. You can use subscription-manager to assign subscriptions.
This system is not registered with RHN Classic or RHN Satellite.
You can use rhn_register to register.
RHN Satellite or RHN Classic support will be disabled.
repolist: 0

[root@jsefler-5 ~]# subscription-manager subscribe --pool 8a90f8313e85026f013e8503deb6083e
Successfully attached a subscription for: Multi-Attribute (non-stackable) (24 cores, 6 sockets, 8GB RAM)

[root@jsefler-5 ~]# yum repolist
Loaded plugins: product-id, rhnplugin, security, subscription-manager
This system is receiving updates from Red Hat Subscription Management.
This system is not registered with RHN Classic or RHN Satellite.
You can use rhn_register to register.
RHN Satellite or RHN Classic support will be disabled.
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in ?
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 309, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 178, in main
    result, resultmsgs = base.doCommands()
  File "/usr/share/yum-cli/cli.py", line 349, in doCommands
    return self.yum_cli_commands[self.basecmd].doCommand(self, self.basecmd, self.extcmds)
  File "/usr/share/yum-cli/yumcommands.py", line 788, in doCommand
    base.repos.populateSack()
  File "/usr/lib/python2.4/site-packages/yum/repos.py", line 260, in populateSack
    sack.populate(repo, mdtype, callback, cacheonly)
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 168, in populate
    if self._check_db_version(repo, mydbtype):
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 226, in _check_db_version
    return repo._check_db_version(mdtype)
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1226, in _check_db_version
    repoXML = self.repoXML
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1399, in <lambda>
    repoXML = property(fget=lambda self: self._getRepoXML(),
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1391, in _getRepoXML
    self._loadRepoXML(text=self)
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1381, in _loadRepoXML
    return self._groupLoadRepoXML(text, ["primary"])
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1365, in _groupLoadRepoXML
    if self._commonLoadRepoXML(text):
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1201, in _commonLoadRepoXML
    result = self._getFileRepoXML(local, text)
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 974, in _getFileRepoXML
    cache=self.http_caching == 'all')
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 811, in _getFile
    http_headers=headers,
  File "/usr/lib/python2.4/site-packages/urlgrabber/mirror.py", line 412, in urlgrab
    return self._mirror_try(func, url, kw)
  File "/usr/lib/python2.4/site-packages/urlgrabber/mirror.py", line 398, in _mirror_try
    return func_ref( *(fullurl,), **kwargs )
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 936, in urlgrab
    return self._retry(opts, retryfunc, url, filename)
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 854, in _retry
    r = apply(func, (opts,) + args, {})
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 922, in retryfunc
    fo = URLGrabberFileObject(url, filename, opts)
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 1010, in __init__
    self._do_open()
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 1093, in _do_open
    fo, hdr = self._make_request(req, opener)
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 1202, in _make_request
    fo = opener.open(req)
  File "/usr/lib64/python2.4/urllib2.py", line 358, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.4/urllib2.py", line 376, in _open
    '_open', req)
  File "/usr/lib64/python2.4/urllib2.py", line 337, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.4/site-packages/M2Crypto/m2urllib2.py", line 82, in https_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/lib64/python2.4/httplib.py", line 810, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.4/httplib.py", line 833, in _send_request
    self.endheaders()
  File "/usr/lib64/python2.4/httplib.py", line 804, in endheaders
    self._send_output()
  File "/usr/lib64/python2.4/httplib.py", line 685, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.4/httplib.py", line 652, in send
    self.connect()
  File "/usr/lib64/python2.4/site-packages/M2Crypto/httpslib.py", line 55, in connect
    sock.connect((self.host, self.port))
  File "/usr/lib64/python2.4/site-packages/M2Crypto/SSL/Connection.py", line 174, in connect
    ret = self.connect_ssl()
  File "/usr/lib64/python2.4/site-packages/M2Crypto/SSL/Connection.py", line 167, in connect_ssl
    return m2.ssl_connect(self.ssl, self._timeout)
M2Crypto.SSL.SSLError: certificate verify failed

  

Additional info:
Nothing was appended to rhsm.log
Comment 1 Adrian Likins 2013-05-14 12:46:35 EDT
The more I look at this, the more I start to think this is doing the right thing
(well, not the crashing part, but that is poor ssl handling on old yum's part...)

On the machines I've seen exhibiting this, they are setup to use either
cdn.redhat.com or cdn.rcm-qe.redhat.com as the rhsm.baseurl (ie, content server)

But, the systems are registrating to a standalone candlepin with test data, and
are subscribing to 'awesomeos' products, that all have fake content (ie, 'always-enabled-content')

The rhsm.conf for those machines also configures rhsm.repo_ca_cert to a non default value, usually the ca cert for the standalone candlepin server.

So instead of:

 repo_ca_cert = %(ca_cert_dir)sredhat-uep.pem

They have

 repo_ca_cert = %(ca_cert_dir)scandlepin-ca.pem

Or similar. When subscription-manager generates the repo defs, it includes
rhsm.repo_ca_cert value into the yum 'sslcacert' repo config value (as well
as setting 'sslclientkey'/'sslclientcert' to the entitlement certs). 

BUT.. the 'baseurl' is something like: 

  https://cdn.redhat.com/foo/path/always/$releasever


When yum tries to get the repo data for that guy, it tries to ssl_connect to 
cdn.redhat.com with the configured 'sslcacert', which in these cases
IS NOT THE RIGHT CA CERT. And the client throws an ssl error indicating
such.

On RHEL6, that is something like:

        https://cdn.redhat.com/foo/path/always/6Server/repodata/repomd.xml: [Errno 14] Peer cert cannot be verified or peer cert invalid


On RHEL5, it's the stack trace in description (yum has pretty terrible error handling for this case, so BOOM). 


WHY DOES IT WORK SOMETIMES?

The default rhsm.repo_ca_cert is redhat-uep.pem. So the repo's will get created with that for their 'sslcacert'. redhat-urp.pem IS A VALID CA CERT FOR cdn.redhat.com. So no 'certificate verify failure' (and no error or stack trace
for rhel5), and then THE SERVER rejects the clients entitlement certificate, and you get the 403 error that yum handles more gracefully.


So my guess is, at some point QA machines started getting configured to specify the rhsm.repo_ca_cert, and for the default CDN or the QA cdn, that is the wrong CA cert. 

tl;dr default repo_ca_cert works for CDN, but specifying it fails as above.
Comment 2 John Sefler 2013-05-14 17:10:23 EDT
Some observational differences between rhel5, rhel6, and the configured cdn which lead to different connection errors...  All yum transactions appear gracefully correct except on RHEL5 against cdn.rcm-qa.redhat.com


__________________________________________________________
RHEL6 w/ rhsm.baseurl=https://cdn.redhat.com

[root@rhsm-compat-rhel64 ~]# subscription-manager config --rhsm.baseurl=https://cdn.redhat.com
[root@rhsm-compat-rhel64 ~]# yum repolist
Loaded plugins: product-id, refresh-packagekit, security, subscription-manager
This system is receiving updates from Red Hat Subscription Management.
https://cdn.redhat.com/foo/path/always/6Server/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403 Forbidden"
Trying other mirror.
repo id                                      repo name                                     status
always-enabled-content                       always-enabled-content                        0
awesomeos                                    awesomeos                                     0
repolist: 0
[root@rhsm-compat-rhel64 ~]# 


__________________________________________________________
RHEL6 w/ rhsm.baseurl=https://cdn.rcm-qa.redhat.com

[root@rhsm-compat-rhel64 ~]# subscription-manager config --rhsm.baseurl=https://cdn.rcm-qa.redhat.com
[root@rhsm-compat-rhel64 ~]# yum repolist
Loaded plugins: product-id, refresh-packagekit, security, subscription-manager
This system is receiving updates from Red Hat Subscription Management.
https://cdn.rcm-qa.redhat.com/foo/path/always/6Server/repodata/repomd.xml: [Errno 14] problem making ssl connection
Trying other mirror.
repo id                                      repo name                                     status
always-enabled-content                       always-enabled-content                        0
awesomeos                                    awesomeos                                     0
repolist: 0
[root@rhsm-compat-rhel64 ~]# 


__________________________________________________________
RHEL5 w/ rhsm.baseurl=https://cdn.redhat.com

[root@rhsm-compat-rhel59 ~]# subscription-manager config --rhsm.baseurl=https://cdn.redhat.com
[root@rhsm-compat-rhel59 ~]# yum repolist
Loaded plugins: product-id, security, subscription-manager
This system is receiving updates from Red Hat Subscription Management.
https://cdn.redhat.com/foo/path/always/5Server/repodata/repomd.xml: [Errno 14] HTTP Error 403: Forbidden
Trying other mirror.
repo id                                      repo name                                     status
always-enabled-content                       always-enabled-content                        0
awesomeos                                    awesomeos                                     0
repolist: 0
[root@rhsm-compat-rhel59 ~]# 


__________________________________________________________
RHEL5 w/ rhsm.baseurl=https://cdn.rcm-qa.redhat.com

[root@rhsm-compat-rhel59 ~]# subscription-manager config --rhsm.baseurl=https://cdn.rcm-qa.redhat.com
[root@rhsm-compat-rhel59 ~]# yum repolist
Loaded plugins: product-id, security, subscription-manager
This system is receiving updates from Red Hat Subscription Management.
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in ?
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 309, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 178, in main
    result, resultmsgs = base.doCommands()
  File "/usr/share/yum-cli/cli.py", line 349, in doCommands
    return self.yum_cli_commands[self.basecmd].doCommand(self, self.basecmd, self.extcmds)
  File "/usr/share/yum-cli/yumcommands.py", line 788, in doCommand
    base.repos.populateSack()
  File "/usr/lib/python2.4/site-packages/yum/repos.py", line 260, in populateSack
    sack.populate(repo, mdtype, callback, cacheonly)
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 168, in populate
    if self._check_db_version(repo, mydbtype):
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 226, in _check_db_version
    return repo._check_db_version(mdtype)
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1226, in _check_db_version
    repoXML = self.repoXML
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1399, in <lambda>
    repoXML = property(fget=lambda self: self._getRepoXML(),
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1391, in _getRepoXML
    self._loadRepoXML(text=self)
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1381, in _loadRepoXML
    return self._groupLoadRepoXML(text, ["primary"])
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1365, in _groupLoadRepoXML
    if self._commonLoadRepoXML(text):
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 1201, in _commonLoadRepoXML
    result = self._getFileRepoXML(local, text)
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 974, in _getFileRepoXML
    cache=self.http_caching == 'all')
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 811, in _getFile
    http_headers=headers,
  File "/usr/lib/python2.4/site-packages/urlgrabber/mirror.py", line 412, in urlgrab
    return self._mirror_try(func, url, kw)
  File "/usr/lib/python2.4/site-packages/urlgrabber/mirror.py", line 398, in _mirror_try
    return func_ref( *(fullurl,), **kwargs )
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 936, in urlgrab
    return self._retry(opts, retryfunc, url, filename)
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 854, in _retry
    r = apply(func, (opts,) + args, {})
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 922, in retryfunc
    fo = URLGrabberFileObject(url, filename, opts)
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 1010, in __init__
    self._do_open()
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 1093, in _do_open
    fo, hdr = self._make_request(req, opener)
  File "/usr/lib/python2.4/site-packages/urlgrabber/grabber.py", line 1202, in _make_request
    fo = opener.open(req)
  File "/usr/lib64/python2.4/urllib2.py", line 358, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.4/urllib2.py", line 376, in _open
    '_open', req)
  File "/usr/lib64/python2.4/urllib2.py", line 337, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.4/site-packages/M2Crypto/m2urllib2.py", line 82, in https_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/lib64/python2.4/httplib.py", line 810, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.4/httplib.py", line 833, in _send_request
    self.endheaders()
  File "/usr/lib64/python2.4/httplib.py", line 804, in endheaders
    self._send_output()
  File "/usr/lib64/python2.4/httplib.py", line 685, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.4/httplib.py", line 652, in send
    self.connect()
  File "/usr/lib64/python2.4/site-packages/M2Crypto/httpslib.py", line 55, in connect
    sock.connect((self.host, self.port))
  File "/usr/lib64/python2.4/site-packages/M2Crypto/SSL/Connection.py", line 174, in connect
    ret = self.connect_ssl()
  File "/usr/lib64/python2.4/site-packages/M2Crypto/SSL/Connection.py", line 167, in connect_ssl
    return m2.ssl_connect(self.ssl, self._timeout)
M2Crypto.SSL.SSLError: tlsv1 alert unknown ca
Comment 3 Adrian Likins 2013-05-14 17:49:45 EDT
for comment #2, what's rhsm_ca_cert configured to?
Comment 4 John Sefler 2013-05-15 14:28:31 EDT
(In reply to comment #3)
> for comment #2, what's rhsm_ca_cert configured to?

__________________________________________________________
RHEL6
[root@rhsm-compat-rhel64 ~]# grep ca_cert /etc/rhsm/rhsm.conf
ca_cert_dir = /etc/rhsm/ca/
repo_ca_cert = %(ca_cert_dir)sredhat-uep.pem
[root@rhsm-compat-rhel64 ~]# ls /etc/rhsm/ca/
candlepin-compat-rhel64.pem  candlepin-stage.pem  redhat-uep.pem

__________________________________________________________
RHEL5
[root@rhsm-compat-rhel59 ~]# grep ca_cert /etc/rhsm/rhsm.conf
ca_cert_dir = /etc/rhsm/ca/
repo_ca_cert = %(ca_cert_dir)sredhat-uep.pem
[root@rhsm-compat-rhel59 ~]# ls /etc/rhsm/ca/
candlepin-compat-rhel59.pem  candlepin-stage.pem  redhat-uep.pem
Comment 5 John Sefler 2013-05-22 10:11:04 EDT
The traceback observed in this bug comes from yum on rhel5.  Since this behavior has improved in newer yum in rhel6, I'm going to close this bug.  It does not belong to component python-rhsm.

Note You need to log in before you can comment on or make changes to this bug.