Bug 1019444

Summary: cdn repo sync failing on el6 pulp
Product: [Retired] Pulp Reporter: Preethi Thomas <pthomas>
Component: rpm-supportAssignee: Michael Hrivnak <mhrivnak>
Status: CLOSED NOTABUG QA Contact: pulp-qe-list
Severity: high Docs Contact:
Priority: urgent    
Version: 2.2 BetaCC: chrobert, mhrivnak, skarmark
Target Milestone: ---Keywords: Reopened, Triaged
Target Release: 2.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-10-31 14:04:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Preethi Thomas 2013-10-15 17:50:45 UTC
Description of problem:

cdn repo sync seems to be failing on el6 with latest nectar

Version-Release number of selected component (if applicable):
[root@qe-blade-04 ~]# rpm -qa pulp-server
pulp-server-2.2.0-2.el6.noarch
[root@qe-blade-04 ~]# rpm -qa |grep nectar
python-nectar-1.1.4-1.el6.noarch
[root@qe-blade-04 ~]# 


How reproducible:


Steps to Reproduce:
1. create and sync a cdn rhel repo on el6
2.
3.

Actual results:


from pulp.log

2013-10-14 16:05:44,584 nectar.downloaders.threaded:ERROR: [Errno 1] _ssl.c:1331: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nectar/downloaders/threaded.py", line 176, in _fetch
    for chunk in chunks:
  File "/usr/lib/python2.6/site-packages/requests/models.py", line 572, in generate
    decode_content=True):
  File "/usr/lib/python2.6/site-packages/requests/packages/urllib3/response.py", line 225, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/usr/lib/python2.6/site-packages/requests/packages/urllib3/response.py", line 174, in read
    data = self._fp.read(amt)
  File "/usr/lib64/python2.6/httplib.py", line 542, in read
    s = self.fp.read(amt)
  File "/usr/lib64/python2.6/socket.py", line 383, in read
    data = self._sock.recv(left)
  File "/usr/lib64/python2.6/ssl.py", line 215, in recv
    return self.read(buflen)
  File "/usr/lib64/python2.6/ssl.py", line 136, in read
    return self._sslobj.read(len)
SSLError: [Errno 1] _ssl.c:1331: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
Expected results:


Additional info:

Comment 1 Sayli Karmarkar 2013-10-17 18:33:35 UTC
We are unable to reproduce it on any other rhel6 machines. Online search for the bug pointed to a possible bug in openssl or hardware issues. We have ruled out the openssl bugs since they have been fixed in the version on rhel6 and also since it working in other rhel6 boxes with the same openssl version. Accounting it to a weird hardware issue or anomaly on that box and closing it as not-a-bug.

Comment 2 Preethi Thomas 2013-10-24 13:50:14 UTC
Reopening this as I saw this happening again with the latest 2.3 build

syncing a rhel5 repo

[root@qe-blade-03 ~]#  pulp-admin rpm repo update --repo-id rhel59-os  --proxy-host http://pulp-upgrade-testing.usersys.redhat.com --proxy-port 3128
Repository [rhel59-os] successfully updated


[root@qe-blade-03 ~]# cat /etc/redhat-release 
Fedora release 19 (Schrödinger’s Cat)


[root@qe-blade-03 ~]# rpm -qa pulp-server
pulp-server-2.3.0-0.24.beta.fc19.noarch
[root@qe-blade-03 ~]# 


2013-10-24 08:59:58,908 nectar.downloaders.threaded:ERROR: [Errno 1] _ssl.c:1419: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 177, in _fetch
    for chunk in chunks:
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 572, in generate
    decode_content=True):
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 225, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 174, in read
    data = self._fp.read(amt)
  File "/usr/lib64/python2.7/httplib.py", line 567, in read
    s = self.fp.read(amt)
  File "/usr/lib64/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
  File "/usr/lib64/python2.7/ssl.py", line 241, in recv
    return self.read(buflen)
  File "/usr/lib64/python2.7/ssl.py", line 160, in read
    return self._sslobj.read(len)
SSLError: [Errno 1] _ssl.c:1419: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
2013-10-24 08:59:58,980 requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): cdn.redhat.com
2013-10-24 09:12:27,120 nectar.downloaders.threaded:ERROR: [Errno 1] _ssl.c:1419: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 177, in _fetch
    for chunk in chunks:
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 572, in generate
    decode_content=True):
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 225, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 174, in read
    data = self._fp.read(amt)
  File "/usr/lib64/python2.7/httplib.py", line 567, in read
    s = self.fp.read(amt)
  File "/usr/lib64/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
  File "/usr/lib64/python2.7/ssl.py", line 241, in recv
    return self.read(buflen)
  File "/usr/lib64/python2.7/ssl.py", line 160, in read
    return self._sslobj.read(len)
SSLError: [Errno 1] _ssl.c:1419: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
2013-10-24 09:12:27,149 requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): cdn.redhat.com
2013-10-24 09:14:19,666 nectar.downloaders.threaded:ERROR: [Errno 1] _ssl.c:1419: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 177, in _fetch
    for chunk in chunks:
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 572, in generate
    decode_content=True):
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 225, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 174, in read
    data = self._fp.read(amt)
  File "/usr/lib64/python2.7/httplib.py", line 567, in read
    s = self.fp.read(amt)
  File "/usr/lib64/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
  File "/usr/lib64/python2.7/ssl.py", line 241, in recv
    return self.read(buflen)
  File "/usr/lib64/python2.7/ssl.py", line 160, in read
    return self._sslobj.read(len)
SSLError: [Errno 1] _ssl.c:1419: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac

Comment 3 Michael Hrivnak 2013-10-25 14:42:06 UTC
This looks like the same issue faced by a different stack: https://github.com/rubygems/rubygems.org/issues/615