Description of problem:

I noticed that when I have large RHEL repositories being synchronized, sometimes they fail to sync due to a timeout reaching out to CDN. I haven't been able to determine how large the repo must be to cause the timeout, but having several large repos being synchronized in parallel seem to increase the likelihood of this happening. To be safe, try to synchronize all of RHEL 5 and 6, both x86_64 and i386 architectures.

Version-Release number of selected component (if applicable):

* apr-util-ldap-1.3.9-3.el6_0.1.x86_64
* candlepin-0.8.25-1.el6sam.noarch
* candlepin-scl-1-5.el6_4.noarch
* candlepin-scl-quartz-2.1.5-5.el6_4.noarch
* candlepin-scl-rhino-1.7R3-1.el6_4.noarch
* candlepin-scl-runtime-1-5.el6_4.noarch
* candlepin-selinux-0.8.25-1.el6sam.noarch
* candlepin-tomcat6-0.8.25-1.el6sam.noarch
* elasticsearch-0.19.9-8.el6sat.noarch
* foreman-1.3.0-6.el6sat.noarch
* foreman-compute-1.3.0-6.el6sat.noarch
* foreman-libvirt-1.3.0-6.el6sat.noarch
* foreman-postgresql-1.3.0-6.el6sat.noarch
* foreman-proxy-1.3.0-1.el6sat.noarch
* katello-1.4.6-14.el6sat.noarch
* katello-all-1.4.6-14.el6sat.noarch
* katello-candlepin-cert-key-pair-1.0-1.noarch
* katello-certs-tools-1.4.4-1.el6sat.noarch
* katello-cli-1.4.3-15.el6sat.noarch
* katello-cli-common-1.4.3-15.el6sat.noarch
* katello-common-1.4.6-14.el6sat.noarch
* katello-configure-1.4.5-8.el6sat.noarch
* katello-configure-foreman-1.4.5-8.el6sat.noarch
* katello-configure-foreman-proxy-1.4.5-8.el6sat.noarch
* katello-foreman-all-1.4.6-14.el6sat.noarch
* katello-glue-candlepin-1.4.6-14.el6sat.noarch
* katello-glue-elasticsearch-1.4.6-14.el6sat.noarch
* katello-glue-pulp-1.4.6-14.el6sat.noarch
* katello-qpid-broker-key-pair-1.0-1.noarch
* katello-qpid-client-key-pair-1.0-1.noarch
* katello-selinux-1.4.4-4.el6sat.noarch
* openldap-2.4.23-31.el6.x86_64
* pulp-katello-plugins-0.2-1.el6sat.noarch
* pulp-nodes-common-2.3.0-0.14.alpha.el6sat.noarch
* pulp-nodes-parent-2.3.0-0.14.alpha.el6sat.noarch
* pulp-puppet-plugins-2.3.0-0.14.alpha.el6sat.noarch
* pulp-rpm-plugins-2.3.0-0.14.alpha.el6sat.noarch
* pulp-selinux-2.3.0-0.14.alpha.el6sat.noarch
* pulp-server-2.3.0-0.14.alpha.el6sat.noarch
* python-ldap-2.3.10-1.el6.x86_64
* ruby193-rubygem-ldap_fluff-0.2.2-2.el6sat.noarch
* ruby193-rubygem-net-ldap-0.3.1-3.el6sat.noarch
* ruby193-rubygem-runcible-1.0.7-1.el6sat.noarch
* signo-0.0.20-3.el6sat.noarch
* signo-katello-0.0.20-3.el6sat.noarch

How reproducible:


Steps to Reproduce:
1. try to synchronize all of RHEL 5 and 6, both x86_64 and i386 architectures
2.
3.

Actual results:

Some repos eventually fail to sync

Expected results:


Additional info:

/var/log/nectar.log shows:

2013-09-30 00:31:09,528 urllib3.connectionpool:INFO: Starting new HTTPS connection (2): cdn.redhat.com
2013-09-30 00:31:09,898 urllib3.connectionpool:INFO: Resetting dropped connection: cdn.redhat.com
2013-09-30 00:31:10,528 urllib3.connectionpool:WARNING: Retrying (0 attempts remain) after connection broken by 'BadStatusLine('',)': /content/dist/rhel/power/6/6Server/ppc64/os/Packages/python-decorator-3.0.1-3.1.el6.noarch.rpm
2013-09-30 00:31:10,532 nectar.downloaders.threaded:ERROR: HTTPSConnectionPool(host='cdn.redhat.com', port=443): Max retries exceeded with url: /content/dist/rhel/power/6/6Server/ppc64/os/Packages/python-decorator-3.0.1-3.1.el6.noarch.rpm
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nectar/downloaders/threaded.py", line 160, in _fetch
    response = session.get(request.url, headers=headers)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 310, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 279, in request
    resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 374, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/adapters.py", line 209, in send
    raise ConnectionError(e)
ConnectionError: HTTPSConnectionPool(host='cdn.redhat.com', port=443): Max retries exceeded with url: /content/dist/rhel/power/6/6Server/ppc64/os/Packages/python-decorator-3.0.1-3.1.el6.noarch.rpm
2013-09-30 00:31:10,539 urllib3.connectionpool:INFO: Starting new HTTPS connection (2): cdn.redhat.com
2013-09-30 00:31:11,315 urllib3.connectionpool:INFO: Resetting dropped connection: cdn.redhat.com
2013-09-30 00:31:13,353 urllib3.connectionpool:INFO: Resetting dropped connection: cdn.redhat.com
2013-09-30 00:37:37,695 urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
2013-09-30 00:37:38,049 nectar.downloaders.threaded:ERROR: Download of https://cdn.redhat.com/content/dist/rhel/power/6/6Server/ppc64/os/.treeinfo failed with code 404: Not Found