Bug 949186 - The pycurl downloader times out on active downloads, even during very good transfer rates
Summary: The pycurl downloader times out on active downloads, even during very good tr...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Pulp
Classification: Retired
Component: z_other
Version: 2.1 Beta
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 2.4.0
Assignee: Randy Barlow
QA Contact: Preethi Thomas
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-04-06 21:12 UTC by Randy Barlow
Modified: 2014-08-09 06:54 UTC (History)
0 users

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-08-09 06:54:53 UTC


Attachments (Terms of Use)

Description Randy Barlow 2013-04-06 21:12:34 UTC
When testing the 2.1.1 branch for the ability to synchronize ISOs from the CDN to the proxy, I realized that the download fails exactly five minutes into the download, and that it is repeatable.

I inserted some logging, and discovered that the pycurl downloader is reporting that the download is timing out:

2013-04-06 16:55:02,192 pulp_rpm.plugins.importers.iso_importer.sync:ERROR: report.error_report: {'response_code': 200, 'error_message': 'Operation timed out after 300053 milliseconds with 69531012 out of 3679453184 bytes received', 'error_code': 28}
2013-04-06 16:55:02,198 pulp_rpm.plugins.importers.iso_importer.sync:ERROR: report.error_report: {'response_code': 200, 'error_message': 'Operation timed out after 300054 milliseconds with 69531012 out of 3720347648 bytes received', 'error_code': 28}
2013-04-06 16:55:02,200 pulp_rpm.plugins.importers.iso_importer.sync:ERROR: report.error_report: {'response_code': 200, 'error_message': 'Operation timed out after 300067 milliseconds with 69531012 out of 3589636096 bytes received', 'error_code': 28}
2013-04-06 16:55:02,206 pulp.server.dispatch.task:ERROR: Importer indicated a failed response
Traceback (most recent call last):
  File "/home/rbarlow/devel/pulp/platform/src/pulp/server/dispatch/task.py", line 138, in _run
    result = call(*args, **kwargs)
  File "/home/rbarlow/devel/pulp/platform/src/pulp/server/managers/repo/sync.py", line 118, in sync
    raise PulpExecutionException(_('Importer indicated a failed response'))
PulpExecutionException: Importer indicated a failed response
2013-04-06 16:55:02,206 pulp.server.dispatch.task:INFO: FAILURE: Task 0a022e11-600b-4e78-af2c-cf12a47fd70a: CallRequest: RepoSyncManager.sync(u'cdn_proxy', sync_config_override=None)

Comment 2 Randy Barlow 2013-04-06 21:32:10 UTC
This is very likely due to line 174 in our curl.py:

174         easy_handle.setopt(pycurl.TIMEOUT, DEFAULT_REQUEST_TIMEOUT)

The DEFAULT_REQUEST_TIMEOUT is set to 300 (seconds). According to the curl documentation, pycurl.TIMEOUT is the maximum time an operation is allowed to take[0].

I believe it could be considered a bug to set this option to any value, and we should stick to their default (0), which means no timeout.

There are options for low speed limits and options that will cause a timeout if no bytes have been received in a given amount of time. I think it would be reasonable to set those values in lieu of this particular value. 

[0] http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTTIMEOUT

Comment 3 Randy Barlow 2013-04-07 00:08:24 UTC
This is fixed by https://github.com/pulp/pulp/pull/427

Comment 4 Randy Barlow 2013-04-08 20:52:25 UTC
Merged.

Comment 5 Randy Barlow 2014-03-18 16:04:41 UTC
This was fixed in a prior release of Pulp, but we never put it through the QE process. Moving to ON_QA.

Comment 6 Preethi Thomas 2014-06-02 15:14:02 UTC
verified

[root@hp-dl380pgen8-01 ~]# rpm -qa pulp-server
pulp-server-2.4.0-0.19.beta.fc20.noarch
[root@hp-dl380pgen8-01 ~]# 

[root@hp-dl380pgen8-01 ~]# pulp-admin iso repo create --repo-id cdn-iso --feed https://cdn.redhat.com/content/dist/rhel/server/6/6.5/x86_64/iso/ --feed-ca-cert CDN/cdn.redhat.com-chain.crt  --feed-cert CDN/1391032266_4771.crt  --feed-key CDN/1391032266_4771.key
Successfully created repository [cdn-iso]

[root@hp-dl380pgen8-01 ~]# 
[root@hp-dl380pgen8-01 ~]# 
[root@hp-dl380pgen8-01 ~]# 
[root@hp-dl380pgen8-01 ~]# pulp-admin iso repo sync run --repo-id cdn-iso
+----------------------------------------------------------------------+
                   Synchronizing Repository [cdn-iso]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


The Pulp Manifest was downloaded successfully.

Downloading 3 ISOs...
[==================================================] 100%
ISOs: 3/3	Data: 10.7 GB/10.7 GB	Avg: 3.8 MB/s


Successfully downloaded 3 ISOs.


Task Succeeded





The repository was successfully published.


Task Succeeded

[root@hp-dl380pgen8-01 ~]# pulp-admin iso repo list
+----------------------------------------------------------------------+
                            ISO Repositories
+----------------------------------------------------------------------+

Id:                  iso
Display Name:        iso
Description:         None
Content Unit Counts: 
  Iso: 1

Id:                  cdn-iso
Display Name:        cdn-iso
Description:         None
Content Unit Counts: 
  Iso: 3


[root@hp-dl380pgen8-01 ~]#

Comment 7 Randy Barlow 2014-08-09 06:54:53 UTC
This has been fixed in Pulp 2.4.0-1.


Note You need to log in before you can comment on or make changes to this bug.