813127 – pycurl error during yum

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 813127 - pycurl error during yum

Summary: pycurl error during yum

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 6
Classification:	Red Hat
Component:	curl
Sub Component:
Version:	6.2
Hardware:	All
OS:	Linux
Priority:	unspecified
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Kamil Dudka
QA Contact:	Jiri Jaburek
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2012-04-17 02:02 UTC by Jon Disnard
Modified:	2013-02-21 10:09 UTC (History)
CC List:	3 users (show)
Fixed In Version:	curl-7.19.7-31.el6
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2013-02-21 10:09:41 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
backport of upstream commit 7fd32ce (1.05 KB, patch) 2012-04-17 11:39 UTC, Kamil Dudka	no flags	Details \| Diff
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2013:0393	0	normal	SHIPPED_LIVE	curl bug fix and enhancement update	2013-02-20 20:51:32 UTC

Description Jon Disnard 2012-04-17 02:02:26 UTC

Description of problem:

  When performing yum operations sometimes the pycurl bindings error:
  
  # yum --noplugins clean all ; yum --noplugins repolist
  Cleaning repos: base updates
  Cleaning up Everything
  http://192.168.1.1:8181/yum/rhel/6.2/i386/repodata/repomd.xml: [Errno 14] PYCURL ERROR 18 - "transfer closed with outstanding read data remaining"
  
  [...]




  Manually using curl we are able to reproduce the problem:
  
  # curl -vv http://192.168.1.1:8181/yum/rhel/6.2/i386/repodata/repomd.xml

  [...]

  * transfer closed with outstanding read data remaining
  * Closing connection #0
  curl: (18) transfer closed with outstanding read data remaining




Version-Release number of selected component (if applicable):

  [root@bluesky ~]# rpm -q curl libcurl
  curl-7.19.7-26.el6_2.4.i686
  libcurl-7.19.7-26.el6_2.4.i686



How reproducible:
  The problem is repeatable most of the time.
  The problem seems to be related to the order in which libcurl receives and processes the last few TCP/HTTP packets, so the issue is not always reproducible unless the other side (the httpd) can always order the packets just so.
  The problem is not reproducible in upstream version from Fedora 16 (currently 7.21.7-7 as of this writing).
  
  



Steps to Reproduce:
1. YUM update with pycurl bindings to the poor httpd implementation (uhttpd in this case)
2.
3.
  
Actual results:
  See the above error 18 from curl.


Expected results:
  Expectation that utilities like python/yum that make use of libcurl to work resiliently without error.
  


Additional info:

  The problem appears to be related to this debian bug report (contains dialog with upstream maintainers of libcurl):
  http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=572276

Comment 2 Kamil Dudka 2012-04-17 11:26:25 UTC

(In reply to comment #0)
>   The problem appears to be related to this debian bug report (contains dialog
> with upstream maintainers of libcurl):
>   http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=572276

This is said to be the upstream fix for the issue:

    https://github.com/bagder/curl/commit/7fd32ce

Could you please check whether the fix solves your problem?

Comment 3 Kamil Dudka 2012-04-17 11:39:44 UTC

Created attachment 578004 [details]
backport of upstream commit 7fd32ce

Comment 4 Jon Disnard 2012-04-17 16:40:41 UTC

> This is said to be the upstream fix for the issue:
> 
>     https://github.com/bagder/curl/commit/7fd32ce
> 
> Could you please check whether the fix solves your problem?


Have cloned the upstream github, checked-out the 7fd32ce version, built and installed that version.

So far I've been unable to reproduce the problem using that specific version.


My testing involves an infinite loop that breaks when curl returns error 18.
This test has been running continuously for over an hour without problem.

while : ; do /usr/local/bin/curl -v http://192.168.1.1:8181/yum/rhel/6.2/i386/repodata/repomd.xml ; case $? in 0) continue ;; *) break ;; esac ; done

Using the system curl 7.19-7 located in /usr/bin/curl we are able to reproduce the problem using the same method very quickly.

I feel safe in saying that this solves the problem.

Hopefully we can get this back-ported.

Thank you.

Comment 5 Kamil Dudka 2012-04-17 21:25:32 UTC

Thanks for the testing, but you tested curl-7_20_1~75, which has not much in common with the version we have in RHEL-6.  Could you please test it with libcurl-7.19.7-26.el6_2.4 with the attached patch applied?  I can prepare an SRPM for testing purposes if needed.

Comment 6 Jon Disnard 2012-04-18 05:12:02 UTC

Sorry about that, I mistakenly thought you wanted the specific upstream version tested.

I installed the src.rpm curl-7.19.7-26.el6_2.4, created the patch, fixed up the spec file, and built the program.

Unfortunately that did not solve the problem as my test case still fails using the new binary.

Here is some info about what was built:

# /root/rpmbuild/BUILDROOT/curl-7.19.7-26.el6.4.i386/usr/bin/curl -V
curl 7.19.7 (i686-pc-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz

The above features your attached patch on this bz.

Here is my test:

while /root/rpmbuild/BUILDROOT/curl-7.19.7-26.el6.4.i386/usr/bin/curl http://192.168.1.1:8181/yum/rhel/6.2/i386/repodata/repomd.xml > /dev/null ;
do continue ;
done

[... about 4 or 5 loop cycles]

curl: (18) transfer closed with outstanding read data remaining

What kills me is that I built another binary from the same curl-7.19.7-26.el6_2.4.src.rpm that does work every single time, and the above loop repeats forever. I just installed that src.rpm, ran buildconf, configure, and make install. The resulting binary in /usr/local/bin/curl doesn't support many features but it works. Please note this binary works with the transfer.c manually patched or left intact.

# /usr/local/bin/curl -V
curl 7.19.7 (i686-pc-linux-gnu) libcurl/7.20.1-DEV
Protocols: dict file ftp http imap pop3 rtsp smtp telnet tftp
Features: IPv6 Largefile

To make it even more confusing, when I start to use ftrace in my testing then for some reason curl never fails.

Comment 7 Kamil Dudka 2012-04-18 07:22:14 UTC

(In reply to comment #6)
> Here is my test:
> 
> while /root/rpmbuild/BUILDROOT/curl-7.19.7-26.el6.4.i386/usr/bin/curl
> http://192.168.1.1:8181/yum/rhel/6.2/i386/repodata/repomd.xml > /dev/null ;
>  do continue ;
> done

This ^^^ is not sufficient for the patch to take an effect.  You are most likely running a new version of the tool (which does not really differ from the old one) against the old version of the library installed on your system.  Please run this command prior to your testing:

> export LD_PRELOAD=$(find /root/rpmbuild/BUILDROOT/curl-7.19.7-26.el6.4.i386 -name libcurl.so)

... and double-check that the patched library is being loaded:

> ldd /root/rpmbuild/BUILDROOT/curl-7.19.7-26.el6.4.i386/usr/bin/curl

Comment 8 Jon Disnard 2012-04-18 13:55:13 UTC

After using LD_PRELOAD the test ran without fail.


Thank you.

Comment 9 Kamil Dudka 2012-04-18 14:33:50 UTC

Great, thanks for testing it!

Comment 12 RHEL Program Management 2012-10-12 11:31:04 UTC

This request was evaluated by Red Hat Product Management for
inclusion in a Red Hat Enterprise Linux release.  Product
Management has requested further review of this request by
Red Hat Engineering, for potential inclusion in a Red Hat
Enterprise Linux release for currently deployed products.
This request is not yet committed for inclusion in a release.

Comment 18 errata-xmlrpc 2013-02-21 10:09:41 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0393.html

Note You need to log in before you can comment on or make changes to this bug.