Bug 813127

Summary: pycurl error during yum
Product: Red Hat Enterprise Linux 6 Reporter: Jon Disnard <jdisnard>
Component: curlAssignee: Kamil Dudka <kdudka>
Status: CLOSED ERRATA QA Contact: Jiri Jaburek <jjaburek>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2CC: jjaburek, ovasik, prc
Target Milestone: rcKeywords: EasyFix, Patch
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: curl-7.19.7-31.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-21 10:09:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
backport of upstream commit 7fd32ce none

Description Jon Disnard 2012-04-17 02:02:26 UTC
Description of problem:

  When performing yum operations sometimes the pycurl bindings error:
  
  # yum --noplugins clean all ; yum --noplugins repolist
  Cleaning repos: base updates
  Cleaning up Everything
  http://192.168.1.1:8181/yum/rhel/6.2/i386/repodata/repomd.xml: [Errno 14] PYCURL ERROR 18 - "transfer closed with outstanding read data remaining"
  
  [...]




  Manually using curl we are able to reproduce the problem:
  
  # curl -vv http://192.168.1.1:8181/yum/rhel/6.2/i386/repodata/repomd.xml

  [...]

  * transfer closed with outstanding read data remaining
  * Closing connection #0
  curl: (18) transfer closed with outstanding read data remaining




Version-Release number of selected component (if applicable):

  [root@bluesky ~]# rpm -q curl libcurl
  curl-7.19.7-26.el6_2.4.i686
  libcurl-7.19.7-26.el6_2.4.i686



How reproducible:
  The problem is repeatable most of the time.
  The problem seems to be related to the order in which libcurl receives and processes the last few TCP/HTTP packets, so the issue is not always reproducible unless the other side (the httpd) can always order the packets just so.
  The problem is not reproducible in upstream version from Fedora 16 (currently 7.21.7-7 as of this writing).
  
  



Steps to Reproduce:
1. YUM update with pycurl bindings to the poor httpd implementation (uhttpd in this case)
2.
3.
  
Actual results:
  See the above error 18 from curl.


Expected results:
  Expectation that utilities like python/yum that make use of libcurl to work resiliently without error.
  


Additional info:

  The problem appears to be related to this debian bug report (contains dialog with upstream maintainers of libcurl):
  http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=572276

Comment 2 Kamil Dudka 2012-04-17 11:26:25 UTC
(In reply to comment #0)
>   The problem appears to be related to this debian bug report (contains dialog
> with upstream maintainers of libcurl):
>   http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=572276

This is said to be the upstream fix for the issue:

    https://github.com/bagder/curl/commit/7fd32ce

Could you please check whether the fix solves your problem?

Comment 3 Kamil Dudka 2012-04-17 11:39:44 UTC
Created attachment 578004 [details]
backport of upstream commit 7fd32ce

Comment 4 Jon Disnard 2012-04-17 16:40:41 UTC
> This is said to be the upstream fix for the issue:
> 
>     https://github.com/bagder/curl/commit/7fd32ce
> 
> Could you please check whether the fix solves your problem?


Have cloned the upstream github, checked-out the 7fd32ce version, built and installed that version.

So far I've been unable to reproduce the problem using that specific version.


My testing involves an infinite loop that breaks when curl returns error 18.
This test has been running continuously for over an hour without problem.

while : ; do /usr/local/bin/curl -v http://192.168.1.1:8181/yum/rhel/6.2/i386/repodata/repomd.xml ; case $? in 0) continue ;; *) break ;; esac ; done

Using the system curl 7.19-7 located in /usr/bin/curl we are able to reproduce the problem using the same method very quickly.

I feel safe in saying that this solves the problem.

Hopefully we can get this back-ported.

Thank you.

Comment 5 Kamil Dudka 2012-04-17 21:25:32 UTC
Thanks for the testing, but you tested curl-7_20_1~75, which has not much in common with the version we have in RHEL-6.  Could you please test it with libcurl-7.19.7-26.el6_2.4 with the attached patch applied?  I can prepare an SRPM for testing purposes if needed.

Comment 6 Jon Disnard 2012-04-18 05:12:02 UTC
Sorry about that, I mistakenly thought you wanted the specific upstream version tested.


I installed the src.rpm curl-7.19.7-26.el6_2.4, created the patch, fixed up the spec file, and built the program. 

Unfortunately that did not solve the problem as my test case still fails using the new binary.

Here is some info about what was built:

# /root/rpmbuild/BUILDROOT/curl-7.19.7-26.el6.4.i386/usr/bin/curl -V
curl 7.19.7 (i686-pc-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz

The above features your attached patch on this bz.



Here is my test:

while /root/rpmbuild/BUILDROOT/curl-7.19.7-26.el6.4.i386/usr/bin/curl http://192.168.1.1:8181/yum/rhel/6.2/i386/repodata/repomd.xml > /dev/null ;
 do continue ;
done

[... about 4 or 5 loop cycles] 

curl: (18) transfer closed with outstanding read data remaining




What kills me is that I built another binary from the same curl-7.19.7-26.el6_2.4.src.rpm that does work every single time, and the above loop repeats forever. I just installed that src.rpm, ran buildconf, configure, and make install. The resulting binary in /usr/local/bin/curl doesn't support many features but it works. Please note this binary works with the transfer.c manually patched or left intact. 







# /usr/local/bin/curl -V
curl 7.19.7 (i686-pc-linux-gnu) libcurl/7.20.1-DEV
Protocols: dict file ftp http imap pop3 rtsp smtp telnet tftp
Features: IPv6 Largefile



To make it even more confusing, when I start to use ftrace in my testing then for some reason curl never fails.

Comment 7 Kamil Dudka 2012-04-18 07:22:14 UTC
(In reply to comment #6)
> Here is my test:
> 
> while /root/rpmbuild/BUILDROOT/curl-7.19.7-26.el6.4.i386/usr/bin/curl
> http://192.168.1.1:8181/yum/rhel/6.2/i386/repodata/repomd.xml > /dev/null ;
>  do continue ;
> done

This ^^^ is not sufficient for the patch to take an effect.  You are most likely running a new version of the tool (which does not really differ from the old one) against the old version of the library installed on your system.  Please run this command prior to your testing:

> export LD_PRELOAD=$(find /root/rpmbuild/BUILDROOT/curl-7.19.7-26.el6.4.i386 -name libcurl.so)

... and double-check that the patched library is being loaded:

> ldd /root/rpmbuild/BUILDROOT/curl-7.19.7-26.el6.4.i386/usr/bin/curl

Comment 8 Jon Disnard 2012-04-18 13:55:13 UTC
After using LD_PRELOAD the test ran without fail.


Thank you.

Comment 9 Kamil Dudka 2012-04-18 14:33:50 UTC
Great, thanks for testing it!

Comment 12 RHEL Program Management 2012-10-12 11:31:04 UTC
This request was evaluated by Red Hat Product Management for
inclusion in a Red Hat Enterprise Linux release.  Product
Management has requested further review of this request by
Red Hat Engineering, for potential inclusion in a Red Hat
Enterprise Linux release for currently deployed products.
This request is not yet committed for inclusion in a release.

Comment 18 errata-xmlrpc 2013-02-21 10:09:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0393.html