Bug 1130239 - Difference in curl performance between RHEL6 and RHEL7
Summary: Difference in curl performance between RHEL6 and RHEL7
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: curl
Version: 7.0
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Kamil Dudka
QA Contact: Stefan Kremen
Laura Bailey
URL:
Whiteboard:
Depends On:
Blocks: 1205796 1110700 1133060 1191021
TreeView+ depends on / blocked
 
Reported: 2014-08-14 15:27 UTC by Alastair Scobie
Modified: 2019-08-15 03:56 UTC (History)
9 users (show)

Fixed In Version: curl-7.29.0-24.el7
Doc Type: Bug Fix
Doc Text:
Improved wait times in libcurl The *libcurl* library used an unnecessarily long blocking delay for actions with no active file descriptors, even for short operations. This meant that some actions, such as resolving a host name using */etc/hosts*, took an artificially long time to complete. The blocking code in *libcurl* has now been modified so that the initial delay is short, and gradually increases until an event occurs. Fast *libcurl* operations now complete more quickly.
Clone Of:
Environment:
Last Closed: 2015-11-19 07:06:25 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2015:2159 normal SHIPPED_LIVE Moderate: curl security, bug fix, and enhancement update 2015-11-19 08:26:18 UTC
Red Hat Knowledge Base (Solution) 1214923 None None None Never

Description Alastair Scobie 2014-08-14 15:27:52 UTC
Description of problem:

Whilst porting an in-house application, which uses libcurl, from RHEL6 to RHEL7 we noticed that the time taken by libcurl to fetch a page had substantially increased. Even an attempt at fetching a non-existent page has increased from 10-20ms to around 150ms. This with just plain HTTP.

Our first thought was that the API has changed in some subtle way and the fault must be in our code, but then we thought to try measuring the performance of the curl tool itself. To our surprise, we found the same performance disparity.

Interestingly...

* Adding the DNS address of the HTTP server to /etc/hosts (and specifying files,dns in /etc/nsswitch.conf) does not improve things

* but, performance when using an IP address in the URL is fine (~ 10-20ms) for both RHEL6 and RHEL7

... which kind of suggests that the delay is in libcurl's DNS code rather than in the upstream resolver.

We did notice that when using DNS names in the URL, curl forks off a process to do the name lookup and then poll()s. There's no such fork when using an IP address in the URL. The poll() timesout after a period of time - we suspect this is the cause of the delay.



Version-Release number of selected component (if applicable):

RHEL 7.0

How reproducible:

Use the "curl" command to fetch a non-existent page off a working HTTP server.

Steps to Reproduce:
1. Setup an HTTP server, say myserver.mydomain.org
2. time curl http://myserver.mydomain.org/non-existent-page.html


Actual results:

Running curl on an RHEL 6.4 box, the time taken by curl is ~10-20ms. 
Running curl on an RHEL 7.0 box, the time taken by curl is ~150ms.

Expected results:

Time taken should be same under both OS releases.

Additional info:

This "bug" was reported to the Curl mailing list, and Daniel Stenberg suggested "this is simply because Redhat switched their build to the threaded resolver and in 7.29.0 we switched some libcurl internals which made us have that performance issue. I think this problem is solved in a more recent libcurl version."

Comment 2 Kamil Dudka 2014-08-14 18:44:42 UTC
Thank you for reporting the bug here, Alastair!

This was originally reported at the upstream mailing-list:

http://thread.gmane.org/gmane.comp.web.curl.general/14247

I was able to reproduce the delay on my RHEl-7 box by resolving localhost:

$ time curl http://localhost -v --trace-time
20:37:13.240666 * About to connect() to localhost port 80 (#0)
20:37:13.240969 *   Trying ::1...
20:37:13.241285 * Connection refused
20:37:13.241477 *   Trying 127.0.0.1...
20:37:13.241648 * Connection refused
20:37:13.241755 * Failed connect to localhost:80; Connection refused
20:37:13.241851 * Closing connection 0
curl: (7) Failed connect to localhost:80; Connection refused

real    0m0.185s
user    0m0.030s
sys     0m0.005s

It seems to be caused by the following upstream patch:

https://github.com/bagder/curl/commit/0feeab78

... that was applied on Fedora 19:

http://pkgs.fedoraproject.org/cgit/curl.git/commit/?id=68cb5f29

The following upstream commit seems to fix it:

https://github.com/bagder/curl/commit/d529f388

Comment 6 Kamil Dudka 2014-08-15 15:31:24 UTC
fixed in curl-7.29.0-22.fc19

Comment 16 Kamil Dudka 2015-07-31 12:32:55 UTC
While testing the package, Stefan Kremen discovered that the fixed version of RHEL-7 libcurl is still _slightly_ slower in resolving hosts from /etc/hosts, compared to RHEL-6 libcurl.

Unfortunately, this behavior is expected and nearly impossible to fix.  The RHEL-6 version of libcurl uses the blocking interface of the system resolver directly whereas the RHEL-7 version of libcurl uses threads to implement a non-blocking interface on top of the blocking interface of the system resolver.  Since the system resolver provides only blocking API, polling is necessary.

On the other hand, single-threaded applications are no longer blocked during name resolving in RHEL-7.  Moreover, it is no longer needed to use setjmp/longjmp to implement timeouts, which is known to occasionally crash the python interpreter in RHEL-6.

Comment 20 errata-xmlrpc 2015-11-19 07:06:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-2159.html


Note You need to log in before you can comment on or make changes to this bug.