Bug 1130239

Summary:	Difference in curl performance between RHEL6 and RHEL7
Product:	Red Hat Enterprise Linux 7	Reporter:	Alastair Scobie <ascobie>
Component:	curl	Assignee:	Kamil Dudka <kdudka>
Status:	CLOSED ERRATA	QA Contact:	Stefan Kremen <skremen>
Severity:	high	Docs Contact:	Laura Bailey <lbailey>
Priority:	high
Version:	7.0	CC:	andreas.schiermeier, msaxena, ovasik, pasteur, riehecky, skremen, stephan.schultchen, tbowling, zpytela
Target Milestone:	rc	Keywords:	Patch
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	curl-7.29.0-24.el7	Doc Type:	Bug Fix
Doc Text:	Improved wait times in libcurl The libcurl library used an unnecessarily long blocking delay for actions with no active file descriptors, even for short operations. This meant that some actions, such as resolving a host name using /etc/hosts, took an artificially long time to complete. The blocking code in libcurl has now been modified so that the initial delay is short, and gradually increases until an event occurs. Fast libcurl operations now complete more quickly.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-11-19 07:06:25 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1110700, 1133060, 1191021, 1205796

Description Alastair Scobie 2014-08-14 15:27:52 UTC

Description of problem:

Whilst porting an in-house application, which uses libcurl, from RHEL6 to RHEL7 we noticed that the time taken by libcurl to fetch a page had substantially increased. Even an attempt at fetching a non-existent page has increased from 10-20ms to around 150ms. This with just plain HTTP.

Our first thought was that the API has changed in some subtle way and the fault must be in our code, but then we thought to try measuring the performance of the curl tool itself. To our surprise, we found the same performance disparity.

Interestingly...

* Adding the DNS address of the HTTP server to /etc/hosts (and specifying files,dns in /etc/nsswitch.conf) does not improve things

* but, performance when using an IP address in the URL is fine (~ 10-20ms) for both RHEL6 and RHEL7

... which kind of suggests that the delay is in libcurl's DNS code rather than in the upstream resolver.

We did notice that when using DNS names in the URL, curl forks off a process to do the name lookup and then poll()s. There's no such fork when using an IP address in the URL. The poll() timesout after a period of time - we suspect this is the cause of the delay.

Version-Release number of selected component (if applicable):

RHEL 7.0

How reproducible:

Use the "curl" command to fetch a non-existent page off a working HTTP server.

Steps to Reproduce:
1. Setup an HTTP server, say myserver.mydomain.org
2. time curl http://myserver.mydomain.org/non-existent-page.html

Actual results:

Running curl on an RHEL 6.4 box, the time taken by curl is ~10-20ms.
Running curl on an RHEL 7.0 box, the time taken by curl is ~150ms.

Expected results:

Time taken should be same under both OS releases.

Additional info:

This "bug" was reported to the Curl mailing list, and Daniel Stenberg suggested "this is simply because Redhat switched their build to the threaded resolver and in 7.29.0 we switched some libcurl internals which made us have that performance issue. I think this problem is solved in a more recent libcurl version."

Comment 2 Kamil Dudka 2014-08-14 18:44:42 UTC

Thank you for reporting the bug here, Alastair!

This was originally reported at the upstream mailing-list:

http://thread.gmane.org/gmane.comp.web.curl.general/14247

I was able to reproduce the delay on my RHEl-7 box by resolving localhost:

$ time curl http://localhost -v --trace-time
20:37:13.240666 * About to connect() to localhost port 80 (#0)
20:37:13.240969 *   Trying ::1...
20:37:13.241285 * Connection refused
20:37:13.241477 *   Trying 127.0.0.1...
20:37:13.241648 * Connection refused
20:37:13.241755 * Failed connect to localhost:80; Connection refused
20:37:13.241851 * Closing connection 0
curl: (7) Failed connect to localhost:80; Connection refused

real    0m0.185s
user    0m0.030s
sys     0m0.005s

It seems to be caused by the following upstream patch:

https://github.com/bagder/curl/commit/0feeab78

... that was applied on Fedora 19:

http://pkgs.fedoraproject.org/cgit/curl.git/commit/?id=68cb5f29

The following upstream commit seems to fix it:

https://github.com/bagder/curl/commit/d529f388

Comment 6 Kamil Dudka 2014-08-15 15:31:24 UTC

fixed in curl-7.29.0-22.fc19

Comment 16 Kamil Dudka 2015-07-31 12:32:55 UTC

While testing the package, Stefan Kremen discovered that the fixed version of RHEL-7 libcurl is still _slightly_ slower in resolving hosts from /etc/hosts, compared to RHEL-6 libcurl.

Unfortunately, this behavior is expected and nearly impossible to fix.  The RHEL-6 version of libcurl uses the blocking interface of the system resolver directly whereas the RHEL-7 version of libcurl uses threads to implement a non-blocking interface on top of the blocking interface of the system resolver.  Since the system resolver provides only blocking API, polling is necessary.

On the other hand, single-threaded applications are no longer blocked during name resolving in RHEL-7.  Moreover, it is no longer needed to use setjmp/longjmp to implement timeouts, which is known to occasionally crash the python interpreter in RHEL-6.

Comment 20 errata-xmlrpc 2015-11-19 07:06:25 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-2159.html