Some kind of bug interacting with github's servers: $ curl -v -L http://raw.github.com/teeedubb/teeedubb-xbmc-repo/master/addons.xml.md5 * Adding handle: conn: 0x96da60 * Adding handle: send: 0 * Adding handle: recv: 0 * Curl_addHandleToPipeline: length: 1 * - Conn 0 (0x96da60) send_pipe: 1, recv_pipe: 0 * About to connect() to raw.github.com port 80 (#0) * Trying 185.31.17.133... * Connected to raw.github.com (185.31.17.133) port 80 (#0) > GET /teeedubb/teeedubb-xbmc-repo/master/addons.xml.md5 HTTP/1.1 > User-Agent: curl/7.32.0 > Host: raw.github.com > Accept: */* > < HTTP/1.1 301 Moved Permanently * Server Varnish is not blacklisted < Server: Varnish < Retry-After: 0 < Location: https://raw.github.com/teeedubb/teeedubb-xbmc-repo/master/addons.xml.md5 < Content-Length: 0 < Accept-Ranges: bytes < Date: Sat, 17 May 2014 14:31:34 GMT < Via: 1.1 varnish < Connection: close < X-Served-By: cache-fra1220-FRA < X-Cache: MISS < X-Cache-Hits: 0 < * Closing connection 0 * Issue another request to this URL: 'https://raw.github.com/teeedubb/teeedubb-xbmc-repo/master/addons.xml.md5' * Adding handle: conn: 0x96da60 * Adding handle: send: 0 * Adding handle: recv: 0 * Curl_addHandleToPipeline: length: 1 * - Conn 1 (0x96da60) send_pipe: 1, recv_pipe: 0 * About to connect() to raw.github.com port 443 (#1) * Trying 185.31.17.133... * Connected to raw.github.com (185.31.17.133) port 443 (#1) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * SSL connection using TLS_RSA_WITH_RC4_128_SHA * Server certificate: * subject: CN=www.github.com,O="Fastly, Inc.",L=San Francisco,ST=California,C=US * start date: Feb 25 00:00:00 2014 GMT * expire date: Mar 02 12:00:00 2015 GMT * common name: www.github.com * issuer: CN=DigiCert High Assurance CA-3,OU=www.digicert.com,O=DigiCert Inc,C=US > GET /teeedubb/teeedubb-xbmc-repo/master/addons.xml.md5 HTTP/1.1 > User-Agent: curl/7.32.0 > Host: raw.github.com > Accept: */* > < HTTP/1.1 301 Moved Permanently < Date: Sat, 17 May 2014 14:31:34 GMT * Server Apache is not blacklisted < Server: Apache < Location: https://raw.githubusercontent.com/teeedubb/teeedubb-xbmc-repo/master/addons.xml.md5 < Content-Length: 0 < Accept-Ranges: bytes < Via: 1.1 varnish < Age: 0 < X-Served-By: cache-fra1222-FRA < X-Cache: MISS < X-Cache-Hits: 0 < Vary: Accept-Encoding < * Connection #1 to host raw.github.com left intact * Issue another request to this URL: 'https://raw.githubusercontent.com/teeedubb/teeedubb-xbmc-repo/master/addons.xml.md5' * Adding handle: conn: 0xa35f70 * Adding handle: send: 0 * Adding handle: recv: 0 * Curl_addHandleToPipeline: length: 1 * - Conn 2 (0xa35f70) send_pipe: 1, recv_pipe: 0 * About to connect() to raw.githubusercontent.com port 443 (#2) * Trying 185.31.17.133... * Connected to raw.githubusercontent.com (185.31.17.133) port 443 (#2) * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * SSL connection using TLS_RSA_WITH_RC4_128_SHA * Server certificate: * subject: CN=www.github.com,O="Fastly, Inc.",L=San Francisco,ST=California,C=US * start date: Feb 25 00:00:00 2014 GMT * expire date: Mar 02 12:00:00 2015 GMT * common name: www.github.com * issuer: CN=DigiCert High Assurance CA-3,OU=www.digicert.com,O=DigiCert Inc,C=US > GET /teeedubb/teeedubb-xbmc-repo/master/addons.xml.md5 HTTP/1.1 > User-Agent: curl/7.32.0 > Host: raw.githubusercontent.com > Accept: */* > < HTTP/1.1 400 Bad Request < Date: Sat, 17 May 2014 14:31:35 GMT * Server Apache is not blacklisted < Server: Apache < Content-Length: 226 < Connection: close < Content-Type: text/html; charset=iso-8859-1 < <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>400 Bad Request</title> </head><body> <h1>Bad Request</h1> <p>Your browser sent a request that this server could not understand.<br /> </p> </body></html> * Closing connection 2
wget has no problems though: $ wget -d http://raw.github.com/teeedubb/teeedubb-xbmc-repo/master/addons.xml.md5 DEBUG output created by Wget 1.14 on linux-gnu. URI encoding = ‘UTF-8’ --2014-05-17 16:33:47-- http://raw.github.com/teeedubb/teeedubb-xbmc-repo/master/addons.xml.md5 Resolving raw.github.com (raw.github.com)... 185.31.16.133 Caching raw.github.com => 185.31.16.133 Connecting to raw.github.com (raw.github.com)|185.31.16.133|:80... connected. Created socket 3. Releasing 0x00000000024cf2f0 (new refcount 1). ---request begin--- GET /teeedubb/teeedubb-xbmc-repo/master/addons.xml.md5 HTTP/1.1 User-Agent: Wget/1.14 (linux-gnu) Accept: */* Host: raw.github.com Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 301 Moved Permanently Server: Varnish Retry-After: 0 Location: https://raw.github.com/teeedubb/teeedubb-xbmc-repo/master/addons.xml.md5 Content-Length: 0 Accept-Ranges: bytes Date: Sat, 17 May 2014 14:33:47 GMT Via: 1.1 varnish Connection: close X-Served-By: cache-am71-AMS X-Cache: MISS X-Cache-Hits: 0 ---response end--- 301 Moved Permanently Location: https://raw.github.com/teeedubb/teeedubb-xbmc-repo/master/addons.xml.md5 [following] Closed fd 3 URI content encoding = None --2014-05-17 16:33:47-- https://raw.github.com/teeedubb/teeedubb-xbmc-repo/master/addons.xml.md5 Found raw.github.com in host_name_addresses_map (0x24cf2f0) Connecting to raw.github.com (raw.github.com)|185.31.16.133|:443... connected. Created socket 3. Releasing 0x00000000024cf2f0 (new refcount 1). Initiating SSL handshake. Handshake successful; connected socket 3 to SSL handle 0x00000000025b5840 certificate: subject: /C=US/ST=California/L=San Francisco/O=Fastly, Inc./CN=www.github.com issuer: /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance CA-3 X509 certificate successfully verified and matches host raw.github.com ---request begin--- GET /teeedubb/teeedubb-xbmc-repo/master/addons.xml.md5 HTTP/1.1 User-Agent: Wget/1.14 (linux-gnu) Accept: */* Host: raw.github.com Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 301 Moved Permanently Date: Sat, 17 May 2014 14:33:48 GMT Server: Apache Location: https://raw.githubusercontent.com/teeedubb/teeedubb-xbmc-repo/master/addons.xml.md5 Content-Length: 0 Accept-Ranges: bytes Via: 1.1 varnish Age: 0 X-Served-By: cache-am71-AMS X-Cache: MISS X-Cache-Hits: 0 Vary: Accept-Encoding Keep-Alive: timeout=10, max=50 Connection: Keep-Alive ---response end--- 301 Moved Permanently Registered socket 3 for persistent reuse. Location: https://raw.githubusercontent.com/teeedubb/teeedubb-xbmc-repo/master/addons.xml.md5 [following] ] done. URI content encoding = None --2014-05-17 16:33:48-- https://raw.githubusercontent.com/teeedubb/teeedubb-xbmc-repo/master/addons.xml.md5 Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.31.16.133 Caching raw.githubusercontent.com => 185.31.16.133 Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.31.16.133|:443... connected. Created socket 4. Releasing 0x00000000025ed5b0 (new refcount 1). Initiating SSL handshake. Handshake successful; connected socket 4 to SSL handle 0x00000000025eac50 certificate: subject: /C=US/ST=California/L=San Francisco/O=Fastly, Inc./CN=www.github.com issuer: /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance CA-3 X509 certificate successfully verified and matches host raw.githubusercontent.com ---request begin--- GET /teeedubb/teeedubb-xbmc-repo/master/addons.xml.md5 HTTP/1.1 User-Agent: Wget/1.14 (linux-gnu) Accept: */* Host: raw.githubusercontent.com Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 200 OK Date: Sat, 17 May 2014 14:33:48 GMT Server: Apache Content-Security-Policy: default-src 'none' Access-Control-Allow-Origin: https://render.githubusercontent.com X-XSS-Protection: 1; mode=block X-Frame-Options: deny X-Content-Type-Options: nosniff Strict-Transport-Security: max-age=31536000 ETag: "42f226a4b34bb8491337de8d809eb843eeae7a8e" Content-Type: text/plain; charset=utf-8 Cache-Control: max-age=300 Content-Length: 32 Accept-Ranges: bytes Via: 1.1 varnish X-Served-By: cache-am69-AMS X-Cache: HIT X-Cache-Hits: 1 Vary: Authorization,Accept-Encoding Expires: Sat, 17 May 2014 14:38:48 GMT Source-Age: 31 Keep-Alive: timeout=10, max=50 Connection: Keep-Alive ---response end--- 200 OK Disabling further reuse of socket 3. Closed 3/SSL 0x00000000025b5840 Registered socket 4 for persistent reuse. URI content encoding = ‘utf-8’ Length: 32 [text/plain] Saving to: ‘addons.xml.md5’ 100%[======================================================================================================================================================================>] 32 --.-K/s in 0s 2014-05-17 16:33:48 (1.89 MB/s) - ‘addons.xml.md5’ saved [32/32]
It seems to be the reuse of the connection that goes wrong. Using the final URL works fine: $ curl -v https://raw.githubusercontent.com/teeedubb/teeedubb-xbmc-repo/master/addons.xml.md5 * Adding handle: conn: 0x1664ab0 * Adding handle: send: 0 * Adding handle: recv: 0 * Curl_addHandleToPipeline: length: 1 * - Conn 0 (0x1664ab0) send_pipe: 1, recv_pipe: 0 * About to connect() to raw.githubusercontent.com port 443 (#0) * Trying 185.31.17.133... * Connected to raw.githubusercontent.com (185.31.17.133) port 443 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * SSL connection using TLS_RSA_WITH_RC4_128_SHA * Server certificate: * subject: CN=www.github.com,O="Fastly, Inc.",L=San Francisco,ST=California,C=US * start date: Feb 25 00:00:00 2014 GMT * expire date: Mar 02 12:00:00 2015 GMT * common name: www.github.com * issuer: CN=DigiCert High Assurance CA-3,OU=www.digicert.com,O=DigiCert Inc,C=US > GET /teeedubb/teeedubb-xbmc-repo/master/addons.xml.md5 HTTP/1.1 > User-Agent: curl/7.32.0 > Host: raw.githubusercontent.com > Accept: */* > < HTTP/1.1 200 OK < Date: Sat, 17 May 2014 14:39:17 GMT * Server Apache is not blacklisted < Server: Apache < Content-Security-Policy: default-src 'none' < Access-Control-Allow-Origin: https://render.githubusercontent.com < X-XSS-Protection: 1; mode=block < X-Frame-Options: deny < X-Content-Type-Options: nosniff < Strict-Transport-Security: max-age=31536000 < ETag: "42f226a4b34bb8491337de8d809eb843eeae7a8e" < Content-Type: text/plain; charset=utf-8 < Cache-Control: max-age=300 < Content-Length: 32 < Accept-Ranges: bytes < Via: 1.1 varnish < X-Served-By: cache-fra1230-FRA < X-Cache: MISS < X-Cache-Hits: 0 < Vary: Authorization,Accept-Encoding < Expires: Sat, 17 May 2014 14:44:17 GMT < Source-Age: 0 < * Connection #0 to host raw.githubusercontent.com left intact
The server does not seem to be compatible with the SSL cache implemented by NSS. I am able to connect with --no-sessionid and the following patch applied: https://github.com/bagder/curl/commit/f63603de I will backport it for Fedora...
Does that mean that there will still be issues if you don't have that flag? (as I'd imagine is the case in most scripts and programs)
(In reply to Pierre Ossman from comment #4) > Does that mean that there will still be issues if you don't have that flag? At this point, it is not clear whether this is a server bug or client bug. By implementing the --no-sessionid option of curl, we make it easier to debug, at least. There is not much we can change in libcurl, except disabling the SSL cache by default, which is a no-go in my view. If it really is a client bug, it needs to be fixed in NSS.
Doesn't firefox use NSS? I don't see any problems there.
This particular problem is caused by SNI host and Host: header being different. Apache doesn't like that, and throws a 400 back if it happens. This is why the request works just fine if you go to the redirect URL directly. It is an issue that is addressed in upstream curl, at least with OpenSSL, not sure about NSS. There were a few commits on November 4 2010 to deal with it, but it seems that code has been moved so it maybe generalized now. TL;DR: Apache returns 400 if TLS handshake uses a different hostname than Host: header.
This seems to be a known issue: https://github.com/hiviah/https-everywhere-checker/commit/861bd9a2 I believe curl correctly sets the hostname via SSL_SetURL() for NSS. There must be something wrong with the NSS' session cache. Anyway, I am going to pick the usptream commit f63603de for now as stated in comment #3.
curl-7.29.0-20.fc19 has been submitted as an update for Fedora 19. https://admin.fedoraproject.org/updates/curl-7.29.0-20.fc19
curl-7.32.0-11.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/curl-7.32.0-11.fc20
Package curl-7.29.0-20.fc19: * should fix your issue, * was pushed to the Fedora 19 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing curl-7.29.0-20.fc19' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2014-6988/curl-7.29.0-20.fc19 then log in and leave karma (feedback).
fwiw, I tested the F19 update, and it retrieved the file with --no-sessionid . still gave 400 without it. i believe this is as expected?
(In reply to Adam Williamson from comment #12) > fwiw, I tested the F19 update, Thanks for checking it! > and it retrieved the file with --no-sessionid . > still gave 400 without it. i believe this is as expected? Yes, the cause is now tracked as bug 1104597.
curl-7.32.0-11.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report.
curl-7.29.0-20.fc19 has been pushed to the Fedora 19 stable repository. If problems still persist, please make note of it in this bug report.