Bug 527771 - curl is unable to connect via SSL, NSS error -12229
Summary: curl is unable to connect via SSL, NSS error -12229
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: curl
Version: 11
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Kamil Dudka
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2009-10-07 16:15 UTC by Wolfram Wagner
Modified: 2017-04-18 21:21 UTC (History)
3 users (show)

Fixed In Version: 7.19.7-2.fc12
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2009-12-18 04:18:09 UTC


Attachments (Terms of Use)
Zabbix utilization of curllib... Seems to be alright. (17.88 KB, text/plain)
2009-10-08 12:54 UTC, Wolfram Wagner
no flags Details

Description Wolfram Wagner 2009-10-07 16:15:44 UTC
Description of problem:
I want to use zabbix to monitor one of our SAP JEE Servers. Zabbix is using curl to make https connections.
Our servers are running their SSL on base of a Certificatte from SAP Trust Center (https://websmp204.sap-ag.de/tcsrootcert => SAP Server CA Certificate: https://tcs.mysap.com/invoke/tc/getCert?SAPServerCA.der)
I have downloaded that file converted it via:
openssl x509 -inform der -in /root/sapcarootcert -text -fingerprint >> /etc/pki/tls/certs/ca-bundle.crt

When I try to connect via curl I get:
* About to connect() to x.x.x.com port 443 (#0)
*   Trying x.x.x.x... connected
* Connected to x.x.x.com (x.x.x.x) port 443 (#0)
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* NSS error -12229
* Closing connection #0
* SSL connect error
curl: (35) SSL connect error

nss 12229 means that nss has received a handshake message that it does not understand...

curl can connect to another similar server that is running a thawte certificate.

I have tested with wget => it is using my new CA certificate and works. 

Also 
openssl s_client -connect x.x.x.com:443 
works fine.


I have made that check and removed the new ca certificate => wget failed to connect...

I have googled all day long and tried many things. I think this is a bug... Is there a way to make curl more verbose (beside -v or -trace)?

Comment 1 Kamil Dudka 2009-10-07 17:19:18 UTC
Please attach the CA certificate in PEM format and paste a link to a publicly available https server you are unable to connect. I'll need to reproduce it myself before writing a fix.

libcurl does not implement SSL itself, it uses NSS libraries. You can use nss' debugging equipment instead (you need to recompile NSS with debugging support first):

$ SSLTRACE=3 curl --verbose -o/dev/null https://tcs.mysap.com/invoke/tc/getCert?SAPServerCA.der
* About to connect() to tcs.mysap.com port 443 (#0)
*   Trying 194.39.131.212... connected
* Connected to tcs.mysap.com (194.39.131.212) port 443 (#0)
* Initializing NSS with certpath: /etc/pki/nssdb
SSL: tracing set to 3
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
13397: SSL[29486096]: sending client-hello
13397: SSL3[29486096]: send client_hello handshake
13397: SSL3[29486096] SendRecord type: handshake  (22) nIn=97
13397: SSL[29486096]: Send record (plain text) [Len: 97]
   01 00 00 5d 03 01 4a cc cd 78 d6 12 6d c1 9b 62   ...]..J..x..m..b
   60 24 0f 1c 4c eb 07 d4 c6 e7 96 82 b5 be 05 41   `$..L..........A
   02 a3 5a 31 0c e8 00 00 1e 00 39 00 38 00 35 00   ..Z1......9.8.5.
   33 00 32 00 04 00 2f fe ff 00 0a fe fe 00 09 00   3.2.../.........
   64 00 62 00 03 00 06 01 00 00 16 00 00 00 12 00   d.b.............
   10 00 00 0d 74 63 73 2e 6d 79 73 61 70 2e 63 6f   ....tcs.mysap.co
   6d                                                m
13397: SSL3[29486096]: handle alert record
* NSS error -12229
* Closing connection #0
* SSL connect error

curl: (35) SSL connect error

Comment 2 Wolfram Wagner 2009-10-07 18:19:15 UTC
You are real quick in answering my request! Thank you!!!

I am not in my office anymore, but the link, that you have used in your example should download the CA Certificate.

We have no public reachable server with that certificate, because nobody has installed SAP's CA Root certificate in their browsers... I think at least...

You seem to have the same problem with SAPs cert download server...Maybe its not the certificate maybe its their server?

But, if you want to test, you will need the root certificate from cybertrust/verizon business.. I haven't found it yet...

I will try to post my log with SSLTRACE=3 as soon as possible... please tell me if I can do/test something else for you!

Comment 3 Wolfram Wagner 2009-10-07 18:39:36 UTC
I have connected to my office and tried your SSLTRACE=3 command with my server: There was not a single character more than before...

So I tried your command:
SSLTRACE=3 curl --verbose -o/dev/null  https://tcs.mysap.com/invoke/tc/getCert?SAPServerCA.der
* About to connect() to proxy proxy.x.x.com port x (#0)
*   Trying y.y.y.y... connected
* Connected to proxy.x.x.com (y.y.y.y) port x (#0)
* Establish HTTP proxy tunnel to tcs.mysap.com:443
> CONNECT tcs.mysap.com:443 HTTP/1.1
> Host: tcs.mysap.com:443
> User-Agent: curl/7.19.6 (i386-redhat-linux-gnu) libcurl/7.19.6 NSS/3.12.4.1 Beta zlib/1.2.3 libidn/1.9 libssh2/1.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Proxy replied OK to CONNECT request
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* NSS error -12229
* Closing connection #0
* SSL connect error

curl: (35) SSL connect error

So from my side it looks different...  Perhaps because of the proxy, perhaps you have added some debug coding... 

I think if you can connect to SAPs server, my problem will be gone too...

Tell me if I can do something to help you!

Comment 4 Kamil Dudka 2009-10-07 19:06:21 UTC
Thanks for your help! It looks like a server problem to me. I am able to download the cert by forcing SSLv3:

$ curl --sslv3 --verbose -o/dev/null \
        https://tcs.mysap.com/invoke/tc/getCert?SAPServerCA.der

> ... and tried your SSLTRACE=3 command with my server:
> There was not a single character more than before...

Of course, I had to recompile the NSS libraries with -DTRACE to make this working! I am appending Elio to CC as the NSS guru.

Elio, do we have any way to turn on SSLTRACE by an ordinary user?

Comment 5 Kamil Dudka 2009-10-07 19:46:44 UTC
Nope, I blame the libcurl code, in particular the game it's playing with SSL_V2_COMPATIBLE_HELLO. The following patch fixes this bug and bug 525496 as well:

diff -up nss.c.orig nss.c
--- nss.c.orig
+++ nss.c
@@ -1072,9 +1072,6 @@ CURLcode Curl_nss_connect(struct connect
   if(SSL_OptionSet(model, SSL_ENABLE_TLS, tlsv1) != SECSuccess)
     goto error;

-  if(SSL_OptionSet(model, SSL_V2_COMPATIBLE_HELLO, ssl2) != SECSuccess)
-    goto error;
-
   /* enable all ciphers from enable_ciphers_by_default */
   cipher_to_enable = enable_ciphers_by_default;
   while (SSL_NULL_WITH_NULL_NULL != *cipher_to_enable) {

Rob, could you please do a review? What was the original purpose of the code? Thanks in advance!

Comment 6 Kamil Dudka 2009-10-07 20:06:18 UTC
Wolfram, could you please check if the prepared scratch build solves the problem for you?

http://koji.fedoraproject.org/koji/taskinfo?taskID=1733961

Comment 7 Wolfram Wagner 2009-10-07 20:47:17 UTC
Kamil, Thank you a lot! It works as expected now!

I am impressed how quick and good you could help me. We wait weeks for a small fix from SAP and get an open source bug fixed on the same day. 

Thank you again and tell everybody how good you have been!

Comment 8 Kamil Dudka 2009-10-07 21:24:40 UTC
Wolfram, you're welcome.

The following thread contains some details provided by Guenter Knauf:
http://curl.haxx.se/mail/lib-2009-10/0071.html

Comment 9 Elio Maldonado Batiz 2009-10-07 21:45:40 UTC
(In reply to comment #4)
> Elio, do we have any way to turn on SSLTRACE by an ordinary user?  
Kamil, According to
https://developer.mozilla.org/en/NSS_reference/NSS_environment_variables
you can set the SSLTRACE environment variable but the code must be built with TRACE defined to use this functionality.

Comment 10 Kamil Dudka 2009-10-07 21:58:51 UTC
Elio, that's no problem for me (as developer). I've already attached the output of SSLTRACE in comment #1. But is there any easy way to do that for "normal" Fedora users? Something similar to installing debuginfo or whatever?

Comment 11 Elio Maldonado Batiz 2009-10-07 22:25:10 UTC
Unfortunately, the code that checks the SSLTRACE environment variable is conditionally compiled on #ifdef TRACE. As far as I can tell thee is no way for the user to trace unless they have also access to a TRACE-enabled build.

Comment 12 Kamil Dudka 2009-10-07 22:41:34 UTC
Elio, thanks for clarifying it! This particular bug seems to be already tracked down. But it's true we can always create a TRACE-enabled build for users interested to help with debugging.

Rob, FYI original purpose of the code was support for "Server Name Indication" (RFC 4366), now discussed at curl-library mailing list...

Comment 13 Wolfram Wagner 2009-10-08 10:01:28 UTC
It's me again: curl is working fine, I think now its about ZABBIX...

Whatever ssl URL I try with curl, wget, openSSL I get results as expected.

The behavior of zabbix has changed too: Before I had constant failures, now I get sporadic SSL connection fails. Let me explain:

I am monitoring a complex application running on some servers. They all are together one application:

Setup:
The app consists of two different web applications, running on two JEE Servers managed by a load balancer.
I want to monitor these two webapps on these two servers and accessed via their outer (loadbalnacer) URL, to check the response time influence and availability of our infrastructure.

So I have created 3 hosts, and web checks (walkthroughs) for both webapps:

Host1: LoadBalancer
  App: MyApp
  Scenario: App1
  Scenario: App2

Host2: JEE Server1
  App: MyApp
  Scenario: App1
  Scenario: App2

Host3: JEE Server2
  App: MyApp
  Scenario: App1
  Scenario: App2

Host1 is running on a certificate from thwate and made never problems.
Host 2 and 3 are running the SAP certificate and made curl show his problems that I got fixed last night. (see above)

Since I have installed the test version of curl and libcurl (at least I could not notice it before) I have the problems on host 2 and 3:

One of the Scenarios is working and the other one is not. Error message on Zabbix screen is: 'Failed on "Login" [1 of 1]  Error: SSL connect error'

In log file (trace level)"Error doing curl_easy_perform [SSL connect error]"

I have double checked the URL for typos and found that it should work.
I have created new single action test scenarios and it showed the same behavior.
Even if I have 3 and 4 scenarios for one host and found, that seemingly every second scenario fails. Maybe is there a problem with closing the SSL connections?

Please tell me, if I can help and test something!

Thank you! 
Wolfram

Comment 14 Kamil Dudka 2009-10-08 10:20:26 UTC
Try to set CURLOPT_VERBOSE to see what exactly happens:
http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTVERBOSE

Please also try set CURLOPT_SSLVERSION to CURL_SSLVERSION_SSLv3. I think you'll need to reduce the problem to a simple example using libcurl only to get some more help from libcurl hackers.

Comment 15 Wolfram Wagner 2009-10-08 10:58:22 UTC
I understand the problem, but my C/C++ times are 10 years behind me.

Maybe we need a Zabbix developper to help in this case, or maybe you have a small test code that I can compile and run...

As I said, I will help, if I am able to...

Comment 16 Kamil Dudka 2009-10-08 11:08:41 UTC
(In reply to comment #15)
> Maybe we need a Zabbix developper to help in this case, or maybe you have a
> small test code that I can compile and run...

Sure. You can pick one of the SSL examples here:
http://curl.haxx.se/libcurl/c/example.html

Comment 17 Wolfram Wagner 2009-10-08 11:37:17 UTC
Thanks... I will try my best to find the error. I guess its in Zabbix... curl standalone does close its connections, I could verify it with a sniffer...

Zabbix supports multiple http calls in one session... So maybe...

Comment 18 Wolfram Wagner 2009-10-08 12:54:50 UTC
Created attachment 364110 [details]
Zabbix utilization of curllib... Seems to be alright.

This is the current stable original code from Zabbix: src/zabbix_server/httppoller/httptest.c (http://sourceforge.net/projects/zabbix/files/ZABBIX%20Latest%20Stable/1.6.6/zabbix-1.6.6.tar.gz/download)
As much as I see, it is programmed as it is defined in your curl examples.

Comment 19 Wolfram Wagner 2009-10-08 13:03:17 UTC
I wonder how it could be that seemingly every second check fails. Please remember that this is the case only for the servers, that could not be contacted via curl until yesterday.
The other server (also a SAP JEE server of same version, but with a different, official Thawte certificate) does not show this behavior. So there is a likelihood that its reason is to be find close to the issue you fixed yesterday...

Your libCurl seems to be really easy to use. It has a straightforward interface with less things that you can make wrong... I hope you can find the needed hint in all my tests and thoughts.
Unfortunally I am not able to write and install my own test application. At least not here and now, because there are different stones laying in my way. (The server in doubt is the only linux machine I have in access at work and the access is via SSH only.) I try to write and compile the example/test at home, but if you can make progress before I succeed, it would be much quicker...

Thank you!

Comment 20 Kamil Dudka 2009-10-08 16:59:35 UTC
Well, I am not sure if the problem is still related to the original bug report or not. Could you please try the referred scratch build of zabbix together with the *original* libcurl from F-11 distribution? Please catch the standard error output to file and attach.

http://koji.fedoraproject.org/koji/taskinfo?taskID=1736003

Are you able to access the same object via different https client in a loop indefinitely?

Comment 21 Wolfram Wagner 2009-10-08 20:04:13 UTC
Hello Kamil, 
It looks good: 
This is what I did: I downloaded the improved zabbix from koji and upgraded my server. Removed curl and libcurl (the version from yesterday). Then I installed the former curl via yum.

Then I restartet the zabbix services and voila: All tests are running and the output file shows what I want to see, when I enable zabbix' trace mode. (Such a trace mode for curl via a single switch would be a nice thing too...)

Now all my checks are online again and work!

Seems as if not curl was to blame, it was the zabbix implementation? I will report tomorrow, if it is stable... :-)

Thank you a lot!!!

(I have written my first C-program since 1996 and got it working, but you bypassed me... I would not have found the reason I think.... Bute I have learned a lot...)
Good night!

Comment 22 Kamil Dudka 2009-10-08 21:45:20 UTC
Nope, it was indeed a server's bug which is suppressed by sort of workaround within Firefox/xulrunner. I've tried to summarize some details here:

http://permalink.gmane.org/gmane.comp.web.curl.library/25367
http://permalink.gmane.org/gmane.comp.web.curl.library/25371

We'll consider implementing such workaround in libcurl, but it's not so easy and definitely won't get into curl-7.19.7. If it will be rejected by curl upstream, you can ask zabbix developers to implement a way to force SSLv3 instead.

Comment 23 Wolfram Wagner 2009-10-09 20:24:10 UTC
Mh. ok... There are some servers here having this problem, some not. One of them, having it, is a server of an external professional hoster... So its not just SAP...

My situation right now is that I have a system that works, if I ignore the many details in log file. My trust in the whole thing is coming back and all other minor problems would have been avoidable, if I would have read the 300 page manual before...

I understand that those workarounds blow up your code and make it harder to maintain, but on the other hand, your library is used in many tools around and there is still a number of servers out there which do require it.

Zabbix is a tool for monitoring and a switch to force SSLv3 does not belong into that tool. It would make live more complex at the wrong spot...
I hope that it will be accepted for upstream...

Thanks for your quick and excellent support!

Comment 24 Kamil Dudka 2009-10-14 19:07:06 UTC
patch ready for review:

http://permalink.gmane.org/gmane.comp.web.curl.library/25440

Comment 25 Kamil Dudka 2009-11-04 23:20:52 UTC
a new version of the patch:

http://permalink.gmane.org/gmane.comp.web.curl.library/25687

Comment 26 Kamil Dudka 2009-11-05 22:21:31 UTC
fixed in curl-7.19.7-2.fc13 for now

As a long-term solution Kaspar Brand has raised the issue at mozilla.org:

https://bugzilla.mozilla.org/show_bug.cgi?id=526806

Comment 27 Fedora Update System 2009-11-26 19:33:20 UTC
curl-7.19.7-3.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/curl-7.19.7-3.fc11

Comment 28 Fedora Update System 2009-11-26 19:33:32 UTC
curl-7.19.7-2.fc12 has been submitted as an update for Fedora 12.
http://admin.fedoraproject.org/updates/curl-7.19.7-2.fc12

Comment 29 Fedora Update System 2009-11-27 21:44:13 UTC
curl-7.19.7-2.fc12 has been pushed to the Fedora 12 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update curl'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F12/FEDORA-2009-12235

Comment 30 Fedora Update System 2009-11-27 21:49:06 UTC
curl-7.19.7-3.fc11 has been pushed to the Fedora 11 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update curl'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-12245

Comment 31 Fedora Update System 2009-12-01 04:21:16 UTC
curl-7.19.7-3.fc11 has been pushed to the Fedora 11 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update curl'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-12245

Comment 32 Fedora Update System 2009-12-01 04:36:31 UTC
curl-7.19.7-2.fc12 has been pushed to the Fedora 12 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update curl'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F12/FEDORA-2009-12235

Comment 33 Fedora Update System 2009-12-18 04:17:49 UTC
curl-7.19.7-3.fc11 has been pushed to the Fedora 11 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 34 Fedora Update System 2009-12-18 04:21:02 UTC
curl-7.19.7-2.fc12 has been pushed to the Fedora 12 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.