Description of problem: When the BOINC client tries to contact the World Community Grid scheduler, you get the error "Scheduler request failed: SSL connect error" Version-Release number of selected component (if applicable): boinc-client-5.10.45-14.20080315svn.fc9.x86_64 How reproducible: Always Steps to Reproduce: 1. Install boinc-client 2. Try to attach to World Community Grid project Actual results: Scheduler request failed: SSL connect error Expected results: Project should attach. Additional info: The machine in question connects through a non-authenticated Squid proxy. I've turned on debug in BOINC and got: 05-Jun-2008 15:18:18 [World Community Grid] Sending scheduler request: Requested by user. Requesting 0 seconds of work, reporting 0 completed tasks 05-Jun-2008 15:18:18 [---] [http_debug] HTTP_OP::init_post(): https://secure.worldcommunitygrid.org/boinc/wcg_cgi/fcgi 05-Jun-2008 15:18:19 [---] [http_debug] [ID#0] info: About to connect() to proxy proxy.pace.co.uk port 8080 (#0) 05-Jun-2008 15:18:19 [---] [http_debug] [ID#0] info: Trying 136.170.144.1... 05-Jun-2008 15:18:19 [---] [http_debug] [ID#0] info: Connected to proxy.pace.co. uk (136.170.144.1) port 8080 (#0) 05-Jun-2008 15:18:19 [---] [http_debug] [ID#0] info: Establish HTTP proxy tunnel to secure.worldcommunitygrid.org:443 05-Jun-2008 15:18:19 [---] [http_debug] [ID#0] Sent header to server: CONNECT secure.worldcommunitygrid.org:443 HTTP/1.0 Host: secure.worldcommunitygrid.org:443 User-Agent: BOINC client (x86_64-pc-linux-gnu 5.10.45) Proxy-Connection: Keep-Alive Content-Type: application/x-www-form-urlencoded 05-Jun-2008 15:18:19 [---] [http_debug] [ID#0] Received header from server: HTTP/1.0 200 Connection established 05-Jun-2008 15:18:19 [---] [http_debug] [ID#0] Received header from server: 05-Jun-2008 15:18:19 [---] [http_debug] [ID#0] info: Proxy replied OK to CONNECT request 05-Jun-2008 15:18:20 [---] [http_debug] [ID#0] info: CAfile: ca-bundle.crt CApath: none 05-Jun-2008 15:18:22 [---] [http_debug] [ID#0] info: SSL connection using SSL_RSA_WITH_RC4_128_MD5 05-Jun-2008 15:18:22 [---] [http_debug] [ID#0] info: Server certificate: 05-Jun-2008 15:18:22 [---] [http_debug] [ID#0] info: subject: CN=secure.worldcommunitygrid.org,OU="MCS Division, Argonne National Laboratory",O=Argonne National Laboratory,C=US 05-Jun-2008 15:18:22 [---] [http_debug] [ID#0] info: start date: Oct 04 21:06 05-Jun-2008 15:18:22 [---] [http_debug] [ID#0] info: expire date: Oct 15 21:38:33 2008 GMT 05-Jun-2008 15:18:22 [---] [http_debug] [ID#0] info: common name: secure.worldcommunitygrid.org 05-Jun-2008 15:18:22 [---] [http_debug] [ID#0] info: issuer: CN=Entrust.net Secure Server Certification Authority,OU=(c) 1999 Entrust.net Limited,OU=www.entrust.net/CPS incorp. by ref. (limits liab.),O=Entrust.net,C=US 05-Jun-2008 15:18:22 [---] [http_debug] [ID#0] info: Connected to proxy.pace.co.uk (136.170.144.1) port 8080 (#0) 05-Jun-2008 15:18:23 [---] [http_debug] [ID#0] info: CAfile: ca-bundle.crt CApath: none 05-Jun-2008 15:18:24 [---] [http_debug] [ID#0] info: NSS error -12250 05-Jun-2008 15:18:24 [---] [http_debug] [ID#0] info: Expire cleared 05-Jun-2008 15:18:24 [---] [http_debug] [ID#0] info: Connection #0 to host proxy.pace.co.uk left intact 05-Jun-2008 15:18:24 [---] [http_debug] HTTP error: SSL connect error 05-Jun-2008 15:18:24 [World Community Grid] Scheduler request failed: SSL connect error If you use the BOINC client available at boinc.berkeley.edu, namely: boinc_ubuntu_5.10.45_x86_64-pc-linux-gnu.sh the problem is not evident, and BOINC can contact the WCG scheduler. It's interesting to note that the stock version from Berkeley uses OpenSSL and works 05-Jun-2008 15:34:32 [---] Libraries: libcurl/7.18.0 OpenSSL/0.9.8g zlib/1.2.3 c -ares/1.5.1 and the Fedora one uses NSS and doesn't 05-Jun-2008 15:18:17 [---] Libraries: libcurl/7.18.1 NSS/3.12 Beta 3 zlib/1.2.3 libidn/0.6.14
Strange, I just did a fresh install and fetched work from WCG without any problems. What version of libcurl and openssl do you have installed? What is the output of "curl-config --ca --features"? Does the situation change if you copy ca-bundle.crt (see the attached one -- not the provided in /etc/...) to /var/lib/boinc?
Created attachment 308524 [details] Ca-bundle.crt from upstream.
$ rpm -q libcurl openssl libcurl-7.18.1-1.fc9.x86_64 openssl-0.9.8g-9.fc9.x86_64 openssl-0.9.8g-9.fc9.i686 $ curl-config --ca --features /etc/pki/tls/certs/ca-bundle.crt SSL IPv6 libz IDN Copying your ca-bundle.crt into my data directory (which is not /var/lib/boinc) doesn't help. Does the file need to be there, or should it be in the data directory I actually use?
Oh, FWIW, various threads in forums etc. I've found about similar issues seem to suggest this issue only shows up on x86_64 systems - like mine.
>Does the file need to be there, or should it be in the data >directory I actually use? It should be in the data directory -- which is /var/lib/boinc if you use the init script to manage boinc. Please try starting BOINC as "service boinc-client start" to be sure there are no other problems like wrong permissions (but I doubt this is the reason). >I've found about similar issues seem to >suggest this issue only shows up on x86_64 systems - like mine. I've tested on x86_64 too so the architecture shouldn't matter. Can you try connecting *without* proxy? The problem seems to be in the remote certificate -- NSS error -12250 says: "SSL received a malformed Alert record." -- see http://www.mozilla.org/projects/security/pki/nss/ref/ssl/sslerr.html Can you try to choose different project (not WCG), if it works?
Ah, my data directory is a local /export/home directory, owned by me, and I run boinc as me, so there are no permissions issues. I can't easily try without the proxy, my machine can't get to the internet without the proxy. I'll see if I can get a hole out of the firewall to test proxyless... As for trying a different project, I got the impression WCG was the only (or perhaps few) that used HTTPS. This machine already does Seti, Rosetta and Climate Prediction, and contacts/fetches correctly for those projects.
Hm, could you try installing an earlier curl version which doesn't use nss? The latest non-nss build can be downloaded from here: http://koji.fedoraproject.org/koji/buildinfo?buildID=13319 Just install it via "rpm -Uvh --force curl-7.16.4-2.fc8.x86_64.rpm" and try again.
(In reply to comment #7) > Hm, could you try installing an earlier curl version which doesn't use nss? The > latest non-nss build can be downloaded from here: > > http://koji.fedoraproject.org/koji/buildinfo?buildID=13319 > > Just install it via "rpm -Uvh --force curl-7.16.4-2.fc8.x86_64.rpm" and try again. Hmm, that's not so feasible, as that requires libssl.so.6, not the libssl.so.7 that f9 has. Also curl/curl-devel from f8 conflict with lincurl/libcurl-devel on f9. I will try to set up an f9 machine that doesn't use a proxy, to narrow it down...
Well, a stock f9 machine that doesn't require a proxy to get to the Internet _does_ work, so that narrows the problem down... [It also doesn't require a ca-bundle.crt in the data directory]
OK, so...I setup a local proxy (squid) and I'm experiencing the same behavior on both F9/F8. When downgrading to curl-7.16.4-2 on F8, it starts working, any later version (which uses NSS) doesn't work. I'm changing the component to nss as I'd like the NSS maintainters to look at it whether this is a NSS bug or not.
On June 11 updates packages got posted for Fedora 9 for both nspr and nss. Does updating those help you? I am testing on F9 i386: nspr-4.7.1-0.9.1.fc9.i386 squid-3.0.STABLE6-1.fc9.i386 curl-7.18.1-1.fc9.i386 nss-3.12.0.3-0.9.1.fc9.i386 curl --proxytunnel --proxy 127.0.0.1:3128 https://secure.worldcommunitygrid.org This seems to work for me and gives me a raw html page. Milos, can you please tell me more about the environment you used to reproduce the problem? - did you test on i386 or x86_64 ? - what is the curl command line you used for testing?
(In reply to comment #11) > On June 11 updates packages got posted for Fedora 9 for both nspr and nss. > Does updating those help you? > > I am testing on F9 i386: > > nspr-4.7.1-0.9.1.fc9.i386 > squid-3.0.STABLE6-1.fc9.i386 > curl-7.18.1-1.fc9.i386 > nss-3.12.0.3-0.9.1.fc9.i386 With: $ rpm -q nspr curl nss libcurl boinc-client nspr-4.7.1-0.9.1.fc9.x86_64 nspr-4.7.1-0.9.1.fc9.i386 curl-7.18.1-1.fc9.x86_64 nss-3.12.0.3-0.9.1.fc9.x86_64 nss-3.12.0.3-0.9.1.fc9.i386 libcurl-7.18.1-1.fc9.x86_64 boinc-client-5.10.45-14.20080315svn.fc9.x86_64 the problem is still evident. I am connecting through a squid 2.6.STABLE6 server, FWIW. > curl --proxytunnel --proxy 127.0.0.1:3128 https://secure.worldcommunitygrid.org For me it's true that if I do an equivalent command I also get HTML from https://secure.worldcommunitygrid.org/. Could it be a problem that only occurs with libcurl?
Created attachment 309667 [details] ltrace -C -s 2000 output WITH proxy.
> curl --proxytunnel --proxy 127.0.0.1:3128 https://secure.worldcommunitygrid.org > > This seems to work for me and gives me a raw html page. Yes, this works. > Milos, can you please tell me more about the environment you used to reproduce > the problem? > - did you test on i386 or x86_64 ? Both (sorry, I forgot to change the HW in the report after confirming on i386). > - what is the curl command line you used for testing? I didn't use any particular curl command, just BOINC via GUI (see its output posted in the first comment). Is there any way how to debug the usage of nss? I'm attaching the output of `lstrace -C -s 2000` when using proxy (boinc.ltrace) and without it (boincok.ltrace).
Created attachment 309668 [details] ltrace -C -s 2000 output WITHOUT proxy.
I've narrowed the problem a little bit. This only happens when using curl_multi_perform() over a proxy. curl_easy_perform() over a proxy works fine.
Changing component to curl
This patch should fix it: diff -u --recursive curl-7.18.1.orig/lib/nss.c curl-7.18.1/lib/nss.c --- curl-7.18.1.orig/lib/nss.c 2008-02-20 04:56:26.000000000 -0500 +++ curl-7.18.1/lib/nss.c 2008-06-17 13:39:46.000000000 -0400 @@ -808,6 +808,9 @@ curlerr = CURLE_SSL_CONNECT_ERROR; + if (connssl->state == ssl_connection_complete) + return CURLE_OK; + /* FIXME. NSS doesn't support multiple databases open at the same time. */ if(!initialized) { initialized = 1;
Fixed in rawhide. F9 updates are out in a while. Thanks!
curl-7.18.2-1.fc9 has been submitted as an update for Fedora 9
Wanted to add that I submitted this patch upstream as well.
I'm sorry but the update didn't change anything, the behavior is still the same on both i386/x86_64...are any specific tasks to be done after updating? John, does it work for you? (You can download the RPMs from http://koji.fedoraproject.org/koji/buildinfo?buildID=53003)
How are you testing? I start boinc in one xterm and run this in another: % boinc_cmd --lookup_account https://secure.worldcommunitygrid.org username password status: Success poll status: operation in progress poll status: operation in progress account key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx This failed prior to the fix.
The boinc problem I have is slightly different from this one. However, it may be all part of the same fix. If I should file a new bug please let me know. When I try to get Boinc working in F9 I get an error dialogue: "BOINC Manager is not able to connect to a BOINC client. Would you like to try to connect again?" >From the command line, I get: ~]$ boincmgr connect: Connection refused execvp(./boinc, -redirectio, -launched_by_manager, -insecure) failed with error 2! connect: Operation now in progress connect: Connection refused connect: Connection refused execvp(./boinc, -redirectio, -launched_by_manager, -insecure) failed with error 2! I had BOINC working fine in F8. Can anybody help me get BOINC working again in F9? Is this another SELinuxv annoyance? (I am set to permissive) I have yum installed boinc twice; removed once.
(In reply to comment #23) > How are you testing? I start boinc in one xterm and run this in another: > > % boinc_cmd --lookup_account https://secure.worldcommunitygrid.org username > password > status: Success > poll status: operation in progress > poll status: operation in progress > account key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > > This failed prior to the fix. >rpm -q curl curl-7.18.2-1.fc9.x86_64 >boinc_cmd --lookup_account https://secure.worldcommunitygrid.org <user> <pass> status: Success poll status: operation in progress poll status: operation in progress poll status: http error Hm -- are you sure you setup the proxy correctly? I'm asking because I just found out that running >boinc_cmd --set_proxy_settings localhost 3128 "" "" "" "" "" "" "" sets proxy settings but does NOT set to use the proxy actually. In order to use proxy be sure that the client_state.xml has: <proxy_info> <use_http_proxy/> <socks_version>5</socks_version> <socks_server_name></socks_server_name> <socks_server_port>80</socks_server_port> <http_server_name>localhost</http_server_name> <http_server_port>3128</http_server_port> <socks5_user_name></socks5_user_name> <socks5_user_passwd></socks5_user_passwd> <http_user_name></http_user_name> <http_user_passwd></http_user_passwd> </proxy_info> (The <use_http_proxy/> tag is NOT set by boinc_cmd) Maybe using the GUI is a bit easier in this case, just run boincmgr, connect to localhost using the password stored in /var/lib/gui_rpc_auth.cfg (menu Advanced->Select computer), cancel the dialog which appears immediately, set proxy in menu Advanced->Options and try to attach the project again (Tools->Attach project). Look at the tab 'Messages' for the output (or tail -f stdoutdae.txt:) BTW the debugging output can be controlled using the cc_config.xml file as described on: http://boinc.berkeley.edu/trac/wiki/ClientMessages (Be sure that you run boinc_cmd --read_cc_config or choose Advanced->Read config file otherwise the changes won't take effect until restarting the client) If you succeed to lookup the account, does attaching to the project work too? (boinc_cmd --project_attach http://www.worldcommunitygrid.org <account_key>)
(In reply to comment #24) > The boinc problem I have is slightly different from this one. However, it may > be all part of the same fix. If I should file a new bug please let me know. > > When I try to get Boinc working in F9 I get an error dialogue: > "BOINC Manager is not able to connect to a BOINC client. > Would you like to try to connect again?" > Did you follow https://fedoraproject.org/wiki/User:Mjakubicek/HowToUseBoinc ? (i.e. did you start boinc as service and use the correct password?)
I didn't follow those instructions, did everything on the command-line. I did set up the proxying right and /var/log/squid/access confirms that it is being used. I was able to attach to the project and it started downloading a ton of stuff so I killed it. All the downloads seemed to be happening over HTTP though. For the simple account lookup test can you attach the http_debug output?
Created attachment 309787 [details] Output from `boinc_cmd --lookup_account https://secure.worldcommunitygrid.org <user> <pass>` with http_debug option in cc_config.xml Of course, here it is.
(In reply to comment #28) > Created an attachment (id=309787) [edit] > Output from `boinc_cmd --lookup_account https://secure.worldcommunitygrid.org > <user> <pass>` with http_debug option in cc_config.xml Sorry if I'm being stupid, but doesn't this line of debug suggest you've not updated libcurl? 18-Jun-2008 22:14:51 [---] Libraries: libcurl/7.18.1 NSS/3.12 Beta 3 zlib/1.2.3 libidn/0.6.14 The fixed version is 7.18.2, yes?
(In reply to comment #29) > (In reply to comment #28) > > Created an attachment (id=309787) [edit] [edit] > > Output from `boinc_cmd --lookup_account https://secure.worldcommunitygrid.org > > <user> <pass>` with http_debug option in cc_config.xml > > Sorry if I'm being stupid, but doesn't this line of debug suggest you've not > updated libcurl? Or at least not restarted boinc to bring in the library version...
(In reply to comment #24) > The boinc problem I have is slightly different from this one. However, it may > be all part of the same fix. If I should file a new bug please let me know. I'd say entirely separate problem, and as comment #26 implies probably caused by you running Boinc incorrectly.
(In reply to comment #29) > (In reply to comment #28) > > Created an attachment (id=309787) [edit] [edit] > > Output from `boinc_cmd --lookup_account https://secure.worldcommunitygrid.org > > <user> <pass>` with http_debug option in cc_config.xml > > Sorry if I'm being stupid, but doesn't this line of debug suggest you've not > updated libcurl? You're not, I am stupid, of course -- I updated only curl but forgot about libcurl (I even missed that there is this separate subpackage), with the updated libcurl it works like a charm. Rob, I'm sorry for bothering you :/
Last thing: Jindrich, could you fix this in F8 too please? Thanks.
curl-7.18.2-1.fc8 has been submitted as an update for Fedora 8
Just to doubly verify, the new curl release does indeed fix my problem. Many thanks for your speedy efforts.
curl-7.18.2-1.fc8 has been pushed to the Fedora 8 stable repository. If problems still persist, please make note of it in this bug report.
curl-7.18.2-1.fc9 has been pushed to the Fedora 9 stable repository. If problems still persist, please make note of it in this bug report.