Bug 450140 - BOINC can't fetch from World Community Grid (via proxy)
Summary: BOINC can't fetch from World Community Grid (via proxy)
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: curl
Version: 9
Hardware: All
OS: Linux
low
medium
Target Milestone: ---
Assignee: Jindrich Novy
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2008-06-05 15:07 UTC by John Beranek
Modified: 2013-07-02 23:29 UTC (History)
4 users (show)

Fixed In Version: curl-7.18.2-1.fc9
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2008-06-18 21:23:43 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
Ca-bundle.crt from upstream. (232.52 KB, text/plain)
2008-06-06 11:28 UTC, Milos Jakubicek
no flags Details
ltrace -C -s 2000 output WITH proxy. (882.94 KB, text/plain)
2008-06-17 19:48 UTC, Milos Jakubicek
no flags Details
ltrace -C -s 2000 output WITHOUT proxy. (5.55 MB, text/plain)
2008-06-17 19:49 UTC, Milos Jakubicek
no flags Details
Output from `boinc_cmd --lookup_account https://secure.worldcommunitygrid.org <user> <pass>` with http_debug option in cc_config.xml (3.93 KB, text/plain)
2008-06-18 20:16 UTC, Milos Jakubicek
no flags Details

Description John Beranek 2008-06-05 15:07:11 UTC
Description of problem:

When the BOINC client tries to contact the World Community Grid scheduler, you
get the error "Scheduler request failed: SSL connect error"

Version-Release number of selected component (if applicable):

boinc-client-5.10.45-14.20080315svn.fc9.x86_64

How reproducible:

Always

Steps to Reproduce:
1. Install boinc-client
2. Try to attach to World Community Grid project
  
Actual results:

Scheduler request failed: SSL connect error

Expected results:

Project should attach.

Additional info:

The machine in question connects through a non-authenticated Squid proxy. I've
turned on debug in BOINC and got:

05-Jun-2008 15:18:18 [World Community Grid] Sending scheduler request: Requested
by user.  Requesting 0 seconds of work, reporting 0 completed tasks
05-Jun-2008 15:18:18 [---] [http_debug] HTTP_OP::init_post():
https://secure.worldcommunitygrid.org/boinc/wcg_cgi/fcgi
05-Jun-2008 15:18:19 [---] [http_debug] [ID#0] info: About to connect() to proxy
proxy.pace.co.uk port 8080 (#0)
05-Jun-2008 15:18:19 [---] [http_debug] [ID#0] info:   Trying 136.170.144.1... 
05-Jun-2008 15:18:19 [---] [http_debug] [ID#0] info: Connected to proxy.pace.co.
uk (136.170.144.1) port 8080 (#0)
05-Jun-2008 15:18:19 [---] [http_debug] [ID#0] info: Establish HTTP proxy tunnel
to secure.worldcommunitygrid.org:443
05-Jun-2008 15:18:19 [---] [http_debug] [ID#0] Sent header to server: CONNECT
secure.worldcommunitygrid.org:443 HTTP/1.0
Host: secure.worldcommunitygrid.org:443
User-Agent: BOINC client (x86_64-pc-linux-gnu 5.10.45)
Proxy-Connection: Keep-Alive
Content-Type: application/x-www-form-urlencoded

05-Jun-2008 15:18:19 [---] [http_debug] [ID#0] Received header from server:
HTTP/1.0 200 Connection established
05-Jun-2008 15:18:19 [---] [http_debug] [ID#0] Received header from server:
05-Jun-2008 15:18:19 [---] [http_debug] [ID#0] info: Proxy replied OK to CONNECT
request
05-Jun-2008 15:18:20 [---] [http_debug] [ID#0] info:   CAfile: ca-bundle.crt
  CApath: none
05-Jun-2008 15:18:22 [---] [http_debug] [ID#0] info: SSL connection using
SSL_RSA_WITH_RC4_128_MD5
05-Jun-2008 15:18:22 [---] [http_debug] [ID#0] info: Server certificate:
05-Jun-2008 15:18:22 [---] [http_debug] [ID#0] info:    subject:
CN=secure.worldcommunitygrid.org,OU="MCS Division, Argonne National
Laboratory",O=Argonne National Laboratory,C=US
05-Jun-2008 15:18:22 [---] [http_debug] [ID#0] info:    start date: Oct 04 21:06
05-Jun-2008 15:18:22 [---] [http_debug] [ID#0] info:    expire date: Oct 15
21:38:33 2008 GMT
05-Jun-2008 15:18:22 [---] [http_debug] [ID#0] info:    common name:
secure.worldcommunitygrid.org
05-Jun-2008 15:18:22 [---] [http_debug] [ID#0] info:    issuer: CN=Entrust.net
Secure Server Certification Authority,OU=(c) 1999 Entrust.net
Limited,OU=www.entrust.net/CPS incorp. by ref. (limits liab.),O=Entrust.net,C=US
05-Jun-2008 15:18:22 [---] [http_debug] [ID#0] info: Connected to
proxy.pace.co.uk (136.170.144.1) port 8080 (#0)
05-Jun-2008 15:18:23 [---] [http_debug] [ID#0] info:   CAfile: ca-bundle.crt
  CApath: none
05-Jun-2008 15:18:24 [---] [http_debug] [ID#0] info: NSS error -12250
05-Jun-2008 15:18:24 [---] [http_debug] [ID#0] info: Expire cleared
05-Jun-2008 15:18:24 [---] [http_debug] [ID#0] info: Connection #0 to host
proxy.pace.co.uk left intact
05-Jun-2008 15:18:24 [---] [http_debug] HTTP error: SSL connect error
05-Jun-2008 15:18:24 [World Community Grid] Scheduler request failed: SSL
connect error


If you use the BOINC client available at boinc.berkeley.edu, namely:

boinc_ubuntu_5.10.45_x86_64-pc-linux-gnu.sh

the problem is not evident, and BOINC can contact the WCG scheduler.

It's interesting to note that the stock version from Berkeley uses OpenSSL and works

05-Jun-2008 15:34:32 [---] Libraries: libcurl/7.18.0 OpenSSL/0.9.8g zlib/1.2.3 c
-ares/1.5.1

and the Fedora one uses NSS and doesn't

05-Jun-2008 15:18:17 [---] Libraries: libcurl/7.18.1 NSS/3.12 Beta 3 zlib/1.2.3
libidn/0.6.14

Comment 1 Milos Jakubicek 2008-06-06 11:25:51 UTC
Strange, I just did a fresh install and fetched work from WCG without any problems. 

What version of libcurl and openssl do you have installed?
What is the output of "curl-config --ca --features"?
Does the situation change if you copy ca-bundle.crt (see the attached one -- not
the provided in /etc/...) to /var/lib/boinc?

Comment 2 Milos Jakubicek 2008-06-06 11:28:01 UTC
Created attachment 308524 [details]
Ca-bundle.crt from upstream.

Comment 3 John Beranek 2008-06-06 13:20:10 UTC
$ rpm -q libcurl openssl
libcurl-7.18.1-1.fc9.x86_64
openssl-0.9.8g-9.fc9.x86_64
openssl-0.9.8g-9.fc9.i686

$ curl-config --ca --features
/etc/pki/tls/certs/ca-bundle.crt
SSL
IPv6
libz
IDN

Copying your ca-bundle.crt into my data directory (which is not /var/lib/boinc)
doesn't help. Does the file need to be there, or should it be in the data
directory I actually use?


Comment 4 John Beranek 2008-06-06 13:23:14 UTC
Oh, FWIW, various threads in forums etc. I've found about similar issues seem to
suggest this issue only shows up on x86_64 systems - like mine.


Comment 5 Milos Jakubicek 2008-06-06 13:44:10 UTC
>Does the file need to be there, or should it be in the data
>directory I actually use?

It should be in the data directory -- which is /var/lib/boinc if you use the
init script to manage boinc. Please try starting BOINC as "service boinc-client
start" to be sure there are no other problems like wrong permissions (but I
doubt this is the reason).

>I've found about similar issues seem to
>suggest this issue only shows up on x86_64 systems - like mine.

I've tested on x86_64 too so the architecture shouldn't matter.

Can you try connecting *without* proxy?
The problem seems to be in the remote certificate -- NSS error -12250 says: "SSL
received a malformed Alert record." -- see
http://www.mozilla.org/projects/security/pki/nss/ref/ssl/sslerr.html

Can you try to choose different project (not WCG), if it works?

Comment 6 John Beranek 2008-06-06 14:03:47 UTC
Ah, my data directory is a local /export/home directory, owned by me, and I run
boinc as me, so there are no permissions issues.

I can't easily try without the proxy, my machine can't get to the internet
without the proxy. I'll see if I can get a hole out of the firewall to test
proxyless...

As for trying a different project, I got the impression WCG was the only (or
perhaps few) that used HTTPS. This machine already does Seti, Rosetta and
Climate Prediction, and contacts/fetches correctly for those projects.


Comment 7 Milos Jakubicek 2008-06-06 15:58:11 UTC
Hm, could you try installing an earlier curl version which doesn't use nss? The
latest non-nss build can be downloaded from here:

http://koji.fedoraproject.org/koji/buildinfo?buildID=13319

Just install it via "rpm -Uvh --force curl-7.16.4-2.fc8.x86_64.rpm" and try again.

Comment 8 John Beranek 2008-06-06 18:38:42 UTC
(In reply to comment #7)
> Hm, could you try installing an earlier curl version which doesn't use nss? The
> latest non-nss build can be downloaded from here:
> 
> http://koji.fedoraproject.org/koji/buildinfo?buildID=13319
> 
> Just install it via "rpm -Uvh --force curl-7.16.4-2.fc8.x86_64.rpm" and try again.

Hmm, that's not so feasible, as that requires libssl.so.6, not the libssl.so.7
that f9 has. Also curl/curl-devel from f8 conflict with lincurl/libcurl-devel on f9.

I will try to set up an f9 machine that doesn't use a proxy, to narrow it down...


Comment 9 John Beranek 2008-06-09 13:06:53 UTC
Well, a stock f9 machine that doesn't require a proxy to get to the Internet
_does_ work, so that narrows the problem down...

[It also doesn't require a ca-bundle.crt in the data directory]


Comment 10 Milos Jakubicek 2008-06-17 14:06:10 UTC
OK, so...I setup a local proxy (squid) and I'm experiencing the same behavior on
both F9/F8. When downgrading to curl-7.16.4-2 on F8, it starts working, any
later version (which uses NSS) doesn't work. I'm changing the component to nss
as I'd like the NSS maintainters to look at it whether this is a NSS bug or not.

Comment 11 Kai Engert (:kaie) (inactive account) 2008-06-17 15:26:56 UTC
On June 11 updates packages got posted for Fedora 9 for both nspr and nss.
Does updating those help you?

I am testing on F9 i386:

nspr-4.7.1-0.9.1.fc9.i386
squid-3.0.STABLE6-1.fc9.i386
curl-7.18.1-1.fc9.i386
nss-3.12.0.3-0.9.1.fc9.i386

curl --proxytunnel --proxy 127.0.0.1:3128 https://secure.worldcommunitygrid.org

This seems to work for me and gives me a raw html page.


Milos, can you please tell me more about the environment you used to reproduce
the problem?
- did you test on i386 or x86_64 ?
- what is the curl command line you used for testing?


Comment 12 John Beranek 2008-06-17 16:12:22 UTC
(In reply to comment #11)
> On June 11 updates packages got posted for Fedora 9 for both nspr and nss.
> Does updating those help you?
> 
> I am testing on F9 i386:
> 
> nspr-4.7.1-0.9.1.fc9.i386
> squid-3.0.STABLE6-1.fc9.i386
> curl-7.18.1-1.fc9.i386
> nss-3.12.0.3-0.9.1.fc9.i386

With:

$ rpm -q nspr curl nss libcurl boinc-client
nspr-4.7.1-0.9.1.fc9.x86_64
nspr-4.7.1-0.9.1.fc9.i386
curl-7.18.1-1.fc9.x86_64
nss-3.12.0.3-0.9.1.fc9.x86_64
nss-3.12.0.3-0.9.1.fc9.i386
libcurl-7.18.1-1.fc9.x86_64
boinc-client-5.10.45-14.20080315svn.fc9.x86_64

the problem is still evident. I am connecting through a squid 2.6.STABLE6
server, FWIW.

> curl --proxytunnel --proxy 127.0.0.1:3128 https://secure.worldcommunitygrid.org

For me it's true that if I do an equivalent command I also get HTML from
https://secure.worldcommunitygrid.org/. Could it be a problem that only occurs
with libcurl?


Comment 13 Milos Jakubicek 2008-06-17 19:48:28 UTC
Created attachment 309667 [details]
ltrace -C -s 2000 output WITH proxy.

Comment 14 Milos Jakubicek 2008-06-17 19:49:12 UTC
> curl --proxytunnel --proxy 127.0.0.1:3128 https://secure.worldcommunitygrid.org
> 
> This seems to work for me and gives me a raw html page.

Yes, this works.
 
> Milos, can you please tell me more about the environment you used to reproduce
> the problem?
> - did you test on i386 or x86_64 ?

Both (sorry, I forgot to change the HW in the report after confirming on i386).

> - what is the curl command line you used for testing?

I didn't use any particular curl command, just BOINC via GUI (see its output
posted in the first comment). Is there any way how to debug the usage of nss?

I'm attaching the output of `lstrace -C -s 2000` when using proxy (boinc.ltrace)
and without it (boincok.ltrace).



Comment 15 Milos Jakubicek 2008-06-17 19:49:34 UTC
Created attachment 309668 [details]
ltrace -C -s 2000 output WITHOUT proxy.

Comment 16 Rob Crittenden 2008-06-17 20:25:43 UTC
I've narrowed the problem a little bit. This only happens when using
curl_multi_perform() over a proxy. curl_easy_perform() over a proxy works fine.

Comment 17 Rob Crittenden 2008-06-17 21:29:43 UTC
Changing component to curl

Comment 18 Rob Crittenden 2008-06-17 21:30:33 UTC
This patch should fix it:

diff -u --recursive curl-7.18.1.orig/lib/nss.c curl-7.18.1/lib/nss.c
--- curl-7.18.1.orig/lib/nss.c  2008-02-20 04:56:26.000000000 -0500
+++ curl-7.18.1/lib/nss.c       2008-06-17 13:39:46.000000000 -0400
@@ -808,6 +808,9 @@
 
   curlerr = CURLE_SSL_CONNECT_ERROR;
 
+  if (connssl->state == ssl_connection_complete)
+    return CURLE_OK;
+
   /* FIXME. NSS doesn't support multiple databases open at the same time. */
   if(!initialized) {
     initialized = 1;


Comment 19 Jindrich Novy 2008-06-18 06:20:12 UTC
Fixed in rawhide. F9 updates are out in a while. Thanks!

Comment 20 Fedora Update System 2008-06-18 07:08:47 UTC
curl-7.18.2-1.fc9 has been submitted as an update for Fedora 9

Comment 21 Rob Crittenden 2008-06-18 12:56:38 UTC
Wanted to add that I submitted this patch upstream as well.

Comment 22 Milos Jakubicek 2008-06-18 14:48:01 UTC
I'm sorry but the update didn't change anything, the behavior is still the same
on both i386/x86_64...are any specific tasks to be done after updating?

John, does it work for you?
(You can download the RPMs from
http://koji.fedoraproject.org/koji/buildinfo?buildID=53003)



Comment 23 Rob Crittenden 2008-06-18 15:02:30 UTC
How are you testing? I start boinc in one xterm and run this in another:

% boinc_cmd  --lookup_account https://secure.worldcommunitygrid.org username
password
status: Success
poll status: operation in progress
poll status: operation in progress
account key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

This failed prior to the fix.

Comment 24 Bill Case 2008-06-18 16:39:02 UTC
The boinc problem I have is slightly different from this one.  However, it may
be all part of the same fix.  If I should file a new bug please let me know.

When I try to get Boinc working in F9 I get an error dialogue:
"BOINC Manager is not able to connect to a BOINC client.
Would you like to try to connect again?"

>From the command line, I get:

~]$ boincmgr
connect: Connection refused
execvp(./boinc, -redirectio, -launched_by_manager, -insecure) failed
with error 2!
connect: Operation now in progress
connect: Connection refused
connect: Connection refused
execvp(./boinc, -redirectio, -launched_by_manager, -insecure) failed
with error 2!

I had BOINC working fine in F8.
Can anybody help me get BOINC working again in F9?
Is this another SELinuxv annoyance?  (I am set to permissive)

I have yum installed boinc twice; removed once.



Comment 25 Milos Jakubicek 2008-06-18 19:15:46 UTC
(In reply to comment #23)
> How are you testing? I start boinc in one xterm and run this in another:
> 
> % boinc_cmd  --lookup_account https://secure.worldcommunitygrid.org username
> password
> status: Success
> poll status: operation in progress
> poll status: operation in progress
> account key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> 
> This failed prior to the fix.

>rpm -q curl
curl-7.18.2-1.fc9.x86_64

>boinc_cmd --lookup_account https://secure.worldcommunitygrid.org <user> <pass>
status: Success
poll status: operation in progress
poll status: operation in progress
poll status: http error

Hm -- are you sure you setup the proxy correctly? I'm asking because I just
found out that running 
>boinc_cmd --set_proxy_settings localhost 3128 "" "" "" "" "" "" ""
sets proxy settings but does NOT set to use the proxy actually.

In order to use proxy be sure that the client_state.xml has:

<proxy_info>
    <use_http_proxy/>
    <socks_version>5</socks_version>
    <socks_server_name></socks_server_name>
    <socks_server_port>80</socks_server_port>
    <http_server_name>localhost</http_server_name>
    <http_server_port>3128</http_server_port>
    <socks5_user_name></socks5_user_name>
    <socks5_user_passwd></socks5_user_passwd>
    <http_user_name></http_user_name>
    <http_user_passwd></http_user_passwd>
</proxy_info>

(The <use_http_proxy/> tag is NOT set by boinc_cmd)

Maybe using the GUI is a bit easier in this case, just run boincmgr, connect to
localhost using the password stored in /var/lib/gui_rpc_auth.cfg (menu
Advanced->Select computer), cancel the dialog which appears immediately, set
proxy in menu Advanced->Options and try to attach the project again
(Tools->Attach project). Look at the tab 'Messages' for the output (or tail -f
stdoutdae.txt:)

BTW the debugging output can be controlled using the cc_config.xml file as
described on:
http://boinc.berkeley.edu/trac/wiki/ClientMessages
(Be sure that you run boinc_cmd --read_cc_config or choose Advanced->Read config
file otherwise the changes won't take effect until restarting the client)

If you succeed to lookup the account, does attaching to the project work too?
(boinc_cmd --project_attach http://www.worldcommunitygrid.org <account_key>)

Comment 26 Milos Jakubicek 2008-06-18 19:18:16 UTC
(In reply to comment #24)
> The boinc problem I have is slightly different from this one.  However, it may
> be all part of the same fix.  If I should file a new bug please let me know.
> 
> When I try to get Boinc working in F9 I get an error dialogue:
> "BOINC Manager is not able to connect to a BOINC client.
> Would you like to try to connect again?"
> 

Did you follow https://fedoraproject.org/wiki/User:Mjakubicek/HowToUseBoinc
?

(i.e. did you start boinc as service and use the correct password?)

Comment 27 Rob Crittenden 2008-06-18 19:42:41 UTC
I didn't follow those instructions, did everything on the command-line. I did
set up the proxying right and /var/log/squid/access confirms that it is being used.

I was able to attach to the project and it started downloading a ton of stuff so
I killed it. All the downloads seemed to be happening over HTTP though.

For the simple account lookup test can you attach the http_debug output?

Comment 28 Milos Jakubicek 2008-06-18 20:16:53 UTC
Created attachment 309787 [details]
Output from `boinc_cmd --lookup_account https://secure.worldcommunitygrid.org <user> <pass>` with http_debug option in cc_config.xml

Of course, here it is.

Comment 29 John Beranek 2008-06-18 20:27:51 UTC
(In reply to comment #28)
> Created an attachment (id=309787) [edit]
> Output from `boinc_cmd --lookup_account https://secure.worldcommunitygrid.org
> <user> <pass>` with http_debug option in cc_config.xml

Sorry if I'm being stupid, but doesn't this line of debug suggest you've not
updated libcurl?

18-Jun-2008 22:14:51 [---] Libraries: libcurl/7.18.1 NSS/3.12 Beta 3 zlib/1.2.3
libidn/0.6.14

The fixed version is 7.18.2, yes?


Comment 30 John Beranek 2008-06-18 20:30:55 UTC
(In reply to comment #29)
> (In reply to comment #28)
> > Created an attachment (id=309787) [edit] [edit]
> > Output from `boinc_cmd --lookup_account https://secure.worldcommunitygrid.org
> > <user> <pass>` with http_debug option in cc_config.xml
> 
> Sorry if I'm being stupid, but doesn't this line of debug suggest you've not
> updated libcurl?

Or at least not restarted boinc to bring in the library version...


Comment 31 John Beranek 2008-06-18 21:01:01 UTC
(In reply to comment #24)
> The boinc problem I have is slightly different from this one.  However, it may
> be all part of the same fix.  If I should file a new bug please let me know.

I'd say entirely separate problem, and as comment #26 implies probably caused by
you running Boinc incorrectly.


Comment 32 Milos Jakubicek 2008-06-18 21:23:43 UTC
(In reply to comment #29)
> (In reply to comment #28)
> > Created an attachment (id=309787) [edit] [edit]
> > Output from `boinc_cmd --lookup_account https://secure.worldcommunitygrid.org
> > <user> <pass>` with http_debug option in cc_config.xml
> 
> Sorry if I'm being stupid, but doesn't this line of debug suggest you've not
> updated libcurl?

You're not, I am stupid, of course -- I updated only curl but forgot about
libcurl (I even missed that there is this separate subpackage), with the updated
libcurl it works like a charm.

Rob, I'm sorry for bothering you :/

Comment 33 Milos Jakubicek 2008-06-18 21:26:16 UTC
Last thing: Jindrich, could you fix this in F8 too please? Thanks.

Comment 34 Fedora Update System 2008-06-19 06:11:46 UTC
curl-7.18.2-1.fc8 has been submitted as an update for Fedora 8

Comment 35 John Beranek 2008-06-19 10:25:51 UTC
Just to doubly verify, the new curl release does indeed fix my problem.

Many thanks for your speedy efforts.


Comment 36 Fedora Update System 2008-06-20 19:11:13 UTC
curl-7.18.2-1.fc8 has been pushed to the Fedora 8 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 37 Fedora Update System 2008-06-20 19:12:31 UTC
curl-7.18.2-1.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.