Bug 767490

Summary: --cert option (perhaps) won't work with SSL read error
Product: [Fedora] Fedora Reporter: Mamoru TASAKA <mtasaka>
Component: curlAssignee: Kamil Dudka <kdudka>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: kdudka, paul
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: curl-7.23.0-3.fc17 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-12-25 23:14:26 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Mamoru TASAKA 2011-12-14 08:09:36 UTC 
Description of problem:
With rpmfusion's "make upload", the following command is executed
but it does not succeed with rawhide curl:

[tasaka1 at localhost devel]$ rpm -q curl
curl-7.23.0-2.fc17.i686
[tasaka1 at localhost devel]$ curl -k --cert /home/tasaka1/.rpmfusion.cert -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a206 -F filename=/home/tasaka1/rpmbuild/SOURCES/cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi ; echo $?
curl: (56) SSL read: errno -12229
56

With just curl only downloaded to F-16 one, the above command succeeds.

[tasaka1 at localhost devel]$ rpm -q curl
curl-7.21.7-5.fc16.i686
[tasaka1 at localhost devel]$ curl -k --cert /home/tasaka1/.rpmfusion.cert -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a206 -F filename=/home/tasaka1/rpmbuild/SOURCES/cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi ; echo $?
Missing
0


Version-Release number of selected component (if applicable):
curl-7.23.0-2.fc17.i686

How reproducible:
100%

Steps to Reproduce:
1. See above
2.
3.
  
Actual results:
See above
Comment 1 Kamil Dudka 2011-12-14 09:25:50 UTC 
libcurl is used to upload packages all the time.  If it had been completely broken, somebody would already have notified us.  The problem must be something specific to your configuration.

By downgrading "just curl" you mean curl and libcurl?

The versions of nss* packages are the same in both cases?

Please paste the output of the following command:
$ rpm -qa nss\*

Do you have any 64bit machine around to check it for the presence of this bug?

Thanks in advance for providing more info.
Comment 2 Mamoru TASAKA 2011-12-14 09:30:20 UTC 
Well,

(In reply to comment #1)
> By downgrading "just curl" you mean curl and libcurl?
Exactly.

> The versions of nss* packages are the same in both cases?
Yes (i.e. rawhide one)

> Please paste the output of the following command:
> $ rpm -qa nss\*
I will post this when I am back.

> Do you have any 64bit machine around to check it for the presence of this bug?
I am afraid I don't have.
Comment 3 Kamil Dudka 2011-12-14 09:47:16 UTC 
Thanks, I will wait for the versions then.  Please attach also your /etc/pki/nssdb/pkcs11.txt and the output of curl -v in both cases.
Comment 4 Paul Howarth 2011-12-14 10:37:22 UTC 
I saw this *once* yesterday after seeing the original post on the rpmfusion list:

$ curl -k --cert ~/.rpmfusion.cert  -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a206 -F filename=cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi 
curl: (56) SSL read: errno -5961

(this is with my own local build of curl 7.23.1 on F-16 x86_64)

I then tried it again with -v and it worked:

$ curl -v -k --cert ~/.rpmfusion.cert  -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a206 -F filename=cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi 
* About to connect() to cvs.rpmfusion.org port 443 (#0)
*   Trying 195.10.6.64... connected
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* skipping SSL peer certificate verification
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
* 	subject: E=lxtnow,CN=*.rpmfusion.org,OU=RPM Fusion,O=RPM Fusion,L=France,ST=France,C=EU
* 	start date: Apr 22 15:31:22 2008 GMT
* 	expire date: Apr 20 15:31:22 2018 GMT
* 	common name: *.rpmfusion.org
* 	issuer: E=lxtnow,CN=*.rpmfusion.org,OU=RPM Fusion,O=RPM Fusion,L=France,ST=France,C=EU
> POST /repo/pkgs/upload.cgi HTTP/1.1
> User-Agent: curl/7.23.1 (x86_64-unknown-linux-gnu) libcurl/7.23.1 NSS/3.12.10.0 zlib/1.2.5 libidn/1.23 libssh2/1.3.0
> Host: cvs.rpmfusion.org
> Accept: */*
> Content-Length: 495
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=----------------------------ede1e9fce1c5
> 
* skipping SSL peer certificate verification
* NSS: client certificate from file
* 	subject: OU=Upload Files,E=paul,ST=Barcelona,O=RPM Fusion,L=Barcelona,CN=pghmcfc,C=ES
* 	start date: Jul 21 10:58:47 2011 GMT
* 	expire date: Jul 20 10:58:47 2012 GMT
* 	common name: pghmcfc
* 	issuer: E=cvsadmin,CN=cvs.rpmfusion.org,OU=Upload Files,O=RPM Fusion,L=France,ST=France,C=EU
< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< Date: Tue, 13 Dec 2011 14:34:25 GMT
< Server: Apache/2.2.3 (Red Hat)
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: text/plain; charset=UTF-8
< 
Available
* Closing connection #0

After that, I couldn't get the original command to fail again:

$ curl -k --cert ~/.rpmfusion.cert  -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a206 -F filename=cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi 
Available
$ curl -k --cert ~/.rpmfusion.cert  -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a206 -F filename=cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi 
Available
$ curl -k --cert ~/.rpmfusion.cert  -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a206 -F filename=cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi 
Available
$ curl -k --cert ~/.rpmfusion.cert  -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a206 -F filename=cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi 
Available

I thought maybe the file had just been uploaded and so maybe that had had an effect, so I changed the md5sum value to check for a file that wouldn't be there:

$ curl -k --cert ~/.rpmfusion.cert  -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a207 -F filename=cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi 
Missing
$ curl -k --cert ~/.rpmfusion.cert  -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a207 -F filename=cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi 
Missing
$ curl -k --cert ~/.rpmfusion.cert  -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a207 -F filename=cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi 
Missing
$ curl -k --cert ~/.rpmfusion.cert  -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a207 -F filename=cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi 
Missing
$ curl -k --cert ~/.rpmfusion.cert  -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a207 -F filename=cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi 
Missing
$ curl -k --cert ~/.rpmfusion.cert  -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a207 -F filename=cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi 
Missing
$ curl -k --cert ~/.rpmfusion.cert  -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a207 -F filename=cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi 
Missing
$ curl -k --cert ~/.rpmfusion.cert  -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a207 -F filename=cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi 
Missing

It just wouldn't fail again for me.
Comment 5 Kamil Dudka 2011-12-14 12:36:19 UTC 
Thank you for giving it a try, Paul.  What is common in both cases is the server.  So chances are that there was some intermittent problem either on the server, or on the network.

The fact you got failure on x86_64 suggests that the issue is not limited to 32bit clients.  What also differs between the two reports are the error codes.  Mamoru gets SSL_ERROR_HANDSHAKE_UNEXPECTED_ALERT whereas you get PR_CONNECT_RESET_ERROR.
Comment 6 Mamoru TASAKA 2011-12-14 14:10:05 UTC 
Well,

[tasaka1@localhost ~]$ grep curl /var/log/rpmpkgs
curl-7.23.0-2.fc17.i686.rpm
curlftpfs-0.9.2-8.fc16.i686.rpm
libcurl-7.23.0-2.fc17.i686.rpm
libcurl-devel-7.23.0-2.fc17.i686.rpm
python-pycurl-7.19.0-9.fc15.i686.rpm
[tasaka1@localhost ~]$ grep openssl /var/log/rpmpkgs
openssl-1.0.0e-3.fc17.i686.rpm
openssl-devel-1.0.0e-3.fc17.i686.rpm
[tasaka1@localhost ~]$ grep ^nss /var/log/rpmpkgs
nss-3.13.1-9.fc17.i686.rpm
nss-devel-3.13.1-9.fc17.i686.rpm
nss-mdns-0.10-9.fc15.i686.rpm
nss-myhostname-0.3-1.fc16.i686.rpm
nss-softokn-3.13.1-14.fc17.i686.rpm
nss-softokn-devel-3.13.1-14.fc17.i686.rpm
nss-softokn-freebl-3.13.1-14.fc17.i686.rpm
nss-softokn-freebl-devel-3.13.1-14.fc17.i686.rpm
nss-sysinit-3.13.1-9.fc17.i686.rpm
nss-util-3.13.1-3.fc16.i686.rpm
nss-util-devel-3.13.1-3.fc16.i686.rpm
nss_compat_ossl-0.9.6-2.fc15.i686.rpm
nss_ldap-265-9.fc16.i686.rpm
[tasaka1@localhost ~]$ curl -k --cert /home/tasaka1/.rpmfusion.cert -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a206 -F filename=/home/tasaka1/rpmbuild/SOURCES/cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi ; echo $?
curl: (56) SSL read: errno -12229
56
[tasaka1@localhost ~]$ curl -v -k --cert /home/tasaka1/.rpmfusion.cert -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a206 -F filename=/home/tasaka1/rpmbuild/SOURCES/cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi ; echo $?
* About to connect() to cvs.rpmfusion.org port 443 (#0)
*   Trying 195.10.6.64... connected
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* skipping SSL peer certificate verification
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
*       subject: E=lxtnow,CN=*.rpmfusion.org,OU=RPM Fusion,O=RPM Fusion,L=France,ST=France,C=EU
*       start date: Apr 22 15:31:22 2008 GMT
*       expire date: Apr 20 15:31:22 2018 GMT
*       common name: *.rpmfusion.org
*       issuer: E=lxtnow,CN=*.rpmfusion.org,OU=RPM Fusion,O=RPM Fusion,L=France,ST=France,C=EU
> POST /repo/pkgs/upload.cgi HTTP/1.1
> User-Agent: curl/7.23.0 (i386-redhat-linux-gnu) libcurl/7.23.0 NSS/3.13.1.0 zlib/1.2.5 libidn/1.23 libssh2/1.3.0
> Host: cvs.rpmfusion.org
> Accept: */*
> Content-Length: 526
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=----------------------------361b47956c2d
> 
* Done waiting for 100-continue
* SSL read: errno -12229
* Closing connection #0
curl: (56) SSL read: errno -12229
56
[tasaka1@localhost ~]$ curl -v -k --cert /home/tasaka1/.rpmfusion.cert -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a206 -F filename=/home/tasaka1/rpmbuild/SOURCES/cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi ; echo $?
* About to connect() to cvs.rpmfusion.org port 443 (#0)
*   Trying 195.10.6.64... connected
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* skipping SSL peer certificate verification
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
*       subject: E=lxtnow,CN=*.rpmfusion.org,OU=RPM Fusion,O=RPM Fusion,L=France,ST=France,C=EU
*       start date: Apr 22 15:31:22 2008 GMT
*       expire date: Apr 20 15:31:22 2018 GMT
*       common name: *.rpmfusion.org
*       issuer: E=lxtnow,CN=*.rpmfusion.org,OU=RPM Fusion,O=RPM Fusion,L=France,ST=France,C=EU
> POST /repo/pkgs/upload.cgi HTTP/1.1
> User-Agent: curl/7.23.0 (i386-redhat-linux-gnu) libcurl/7.23.0 NSS/3.13.1.0 zlib/1.2.5 libidn/1.23 libssh2/1.3.0
> Host: cvs.rpmfusion.org
> Accept: */*
> Content-Length: 526
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=----------------------------f59a0825378f
> 
* Done waiting for 100-continue
* SSL read: errno -12229
* Closing connection #0
curl: (56) SSL read: errno -12229
56
[tasaka1@localhost ~]$ curl -k --cert /home/tasaka1/.rpmfusion.cert -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a206 -F filename=/home/tasaka1/rpmbuild/SOURCES/cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi ; echo $?
curl: (56) SSL read: errno -12229
56
[tasaka1@localhost ~]$ cat /etc/pki/nssdb/pkcs11.txt
library=libnsssysinit.so
name=NSS Internal PKCS #11 Module
parameters=configdir='sql:/etc/pki/nssdb'  certPrefix='' keyPrefix='' secmod='secmod.db' flags= updatedir='' updateCertPrefix='' updateKeyPrefix='' updateid='' updateTokenDescription='' 
NSS=Flags=internal,moduleDBOnly,critical trustOrder=75 cipherOrder=100 slotParams=(1={slotFlags=[RSA,DSA,DH,RC2,RC4,DES,RANDOM,SHA1,MD5,MD2,SSL,TLS,AES,Camellia,SEED,SHA256,SHA512] askpw=any timeout=30})

[tasaka1@localhost ~]$
Comment 7 Mamoru TASAKA 2011-12-14 15:10:31 UTC 
And with curl, libcurl downgraded:

[tasaka1@localhost curl-issue]$ rpm -q curl
curl-7.21.7-5.fc16.i686
[tasaka1@localhost curl-issue]$ curl -v -k --cert /home/tasaka1/.rpmfusion.cert -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a206 -F filename=/home/tasaka1/rpmbuild/SOURCES/cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi              
* About to connect() to cvs.rpmfusion.org port 443 (#0)
*   Trying 195.10.6.64... connected
* Connected to cvs.rpmfusion.org (195.10.6.64) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* skipping SSL peer certificate verification
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
*       subject: E=lxtnow,CN=*.rpmfusion.org,OU=RPM Fusion,O=RPM Fusion,L=France,ST=France,C=EU
*       start date: Apr 22 15:31:22 2008 GMT
*       expire date: Apr 20 15:31:22 2018 GMT
*       common name: *.rpmfusion.org
*       issuer: E=lxtnow,CN=*.rpmfusion.org,OU=RPM Fusion,O=RPM Fusion,L=France,ST=France,C=EU
> POST /repo/pkgs/upload.cgi HTTP/1.1
> User-Agent: curl/7.21.7 (i386-redhat-linux-gnu) libcurl/7.21.7 NSS/3.12.10.0 zlib/1.2.5 libidn/1.23 libssh2/1.2.7
> Host: cvs.rpmfusion.org
> Accept: */*
> Content-Length: 526
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=----------------------------97c29fab56ff
> 
* skipping SSL peer certificate verification
* NSS: client certificate: PEM Token #1:.rpmfusion.cert
*       subject: OU=Upload Files,E=mtasaka,ST=Barcelona,O=RPM Fusion,L=Barcelona,CN=mtasaka,C=ES
*       start date: Dec 12 21:56:42 2011 GMT
*       expire date: Dec 11 21:56:42 2012 GMT
*       common name: mtasaka
*       issuer: E=cvsadmin,CN=cvs.rpmfusion.org,OU=Upload Files,O=RPM Fusion,L=France,ST=France,C=EU
< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< Date: Wed, 14 Dec 2011 15:09:10 GMT
< Server: Apache/2.2.3 (Red Hat)
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: text/plain; charset=UTF-8
< 
Available
* Closing connection #0
Comment 8 Kamil Dudka 2011-12-19 12:09:32 UTC 
I tried the exactly same versions of (lib)curl and nss* on an x86_64 rawhide machine and the --cert option seemed to work fine for me.  I do not have access to cvs.rpmfusion.org to try your example.  Do you have access to Fedora Koji hub?  Could you please try this on your machine?

curl -Lsvo/dev/null --cacert ~/.fedora-server-ca.cert --cert ~/.fedora.cert https://koji.fedoraproject.org/koji/login
Comment 9 Mamoru TASAKA 2011-12-19 13:46:38 UTC 
Well,

[tasaka1@localhost BUILD]$ curl -Lsvo/dev/null --cacert ~/.fedora-server-ca.cert --cert ~/.fedora.cert https://koji.fedoraproject.org/koji/login
* About to connect() to koji.fedoraproject.org port 443 (#0)
*   Trying 209.132.181.7... connected
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /home/tasaka1/.fedora-server-ca.cert
  CApath: none
* SSL connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
*       subject: E=buildsys,CN=koji.fedoraproject.org,OU=Buildsys,O=Fedora Project,ST=North Carolina,C=US
*       start date: Sep 09 03:57:30 2011 GMT
*       expire date: Sep 06 03:57:30 2021 GMT
*       common name: koji.fedoraproject.org
*       issuer: E=admin,CN=Fedora Project CA,OU=Fedora Project CA,O=Fedora Project,L=Raleigh,ST=North Carolina,C=US
> GET /koji/login HTTP/1.1
> User-Agent: curl/7.23.0 (i386-redhat-linux-gnu) libcurl/7.23.0 NSS/3.13.1.0 zlib/1.2.5 libidn/1.23 libssh2/1.3.0
> Host: koji.fedoraproject.org
> Accept: */*
> 
* NSS: client certificate from file
*       subject: E=mtasaka.ne.jp,CN=mtasaka,OU=Fedora User Cert,O=Fedora Project,ST=North Carolina,C=US
*       start date: Aug 14 05:07:58 2011 GMT
*       expire date: Feb 10 05:07:58 2012 GMT
*       common name: mtasaka
*       issuer: E=admin,CN=Fedora Project CA,OU=Fedora Project CA,O=Fedora Project,L=Raleigh,ST=North Carolina,C=US
< HTTP/1.1 302 Found
< Date: Mon, 19 Dec 2011 13:43:01 GMT
< Server: Apache/2.2.3 (Red Hat)
< Location: https://koji.fedoraproject.org/koji/index
< Cache-Control: no-cache="set-cookie"
< Set-Cookie: user=a8afd38ffebec730c2f820275f675b13mtasaka:1324302182.91; path=/koji; secure; expires=Thu, 22-Dec-2011 13:43:02 GMT
< AppTime: D=1449030
< AppServer: koji01.phx2.fedoraproject.org
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: text/plain
< 
* Closing connection #0
* Issue another request to this URL: 'https://koji.fedoraproject.org/koji/index'
* About to connect() to koji.fedoraproject.org port 443 (#0)
*   Trying 209.132.181.7... connected
*   CAfile: /home/tasaka1/.fedora-server-ca.cert
  CApath: none
* SSL connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
*       subject: E=buildsys,CN=koji.fedoraproject.org,OU=Buildsys,O=Fedora Project,ST=North Carolina,C=US
*       start date: Sep 09 03:57:30 2011 GMT
*       expire date: Sep 06 03:57:30 2021 GMT
*       common name: koji.fedoraproject.org
*       issuer: E=admin,CN=Fedora Project CA,OU=Fedora Project CA,O=Fedora Project,L=Raleigh,ST=North Carolina,C=US
> GET /koji/index HTTP/1.1
> User-Agent: curl/7.23.0 (i386-redhat-linux-gnu) libcurl/7.23.0 NSS/3.13.1.0 zlib/1.2.5 libidn/1.23 libssh2/1.3.0
> Host: koji.fedoraproject.org
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Mon, 19 Dec 2011 13:43:03 GMT
< Server: Apache/2.2.3 (Red Hat)
< AppTime: D=157458
< AppServer: koji01.phx2.fedoraproject.org
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: text/html
< 
{ [data not shown]
* Closing connection #0
[tasaka1@localhost BUILD]$
Comment 10 Mamoru TASAKA 2011-12-19 13:47:26 UTC 
And still I am getting the same error on cvs.rpmfusion.org
Comment 11 Kamil Dudka 2011-12-19 14:11:57 UTC 
Then the error seems specific to cvs.rpmfusion.org.  Are you sure that the version of libcurl takes an effect here?  If yes, could you please bisect the breakage more precisely?

Does it work with curl-7.22.0-1.fc17 ?

Does it work with curl-7.22.0-2.fc17 ?
Comment 12 Mamoru TASAKA 2011-12-19 23:04:43 UTC 
(In reply to comment #11)
> Then the error seems specific to cvs.rpmfusion.org.  Are you sure that the
> version of libcurl takes an effect here?  If yes, could you please bisect the
> breakage more precisely?
> 
> Does it work with curl-7.22.0-1.fc17 ?
> Does it work with curl-7.22.0-2.fc17 ?

No problems with these curl.
7.23.0-1.fc17 causes the same error with cvs.rpmfusion.org as 7.23.0-2.fc17.
Comment 13 Kamil Dudka 2011-12-20 11:59:16 UTC 
That's something hard to understand for me since there was no relevant code change between curl-7.22.0-2.fc17 and 7.23.0-1.fc17.  Diffing lib/nss.c, I can see only comments and white-space changes between those two versions.
Comment 14 Kamil Dudka 2011-12-20 12:04:05 UTC 
However, comparing the log files from Koji, I can see one difference.  The first one is built against nss-3.12.x whereas the second one is built against nss-3.13.x.  I will prepare a scratch build for testing...
Comment 15 Kamil Dudka 2011-12-20 12:57:55 UTC 
Well, I am not able to make the scratch build because of bug 760060 ... and this bug is likely a duplicate of bug 760060 anyway.  Please try to downgrade nss-{,-sysinit,-devel} to 3.12.11-3.fc17 and check whether it solves the problem.
Comment 16 Mamoru TASAKA 2011-12-20 22:39:43 UTC 
(In reply to comment #15)
> Please try to downgrade
> nss-{,-sysinit,-devel} to 3.12.11-3.fc17 and check whether it solves the
> problem.

No good.
Also I rebuilt curl-7.23.0-2.fc17 using F-16 + F-16-updates mock environ
(currenly using nss-3.12.11) and reinstalled it (and using 3.12.11-3.fc17),
however still no good.
Comment 17 Mamoru TASAKA 2011-12-20 22:51:44 UTC 
(In reply to comment #16)
> (In reply to comment #15)
> > Please try to downgrade
> > nss-{,-sysinit,-devel} to 3.12.11-3.fc17 and check whether it solves the
> > problem.
> 
> No good.
> Also I rebuilt curl-7.23.0-2.fc17 using F-16 + F-16-updates mock environ
> (currenly using nss-3.12.11)

  This is "currently using nss 3.12.10"

> and reinstalled it (and using 3.12.11-3.fc17),
> however still no good.
Comment 18 Kamil Dudka 2011-12-22 14:45:35 UTC 
Mamoru, please check whether the SelectClientCert() callback is called by nss:

$ gdb -q --args curl ...
(gdb) break SelectClientCert
(gdb) run
[...]
Breakpoint 1, SelectClientCert (arg=0x631fc0, sock=0x6d6c70, caNames=0x7fffffffd730, pRetCert=0x6e1648, pRetKey=0x6e1650) at nss.c:773
773     {
Comment 19 Mamoru TASAKA 2011-12-22 17:31:28 UTC 
(In reply to comment #18)
> Mamoru, please check whether the SelectClientCert() callback is called by nss:
Does not seem.

[tasaka1@localhost TEMP]$ rpm -q curl curl-debuginfo nss nss-debuginfo
curl-7.23.0-2.fc17.i686
curl-debuginfo-7.23.0-2.fc17.i686
nss-3.13.1-9.fc17.i686
nss-debuginfo-3.13.1-9.fc17.i686
[tasaka1@localhost TEMP]$ gdb -q
(gdb) file curl
Reading symbols from /usr/bin/curl...Reading symbols from /usr/lib/debug/usr/bin/curl.debug...done.
done.
(gdb) set args -v -k --cert /home/tasaka1/.rpmfusion.cert -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a206 -F filename=/home/tasaka1/rpmbuild/SOURCES/cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi
(gdb) break SelectClientCert
Function "SelectClientCert" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (SelectClientCert) pending.
(gdb) run
Starting program: /usr/bin/curl -v -k --cert /home/tasaka1/.rpmfusion.cert -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a206 -F filename=/home/tasaka1/rpmbuild/SOURCES/cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
[New Thread 0xb7cdeb40 (LWP 14154)]
* About to connect() to cvs.rpmfusion.org port 443 (#0)
*   Trying 195.10.6.64... [Thread 0xb7cdeb40 (LWP 14154) exited]
connected
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* skipping SSL peer certificate verification
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
*       subject: E=lxtnow,CN=*.rpmfusion.org,OU=RPM Fusion,O=RPM Fusion,L=France,ST=France,C=EU
*       start date:  4月 22 15:31:22 2008 GMT
*       expire date:  4月 20 15:31:22 2018 GMT
*       common name: *.rpmfusion.org
*       issuer: E=lxtnow,CN=*.rpmfusion.org,OU=RPM Fusion,O=RPM Fusion,L=France,ST=France,C=EU
> POST /repo/pkgs/upload.cgi HTTP/1.1
> User-Agent: curl/7.23.0 (i386-redhat-linux-gnu) libcurl/7.23.0 NSS/3.13.1.0 zlib/1.2.5 libidn/1.23 libssh2/1.3.0
> Host: cvs.rpmfusion.org
> Accept: */*
> Content-Length: 526
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=----------------------------54034800fb12
> 
* Done waiting for 100-continue
* SSL read: errno -12229
* Closing connection #0
curl: (56) SSL read: errno -12229
[Inferior 1 (process 14137) exited with code 070]
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-27.fc17.i686 keyutils-libs-1.5.5-1.fc17.i686 krb5-libs-1.10-0.fc17.alpha2.1.i686 libcom_err-1.42-1.fc17.i686 libidn-1.23-1.fc17.i686 libselinux-2.1.9-1.fc17.i686 libssh2-1.3.0-1.fc17.i686 nspr-4.9-0.1.fc17.beta3.i686 nss-mdns-0.10-9.fc15.i686 nss-softokn-3.13.1-14.fc17.i686 nss-softokn-freebl-3.13.1-14.fc17.i686 nss-util-3.13.1-3.fc16.i686 openldap-2.4.28-1.fc17.i686 openssl-1.0.0e-3.fc17.i686 sqlite-3.7.9-1.fc17.i686
(gdb) quit
Comment 20 Mamoru TASAKA 2011-12-22 17:43:13 UTC 
With curl-7.22.0-2.fc17.i686:

[tasaka1@localhost TEMP]$ rpm -q curl curl-debuginfo nss nss-debuginfo
curl-7.22.0-2.fc17.i686
curl-debuginfo-7.22.0-2.fc17.i686
nss-3.13.1-9.fc17.i686
nss-debuginfo-3.13.1-9.fc17.i686
[tasaka1@localhost TEMP]$ gdb -q
(gdb) file curl
Reading symbols from /usr/bin/curl...Reading symbols from /usr/lib/debug/usr/bin/curl.debug...done.
done.
(gdb) set args -v -k --cert /home/tasaka1/.rpmfusion.cert -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a206 -F filename=/home/tasaka1/rpmbuild/SOURCES/cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi
(gdb) break SelectClientCert
Function "SelectClientCert" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (SelectClientCert) pending.
(gdb) run
Starting program: /usr/bin/curl -v -k --cert /home/tasaka1/.rpmfusion.cert -F tree=free -F name=cairo-dock -F md5sum=2407e97b74a97c383dd7ef4f0c83a206 -F filename=/home/tasaka1/rpmbuild/SOURCES/cairo-dock-2.4.0~2.tar.gz https://cvs.rpmfusion.org/repo/pkgs/upload.cgi
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
[New Thread 0xb7cdeb40 (LWP 16208)]
* About to connect() to cvs.rpmfusion.org port 443 (#0)
*   Trying 195.10.6.64... [Thread 0xb7cdeb40 (LWP 16208) exited]
connected
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* skipping SSL peer certificate verification
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
*       subject: E=lxtnow,CN=*.rpmfusion.org,OU=RPM Fusion,O=RPM Fusion,L=France,ST=France,C=EU
*       start date:  4月 22 15:31:22 2008 GMT
*       expire date:  4月 20 15:31:22 2018 GMT
*       common name: *.rpmfusion.org
*       issuer: E=lxtnow,CN=*.rpmfusion.org,OU=RPM Fusion,O=RPM Fusion,L=France,ST=France,C=EU
> POST /repo/pkgs/upload.cgi HTTP/1.1
> User-Agent: curl/7.22.0 (i386-redhat-linux-gnu) libcurl/7.22.0 NSS/3.12.11.0 zlib/1.2.5 libidn/1.23 libssh2/1.3.0
> Host: cvs.rpmfusion.org
> Accept: */*
> Content-Length: 526
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=----------------------------4cadbe988127
> 
* skipping SSL peer certificate verification

Breakpoint 1, SelectClientCert (arg=0x8072ed8, sock=0x80effc8, caNames=0xbfffe5f4, pRetCert=0x80f5ab8, pRetKey=0x80f5abc) at nss.c:772
772     {
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-27.fc17.i686 keyutils-libs-1.5.5-1.fc17.i686 krb5-libs-1.10-0.fc17.alpha2.1.i686 libcom_err-1.42-1.fc17.i686 libidn-1.23-1.fc17.i686 libselinux-2.1.9-1.fc17.i686 libssh2-1.3.0-1.fc17.i686 nspr-4.9-0.1.fc17.beta3.i686 nss-mdns-0.10-9.fc15.i686 nss-softokn-3.13.1-14.fc17.i686 nss-softokn-freebl-3.13.1-14.fc17.i686 nss-util-3.13.1-3.fc16.i686 openldap-2.4.28-1.fc17.i686 openssl-1.0.0e-3.fc17.i686 sqlite-3.7.9-1.fc17.i686
(gdb) cont
Continuing.
* NSS: client certificate from file
*       subject: OU=Upload Files,E=mtasaka,ST=Barcelona,O=RPM Fusion,L=Barcelona,CN=mtasaka,C=ES
*       start date: 12月 12 21:56:42 2011 GMT
*       expire date: 12月 11 21:56:42 2012 GMT
*       common name: mtasaka
*       issuer: E=cvsadmin,CN=cvs.rpmfusion.org,OU=Upload Files,O=RPM Fusion,L=France,ST=France,C=EU
< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< Date: Thu, 22 Dec 2011 17:41:38 GMT
< Server: Apache/2.2.3 (Red Hat)
< Content-Length: 10
< Connection: close
< Content-Type: text/plain; charset=UTF-8
< 
Available
* Closing connection #0
[Inferior 1 (process 16201) exited normally]
(gdb) quit
Comment 21 Kamil Dudka 2011-12-22 20:42:57 UTC 
Well, libcurl can hardly provide a client certificate if it is not asked to do so.  Unfortunately, I still did not find any change in libcurl that would cause either the server, or nss to forget to ask for a client certificate.

Please try to set a fixed user agent identification to rule out server's sensitivity to this header field:

--user-agent 'curl/7.22.0 (i386-redhat-linux-gnu) libcurl/7.22.0 NSS/3.12.11.0 zlib/1.2.5 libidn/1.23 libssh2/1.3.0'

It may also be worth to try to switch the SSL version using the --tlsv1/--sslv3 options.  I could try git-bisect on upstream libcurl, but the key problem is that I am not able to reproduce the failure myself (and have no access to cvs.rpmfusion.org).
Comment 22 Mamoru TASAKA 2011-12-23 11:07:22 UTC 
Now I had some time to investigate this a bit further:

git repo I used: https://github.com/bagder/curl.git

   65  git bisect start
   66  git bisect bad 7cfd10e2553a4239ef10d924172da3affe30
   67  git bisect good d52cd3bd1799a4f46c58d7bb372ed636
   68  git bisect run bash ./git-bisect-trial.sh

->
9dd85bced56f6951107f69e581c872c1e7e3e58e is the first bad commit
commit 9dd85bced56f6951107f69e581c872c1e7e3e58e
Author: Daniel Stenberg <daniel>
Date:   Sun Oct 2 19:28:39 2011 +0200

    multi: progress function abort must close connection
    
    When the progress function returns to cancel the request, we must mark
    the connection to get closed and it must do to the DONE state.
    
    do_init() must be called as early as possible so that state variables
    for new connections are reset early. We could otherwise see that the old
    values were still there when a connection was to be disconnected very
    early and it would make it behave wrongly.
    
    Bug: http://curl.haxx.se/mail/lib-2011-10/0006.html
    Reported by: Vladimir Grishchenko

:040000 040000 0408b0623d619d2d416ea0260283c13494fcf045 d515eac330c6187098fdaf824ce9ae2760949c6c M      lib
bisect run success
Comment 23 Mamoru TASAKA 2011-12-23 13:45:31 UTC 
I rebuilt curl-7.23.0-2.fc17 with commit 9dd85bced56f6951107f69e581c872c1e7e3e58e only reverted and actually it seems to be working.
Comment 24 Kamil Dudka 2011-12-24 00:51:29 UTC 
Thanks for debugging it.  I think we are running out of CURL_TIMEOUT_EXPECT_100, which is set to 1000 ms.  If it indeed is the reason, we should either increase the timeout, or try to compute the elapsed time more precisely.
Comment 25 Kamil Dudka 2011-12-24 09:10:29 UTC 
Mamoru, please try to apply this patch instead of the reverted one:

--- a/lib/transfer.c
+++ b/lib/transfer.c
@@ -2364,7 +2364,7 @@ Curl_setup_transfer(
          (data->state.proto.http->sending == HTTPSEND_BODY)) {
         /* wait with write until we either got 100-continue or a timeout */
         k->exp100 = EXP100_AWAITING_CONTINUE;
-        k->start100 = k->start;
+        k->start100 = Curl_tvnow();

         /* set a timeout for the multi interface */
         Curl_expire(data, CURL_TIMEOUT_EXPECT_100);
Comment 26 Mamoru TASAKA 2011-12-24 14:41:28 UTC 
(In reply to comment #25)
> Mamoru, please try to apply this patch instead of the reverted one:
> 
> --- a/lib/transfer.c
> +++ b/lib/transfer.c
> @@ -2364,7 +2364,7 @@ Curl_setup_transfer(
>           (data->state.proto.http->sending == HTTPSEND_BODY)) {
>          /* wait with write until we either got 100-continue or a timeout */
>          k->exp100 = EXP100_AWAITING_CONTINUE;
> -        k->start100 = k->start;
> +        k->start100 = Curl_tvnow();
> 
>          /* set a timeout for the multi interface */
>          Curl_expire(data, CURL_TIMEOUT_EXPECT_100);

This seems to be working.
Comment 27 Kamil Dudka 2011-12-25 22:12:14 UTC 
Thanks for confirmation.  Pushed upstream:

https://github.com/bagder/curl/commit/9f7f6a6
Comment 28 Kamil Dudka 2011-12-25 23:14:26 UTC 
fixed in curl-7.23.0-3.fc17
Comment 29 Mamoru TASAKA 2011-12-26 15:43:27 UTC 
(In reply to comment #28)
> fixed in curl-7.23.0-3.fc17

Confirmed. Thank you.