Bug 429175 - fedora's make upload fails due to curl breakage.
fedora's make upload fails due to curl breakage.
Status: CLOSED NEXTRELEASE
Product: Fedora
Classification: Fedora
Component: nss (Show other bugs)
rawhide
x86_64 Linux
low Severity low
: ---
: ---
Assigned To: Kai Engert (:kaie)
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-01-17 14:04 EST by Dave Jones
Modified: 2015-01-04 17:30 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-09-24 19:58:01 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Fix size of variable to work on x86_64 systems (609 bytes, patch)
2008-08-13 21:15 EDT, Rob Crittenden
no flags Details | Diff

  None (edit)
Description Dave Jones 2008-01-17 14:04:17 EST
Checking : patch-2.6.24-rc8-git1.bz2 on
https://cvs.fedoraproject.org/repo/pkgs/upload.cgi...
ERROR: could not check remote file status

Running in verbose mode ..

$ curl -v -k --cert /home/davej/.fedora.cert --fail -F name=kernel -F
md5sum=0ea47257a4afdb2f079a40ccf93adaad -F filename=patch-2.6.24-rc8-git1.bz2
https://cvs.fedoraproject.org/repo/pkgs/upload.cgi
* About to connect() to cvs.fedoraproject.org port 443 (#0)
*   Trying 209.132.176.51... connected
* Connected to cvs.fedoraproject.org (209.132.176.51) port 443 (#0)
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* Unable to load certificate /home/davej/.fedora.cert

* Closing connection #0
curl: (58) Unable to load certificate /home/davej/.fedora.cert


strace'ing the above command shows that it succeeds in loading the file..

stat("/home/davej/.fedora.cert", {st_mode=S_IFREG|0644, st_size=5029, ...}) = 0
stat("/home/davej/.fedora.cert", {st_mode=S_IFREG|0644, st_size=5029, ...}) = 0
open("/home/davej/.fedora.cert", O_RDONLY) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=5029, ...}) = 0
read(7, "Certificate:\n    Data:\n        V"..., 5029) = 5029
close(7)                                = 0

$ ll ~/.fedora.cert
-rw-r--r-- 1 davej davej 5029 2007-11-27 15:27 /home/davej/.fedora.cert

5029 bytes were read, so it's the parsing of the file that it fails on.

Looking closer at the error message..
curl: (58) Unable to load certificate /home/davej/.fedora.cert

Grepping for 58 in the source yields..

include/curl/curl.h:  CURLE_SSL_CERTPROBLEM,         /* 58 - problem with the
local certificate */

which is returned in a number of places.


Any clues what I can do to further debug this?  I'm really puzzled why I'm the
only person seeing this problem.  I've tried recreating my certificate, which
made no difference.

Could we add some more messages to curl to print out exactly what the problem is?
Comment 1 Dave Jones 2008-01-17 17:11:39 EST
Spent some time with gdb.
In nss_load_cert(), this happens..

    /* An invalid nickname was passed in */
    if (cert == NULL) {
      free(nickname);
      PR_SetError(SEC_ERROR_UNKNOWN_CERT, 0);
      return 0;
    }

352	    cert = PK11_FindCertFromNickname((char *)nickname, NULL);
(gdb) 
355	    if (cert == NULL) {
(gdb) n
356	      free(nickname);

(gdb) print nickname
$8 = 0x69e0e0 "PEM Token #1:.fedora.cert"

Comment 2 Dave Jones 2008-01-19 14:50:37 EST
Some additional debugging happened here in the comments section of my blog.. 
http://kernelslacker.livejournal.com/104675.html?thread=452067
Comment 3 Jindrich Novy 2008-01-20 05:41:49 EST
Just out of curiosity, what does "cat ~/.fedora.cert | grep OU" say?
Comment 4 Dave Jones 2008-01-22 12:42:22 EST
        Issuer: C=US, ST=North Carolina, L=Raleigh, O=Fedora Project, OU=Upload
Files, CN=cvs.fedora.redhat.com/emailAddress=webmaster@fedora.redhat.com
        Subject: C=US, ST=North Carolina, O=Fedora Project, OU=Dave Jones,
CN=davej/emailAddress=davej@redhat.com
                DirName:/C=US/ST=North Carolina/L=Raleigh/O=Fedora
Project/OU=Upload
Files/CN=cvs.fedora.redhat.com/emailAddress=webmaster@fedora.redhat.com
WNT1Icps2c26soKZlVKEW4WkkxSayT/7YrFvUzuFCtkQOU7PvLOp/y0eoY/qai2e
Comment 5 Jindrich Novy 2008-01-29 04:56:51 EST
The certificate is sane IMO.

Have you tried if the latest curl (7.18.0) which is now in rawhide helps anyhow?
Comment 6 Dave Jones 2008-02-12 00:59:16 EST
I don't have a rawhide system to test on right now, because the rawhide compose
seems to fail every time I go to do an install.
And I can't install the rawhide curl on F8 because of so much stuff I have
installed depending on libcurl.so.4
Comment 7 Dave Jones 2008-02-18 11:36:03 EST
does any of the certutil magic at
http://kernelslacker.livejournal.com/104675.html yield any clues ?   Any ideas
of further things I could run to help diagnose this?

I still can't get rawhide installing, so I can't test the newer curl.
Comment 8 Jindrich Novy 2008-02-19 02:30:44 EST
Ok, I mocked the newst curl for F8, could you give it a try?

http://people.redhat.com/jnovy/files/curl/
Comment 9 Dave Jones 2008-02-23 19:10:22 EST
I managed to get rawhide installing on a box.
curl-7.18.0-2.fc9.i386 works fine.

I'll try out the f8 rpms next.
Comment 10 Dave Jones 2008-02-23 19:14:35 EST
hmm, the f8 rpms fail the same way they did before.
maybe some change in a library that curl is dependant upon ?
Comment 11 Dave Jones 2008-04-01 19:33:16 EDT
ok, I updated my desktop machine to rawhide, and now that also fails, so there's
something special about my setup that causes it to happen on this machine, but
not on another machine also running rawhide.

What files should I compare ?

They both have the same ~/.fedora.cert
Comment 12 Dave Jones 2008-04-01 20:01:51 EDT
hmm, the machine that it works on is i386.  Doing a rawhide install on another
x86-64 box, to see if it's really arch specific.
Comment 13 Dave Jones 2008-04-01 20:27:50 EDT
ok, confirmed. this definitely only happens on x86-64.
Comment 14 Dave Jones 2008-04-01 21:08:46 EDT
and if I run the 32bit version of curl (after installing all the 32bit
libraries) on the affected box, it works.
Comment 15 Dave Jones 2008-04-01 21:32:29 EDT
bingo.  It only shows up if you set the environment variable MALLOC_PERTURB_=123
and only on x86-64.

some allocation _somewhere_ isn't clearing the memory handed to it by malloc,
and when that memory happens to contain garbage, bad things happen.

now, it could be curl, or it could be in one of the many libraries on which it's
dependant.  Nasty.
Comment 16 Dave Jones 2008-04-01 21:38:36 EDT
hmm, valgrind doesn't seem to find anything obvious.  Jakub, any ideas on how to
track this down ?
Comment 17 Dave Jones 2008-04-01 23:13:10 EDT
further experimenting with gdb shows that in nssCertificate_GetDecoding c->type
is NSSCertificateType_Unknown if we set the poison.  Otherwise, it's
NSSCertificateType_PKIX
Comment 18 Bug Zapper 2008-05-14 00:47:58 EDT
Changing version to '9' as part of upcoming Fedora 9 GA.
More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 19 Dave Jones 2008-05-17 13:37:14 EDT
This bug should possibly get reassigned to another component, but even after my
digging above, I'm not entire sure where the bug belongs.
Comment 20 Paul W. Frields 2008-08-09 12:03:57 EDT
Triage here.  I'm going to set this to ASSIGNED because Dave and Jindrich have already started collaborating on this.  I'm assuming this is still a problem?
Comment 21 Dave Jones 2008-08-09 13:22:22 EDT
Yes. No progress has been made on this bug since the initial investigation.
Comment 22 Rob Crittenden 2008-08-11 11:08:33 EDT
I've reproduced this and found what may be another way to go about debugging it. The problem is not with curl, that's for sure.

The PEM module, which handles loading ASCII keys and certs on the fly, can be added to an NSS database for a more "persistent" view. First I'm going to create a new NSS database to play with:

% cd /tmp
% certutil -N -d . (set a password if you like)
% modutil -dbdir . -add PEM -libfile /usr/lib64/libnsspem.so  -string "/home/rcrit/.fedora.cert;/home/rcrit/.fedora.cert"
% Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

PEM Token #0:.fedora.cert                                    u,u,u

So there's my cert.

Let's have NSS verify it:

% unset MALLOC_PERTURB_
% certutil -V -u V -n "PEM Token #0:.fedora.cert" -d .
certutil: certificate is invalid: Peer's Certificate issuer is not recognized.

Ok, that isn't unexpected because I haven't loaded the Fedora CA. Let's do that and try again:

% certutil -A -n "Fedora CA" -d . -t CT,CT, -a < /home/rcrit/.koji/clientca.crt
% certutil -V -u V -n "PEM Token #0:.fedora.cert" -d .
certutil: certificate is valid

Bingo.

And now with MALLOC_PERTURB_ set...

% export MALLOC_PERTURB_=123
% certutil -V -u V -n "PEM Token #0:.fedora.cert" -d .
certutil: could not find certificate named "PEM Token #0:.fedora.cert": Unrecognized Object Identifier.

So it would appear that the bug is either in NSS or in the NSS PKCS#11 PEM module.

An interesting factoid: the verify succeeds when run in valgrind with MALLOC_PERTURB_ set.
Comment 23 Rob Crittenden 2008-08-13 21:15:48 EDT
Created attachment 314273 [details]
Fix size of variable to work on x86_64 systems

From Bob Relyea:

I think you are on a 64 bit platform and the size is wrong (should be 8, not 4). 

You are only copying the bottom 4 bytes of the U_LONG (intel is a little endian platform).
Comment 24 Rob Crittenden 2008-08-13 21:18:40 EDT
Kai, can you add this patch to NSS?
Comment 25 Kai Engert (:kaie) 2008-08-15 07:04:40 EDT
I'm ready to add it, but the build infrastructure is down right now
Comment 26 Kai Engert (:kaie) 2008-08-22 15:46:39 EDT
The following packages contain Rob's patch.
They are not (yet) available through yum, but we'd appreciate your testing!

rawhide: http://koji.fedoraproject.org/koji/taskinfo?taskID=778589
f8: http://koji.fedoraproject.org/koji/taskinfo?taskID=778813
f9: http://koji.fedoraproject.org/koji/taskinfo?taskID=778825
Comment 27 Fedora Update System 2008-08-22 19:24:05 EDT
nss-3.12.1.0-1.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/nss-3.12.1.0-1.fc9
Comment 28 Fedora Update System 2008-08-22 19:24:49 EDT
nss-3.12.1.0-1.fc8 has been submitted as an update for Fedora 8.
http://admin.fedoraproject.org/updates/nss-3.12.1.0-1.fc8
Comment 29 Dave Jones 2008-08-22 21:04:28 EDT
Works for me!
Comment 30 Fedora Update System 2008-09-05 17:35:31 EDT
nss-3.12.1.1-1.fc8 has been submitted as an update for Fedora 8.
http://admin.fedoraproject.org/updates/nss-3.12.1.1-1.fc8
Comment 31 Fedora Update System 2008-09-05 17:36:29 EDT
nss-3.12.1.1-1.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/nss-3.12.1.1-1.fc9
Comment 32 Fedora Update System 2008-09-11 13:05:40 EDT
nss-3.12.1.1-1.fc8 has been pushed to the Fedora 8 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update nss'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F8/FEDORA-2008-7895
Comment 33 Fedora Update System 2008-09-11 13:08:39 EDT
nss-3.12.1.1-1.fc9 has been pushed to the Fedora 9 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update nss'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F9/FEDORA-2008-7921
Comment 34 Fedora Update System 2008-09-24 19:57:57 EDT
nss-3.12.1.1-1.fc8 has been pushed to the Fedora 8 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 35 Fedora Update System 2008-09-24 20:10:44 EDT
nss-3.12.1.1-1.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.