Checking : patch-2.6.24-rc8-git1.bz2 on https://cvs.fedoraproject.org/repo/pkgs/upload.cgi... ERROR: could not check remote file status Running in verbose mode .. $ curl -v -k --cert /home/davej/.fedora.cert --fail -F name=kernel -F md5sum=0ea47257a4afdb2f079a40ccf93adaad -F filename=patch-2.6.24-rc8-git1.bz2 https://cvs.fedoraproject.org/repo/pkgs/upload.cgi * About to connect() to cvs.fedoraproject.org port 443 (#0) * Trying 209.132.176.51... connected * Connected to cvs.fedoraproject.org (209.132.176.51) port 443 (#0) * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * Unable to load certificate /home/davej/.fedora.cert * Closing connection #0 curl: (58) Unable to load certificate /home/davej/.fedora.cert strace'ing the above command shows that it succeeds in loading the file.. stat("/home/davej/.fedora.cert", {st_mode=S_IFREG|0644, st_size=5029, ...}) = 0 stat("/home/davej/.fedora.cert", {st_mode=S_IFREG|0644, st_size=5029, ...}) = 0 open("/home/davej/.fedora.cert", O_RDONLY) = 7 fstat(7, {st_mode=S_IFREG|0644, st_size=5029, ...}) = 0 read(7, "Certificate:\n Data:\n V"..., 5029) = 5029 close(7) = 0 $ ll ~/.fedora.cert -rw-r--r-- 1 davej davej 5029 2007-11-27 15:27 /home/davej/.fedora.cert 5029 bytes were read, so it's the parsing of the file that it fails on. Looking closer at the error message.. curl: (58) Unable to load certificate /home/davej/.fedora.cert Grepping for 58 in the source yields.. include/curl/curl.h: CURLE_SSL_CERTPROBLEM, /* 58 - problem with the local certificate */ which is returned in a number of places. Any clues what I can do to further debug this? I'm really puzzled why I'm the only person seeing this problem. I've tried recreating my certificate, which made no difference. Could we add some more messages to curl to print out exactly what the problem is?
Spent some time with gdb. In nss_load_cert(), this happens.. /* An invalid nickname was passed in */ if (cert == NULL) { free(nickname); PR_SetError(SEC_ERROR_UNKNOWN_CERT, 0); return 0; } 352 cert = PK11_FindCertFromNickname((char *)nickname, NULL); (gdb) 355 if (cert == NULL) { (gdb) n 356 free(nickname); (gdb) print nickname $8 = 0x69e0e0 "PEM Token #1:.fedora.cert"
Some additional debugging happened here in the comments section of my blog.. http://kernelslacker.livejournal.com/104675.html?thread=452067
Just out of curiosity, what does "cat ~/.fedora.cert | grep OU" say?
Issuer: C=US, ST=North Carolina, L=Raleigh, O=Fedora Project, OU=Upload Files, CN=cvs.fedora.redhat.com/emailAddress=webmaster.com Subject: C=US, ST=North Carolina, O=Fedora Project, OU=Dave Jones, CN=davej/emailAddress=davej DirName:/C=US/ST=North Carolina/L=Raleigh/O=Fedora Project/OU=Upload Files/CN=cvs.fedora.redhat.com/emailAddress=webmaster.com WNT1Icps2c26soKZlVKEW4WkkxSayT/7YrFvUzuFCtkQOU7PvLOp/y0eoY/qai2e
The certificate is sane IMO. Have you tried if the latest curl (7.18.0) which is now in rawhide helps anyhow?
I don't have a rawhide system to test on right now, because the rawhide compose seems to fail every time I go to do an install. And I can't install the rawhide curl on F8 because of so much stuff I have installed depending on libcurl.so.4
does any of the certutil magic at http://kernelslacker.livejournal.com/104675.html yield any clues ? Any ideas of further things I could run to help diagnose this? I still can't get rawhide installing, so I can't test the newer curl.
Ok, I mocked the newst curl for F8, could you give it a try? http://people.redhat.com/jnovy/files/curl/
I managed to get rawhide installing on a box. curl-7.18.0-2.fc9.i386 works fine. I'll try out the f8 rpms next.
hmm, the f8 rpms fail the same way they did before. maybe some change in a library that curl is dependant upon ?
ok, I updated my desktop machine to rawhide, and now that also fails, so there's something special about my setup that causes it to happen on this machine, but not on another machine also running rawhide. What files should I compare ? They both have the same ~/.fedora.cert
hmm, the machine that it works on is i386. Doing a rawhide install on another x86-64 box, to see if it's really arch specific.
ok, confirmed. this definitely only happens on x86-64.
and if I run the 32bit version of curl (after installing all the 32bit libraries) on the affected box, it works.
bingo. It only shows up if you set the environment variable MALLOC_PERTURB_=123 and only on x86-64. some allocation _somewhere_ isn't clearing the memory handed to it by malloc, and when that memory happens to contain garbage, bad things happen. now, it could be curl, or it could be in one of the many libraries on which it's dependant. Nasty.
hmm, valgrind doesn't seem to find anything obvious. Jakub, any ideas on how to track this down ?
further experimenting with gdb shows that in nssCertificate_GetDecoding c->type is NSSCertificateType_Unknown if we set the poison. Otherwise, it's NSSCertificateType_PKIX
Changing version to '9' as part of upcoming Fedora 9 GA. More information and reason for this action is here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
This bug should possibly get reassigned to another component, but even after my digging above, I'm not entire sure where the bug belongs.
Triage here. I'm going to set this to ASSIGNED because Dave and Jindrich have already started collaborating on this. I'm assuming this is still a problem?
Yes. No progress has been made on this bug since the initial investigation.
I've reproduced this and found what may be another way to go about debugging it. The problem is not with curl, that's for sure. The PEM module, which handles loading ASCII keys and certs on the fly, can be added to an NSS database for a more "persistent" view. First I'm going to create a new NSS database to play with: % cd /tmp % certutil -N -d . (set a password if you like) % modutil -dbdir . -add PEM -libfile /usr/lib64/libnsspem.so -string "/home/rcrit/.fedora.cert;/home/rcrit/.fedora.cert" % Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI PEM Token #0:.fedora.cert u,u,u So there's my cert. Let's have NSS verify it: % unset MALLOC_PERTURB_ % certutil -V -u V -n "PEM Token #0:.fedora.cert" -d . certutil: certificate is invalid: Peer's Certificate issuer is not recognized. Ok, that isn't unexpected because I haven't loaded the Fedora CA. Let's do that and try again: % certutil -A -n "Fedora CA" -d . -t CT,CT, -a < /home/rcrit/.koji/clientca.crt % certutil -V -u V -n "PEM Token #0:.fedora.cert" -d . certutil: certificate is valid Bingo. And now with MALLOC_PERTURB_ set... % export MALLOC_PERTURB_=123 % certutil -V -u V -n "PEM Token #0:.fedora.cert" -d . certutil: could not find certificate named "PEM Token #0:.fedora.cert": Unrecognized Object Identifier. So it would appear that the bug is either in NSS or in the NSS PKCS#11 PEM module. An interesting factoid: the verify succeeds when run in valgrind with MALLOC_PERTURB_ set.
Created attachment 314273 [details] Fix size of variable to work on x86_64 systems From Bob Relyea: I think you are on a 64 bit platform and the size is wrong (should be 8, not 4). You are only copying the bottom 4 bytes of the U_LONG (intel is a little endian platform).
Kai, can you add this patch to NSS?
I'm ready to add it, but the build infrastructure is down right now
The following packages contain Rob's patch. They are not (yet) available through yum, but we'd appreciate your testing! rawhide: http://koji.fedoraproject.org/koji/taskinfo?taskID=778589 f8: http://koji.fedoraproject.org/koji/taskinfo?taskID=778813 f9: http://koji.fedoraproject.org/koji/taskinfo?taskID=778825
nss-3.12.1.0-1.fc9 has been submitted as an update for Fedora 9. http://admin.fedoraproject.org/updates/nss-3.12.1.0-1.fc9
nss-3.12.1.0-1.fc8 has been submitted as an update for Fedora 8. http://admin.fedoraproject.org/updates/nss-3.12.1.0-1.fc8
Works for me!
nss-3.12.1.1-1.fc8 has been submitted as an update for Fedora 8. http://admin.fedoraproject.org/updates/nss-3.12.1.1-1.fc8
nss-3.12.1.1-1.fc9 has been submitted as an update for Fedora 9. http://admin.fedoraproject.org/updates/nss-3.12.1.1-1.fc9
nss-3.12.1.1-1.fc8 has been pushed to the Fedora 8 testing repository. If problems still persist, please make note of it in this bug report. If you want to test the update, you can install it with su -c 'yum --enablerepo=updates-testing update nss'. You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F8/FEDORA-2008-7895
nss-3.12.1.1-1.fc9 has been pushed to the Fedora 9 testing repository. If problems still persist, please make note of it in this bug report. If you want to test the update, you can install it with su -c 'yum --enablerepo=updates-testing update nss'. You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F9/FEDORA-2008-7921
nss-3.12.1.1-1.fc8 has been pushed to the Fedora 8 stable repository. If problems still persist, please make note of it in this bug report.
nss-3.12.1.1-1.fc9 has been pushed to the Fedora 9 stable repository. If problems still persist, please make note of it in this bug report.