Bug 969463

Summary: Every site I go to gives me the cert error. Not sure if this is Firefox fault all ca-certificates
Product: [Fedora] Fedora Reporter: Daniel Walsh <dwalsh>
Component: ca-certificatesAssignee: Kai Engert (:kaie) (inactive account) <kengert>
Status: CLOSED DUPLICATE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 19CC: amarecek, andrew, bastiaan, dwalsh, jorton, kengert, michele, pwouters, stefw, tchollingsworth, tmraz
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-06-21 14:30:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Daniel Walsh 2013-05-31 13:20:03 UTC
Going to sites like my bank, facebook, google Red Hat sites are all asking for confirmation of the Certificate.  Installed another F19 machine and am seeing the same problems.

Comment 1 Kai Engert (:kaie) (inactive account) 2013-05-31 13:27:28 UTC
Thanks for your bug report, I'd like to ask about more details to allow me to investigate.

Fresh f19 install or upgrade from earlier version?

Using Firefox or another browser? Did you try more than one browser?

Comment 2 Kai Engert (:kaie) (inactive account) 2013-05-31 13:28:50 UTC
Ok, subject says firefox.

Can you please paste the output of the command below?

$ rpm -qa firefox\* nss\* nspr\* ca-certificate\* p11-kit\*

Comment 3 Kai Engert (:kaie) (inactive account) 2013-05-31 13:32:11 UTC
Please also provide output of two additional commands:

$ ls -l /etc/alternatives/libnssckbi.*

$ ls -l /etc/pki/ca-trust/source/anchors/ /etc/pki/ca-trust/source/ /etc/pki/ca-trust/source/blacklist/ /usr/share/pki/ca-trust-source/ /usr/share/pki/ca-trust-source/anchors/ /usr/share/pki/ca-trust-source/blacklist/ /etc/pki/tls/ /etc/ssl/certs/

Thanks

Comment 4 Kai Engert (:kaie) (inactive account) 2013-05-31 13:52:11 UTC
I wonder if this related to bug 960230, where users report, that CAs become untrusted after suspend/resume.

Comment 5 Tomas Mraz 2013-05-31 13:58:35 UTC
Or it could be some SELinux or sandboxing problem. Dan are you running Firefox in some kind of sandbox or with SELinux confinement?

Note that firefox (NSS) now needs to read the /etc/pki/ca-trust directories, that is cert_t domain.

Or is that unmodified default setup?

Comment 6 Kai Engert (:kaie) (inactive account) 2013-05-31 14:00:32 UTC
> Note that firefox (NSS) now needs to read the /etc/pki/ca-trust directories,
> that is cert_t domain.

Also /usr/share/pki/ca-trust-source/ which contains the default trust list.

Comment 7 Daniel Walsh 2013-05-31 14:55:47 UTC
rpm -qa firefox\* nss\* nspr\* ca-certificate\* p11-kit\*
p11-kit-trust-0.18.2-1.fc19.x86_64
nss-util-3.14.3-1.fc19.x86_64
p11-kit-0.18.2-1.fc19.x86_64
nss-3.14.3-13.0.fc19.x86_64
ca-certificates-2012.87-10.2.fc19.noarch
nspr-4.9.6-1.fc19.x86_64
nss-softokn-3.14.3-1.fc19.x86_64
nss-tools-3.14.3-13.0.fc19.x86_64
p11-kit-devel-0.18.2-1.fc19.x86_64
firefox-21.0-3.fc19.x86_64
nss-softokn-freebl-3.14.3-1.fc19.x86_64
nss-sysinit-3.14.3-13.0.fc19.x86_64

ls -l /etc/alternatives/libnssckbi.*
lrwxrwxrwx. 1 root root 34 May 16 09:20 /etc/alternatives/libnssckbi.so.x86_64 -> /usr/lib64/pkcs11/p11-kit-trust.so

ls -l /etc/pki/ca-trust/source/anchors/ /etc/pki/ca-trust/source/ /etc/pki/ca-trust/source/blacklist/ /usr/share/pki/ca-trust-source/ /usr/share/pki/ca-trust-source/anchors/ /usr/share/pki/ca-trust-source/blacklist/ /etc/pki/tls/ /etc/ssl/certs/
/etc/pki/ca-trust/source/:
total 12
drwxr-xr-x. 2 root root 4096 May 27 09:36 anchors
drwxr-xr-x. 2 root root 4096 May 27 09:36 blacklist
-rw-r--r--. 1 root root 3920 May 27 09:33 README

/etc/pki/ca-trust/source/anchors/:
total 0

/etc/pki/ca-trust/source/blacklist/:
total 0

/etc/pki/tls/:
total 24
lrwxrwxrwx. 1 root root    49 May 28 08:18 cert.pem -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
drwxr-xr-x. 2 root root  4096 May 28 08:18 certs
drwxr-xr-x. 2 root root  4096 Mar 19 13:38 misc
-rw-r--r--. 1 root root 10906 Mar 18 16:56 openssl.cnf
drwxr-xr-x. 2 root root  4096 Mar 18 17:02 private

/etc/ssl/certs/:
total 1472
lrwxrwxrwx. 1 root root     49 May 28 08:18 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
-rw-r--r--. 1 root root 704495 Jul 23  2012 ca-bundle.crt.rpmsave
lrwxrwxrwx. 1 root root     55 May 28 08:18 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
-rw-r--r--. 1 root root 787088 Jul 23  2012 ca-bundle.trust.crt.rpmsave
-rwxr-xr-x. 1 root root    610 Mar 18 17:03 make-dummy-cert
-rw-r--r--. 1 root root   2242 Mar 18 17:03 Makefile
-rwxr-xr-x. 1 root root    829 Mar 18 17:03 renew-dummy-cert

/usr/share/pki/ca-trust-source/:
total 916
drwxr-xr-x. 2 root root   4096 May 27 09:36 anchors
drwxr-xr-x. 2 root root   4096 May 27 09:36 blacklist
-rw-r--r--. 1 root root   4940 May 27 09:33 ca-bundle.neutral-trust.crt
-rw-r--r--. 1 root root   2117 May 27 09:33 ca-bundle.supplement.p11-kit
-rw-r--r--. 1 root root 912138 May 27 09:33 ca-bundle.trust.crt
-rw-r--r--. 1 root root   3961 May 27 09:33 README

/usr/share/pki/ca-trust-source/anchors/:
total 0

/usr/share/pki/ca-trust-source/blacklist/:
total 0

Both machines were installed with F18 and updated to F19.

I don't think it is SELinux related.  How can I blow away the ones I said to trust to see if it is an SELinux issue.  My process seesm to be allowed to read the /usr/share/pki/ca-trust-source.

Comment 8 Kai Engert (:kaie) (inactive account) 2013-05-31 19:03:53 UTC
The information in comment 7 looks as expected. I have the same packages and files on my F19 VM, which has selinux enabled. In that VM, I can use firefox to connect to google sites and facebook, no warnings in firefox.


What does this say:

# ls -l /etc/pki/ca-trust/extracted/openssl/

It should contain a file with about the following size:
-r--r--r--. 1 root root 296995 May 31 09:38 ca-bundle.trust.crt

That file is dynamically used by 
 /etc/alternatives/libnssckbi.so.x86_64 -> /usr/lib64/pkcs11/p11-kit-trust.so
to provide the list of trusted root certs for NSS in Firefox.

If that file were missing, it could be created/updated by running (as root):
# update-ca-trust
and restart firefox,


(In reply to Daniel Walsh from comment #7)
> 
> How can I blow away the ones I said to
> trust to see if it is an SELinux issue.


Do you mean, how you could delete the overrides that you had added in Firefox?

To delete them, use:
- menu edit/preferences/advanced/encryption/view certificates
- click the "servers tab"

That UI isn't the best, the display includes things that you cannot delete. However, ignore the entries with a * in the second column. All other entries are your overrides that you could delete.

While you're at it, in this same certificate manager window, click the "Authorities" tab.

Scroll down to Verisign. You should see several entries that have "default trust" in the second column. Can you confirm?

Comment 9 Kai Engert (:kaie) (inactive account) 2013-05-31 19:14:30 UTC
> While you're at it, in this same certificate manager window, click the
> "Authorities" tab.
> 
> Scroll down to Verisign. You should see several entries that have "default
> trust" in the second column. Can you confirm?

Most entries in that list should be marked as trusted already. Click a few of them, one after the other, and click the "edit trust" button for each of them. the window that opens should have the first item checked, for many of them.

I currently have no real idea what might be broken on your system.

One idea is to visit sites that use certs from different CAs, in the hope that only some CAs are not trusted on your system:

Do you get an error for each of the following links?

- https://startssl.com/ (startcom)
- https://www.paypal.com/ (verisign)
- https://bugzilla.redhat.com (geotrust)
- https://code.google.com/ (equifax)
- https://mobile2025.cybertrust.ne.jp/ (cybertrust)

If some work, then maybe there have been changes made to the trust settings in your firefox profile.

If none of them work, then we have a general problem. Maybe your firefox doesn't load the libnssckbi.so module (it's not a link time dependency, it's dynamically loaded).

Comment 10 Kai Engert (:kaie) (inactive account) 2013-05-31 19:16:55 UTC
$ strace -f firefox 2>&1 |egrep "nssckbi|p11-kit"

should print:

[pid 18825] open("/lib64/libnssckbi.so", O_RDONLY|O_CLOEXEC) = 43
[pid 18825] stat("/usr/share/pki/ca-trust-source/ca-bundle.supplement.p11-kit", {st_mode=S_IFREG|0644, st_size=2117, ...}) = 0
[pid 18825] open("/usr/share/pki/ca-trust-source/ca-bundle.supplement.p11-kit", O_RDONLY) = 44


$ ls -l /lib64/libnssckbi.so 

should print:

lrwxrwxrwx. 1 root root 38 May 27 08:43 /lib64/libnssckbi.so -> /etc/alternatives/libnssckbi.so.x86_64

Comment 11 Kai Engert (:kaie) (inactive account) 2013-05-31 19:21:52 UTC
I have asked for a lot of tests, and I'm interested in the results of the above.

However, here is one more test, which can be used to test Firefox with default settings:

- firefox -no-remote -CreateProfile test
- firefox -no-remote -P test

Try to load pages in this default firefox. If it works, then your old profile is messed up. If it is, I'd be interested to analyze it.

Comment 12 Andrew Hutchings 2013-06-03 19:00:11 UTC
Coming over from bug #960230 since I'm seeing this without suspend on Firefox, Thunderbird, Chrome and Evolution.  Happens to me once or twice a day.  Restarting any of these apps fixes it for that app.

What would you like me to do when it happens again?

Comment 13 Andrew Hutchings 2013-06-03 19:01:00 UTC
Also in my case SELinux is disabled (since some proprietary stuff I need to run for HP is difficult to run with SELinux)

Comment 14 T.C. Hollingsworth 2013-06-03 21:55:28 UTC
Dan, do you by chance have Google Chrome installed?  We were discussing this issue in the other bug and we think that might be related to this.

Comment 15 Kai Engert (:kaie) (inactive account) 2013-06-04 00:34:25 UTC
export P11_KIT_DEBUG=all
firefox -no-remote > log 2>&1

Load a page that gives you the errors.

Afterwards, please look at the logfile (or send it to me).
Does it talk about errors?

Comment 16 Bastiaan Jacques 2013-06-05 08:19:25 UTC
I also have this problem on both my F19 machines: after a variable runtime (between 15 minutes and 24 hours), the certificate errors start to appear everywhere, and restarting Firefox solves it.

I don't have Chrome installed on either machine, but I have Chromium on one. SELinux is enabled on both machines.

Comment 17 Stef Walter 2013-06-05 08:47:44 UTC
(In reply to Kai Engert (:kaie) from comment #15)
> export P11_KIT_DEBUG=all
> firefox -no-remote > log 2>&1
> 
> Load a page that gives you the errors.
> 
> Afterwards, please look at the logfile (or send it to me).
> Does it talk about errors?

Dan, could you please provide this information? Doing so would allow us to identify if this is a duplicate of bug #969463.

Comment 18 Kai Engert (:kaie) (inactive account) 2013-06-05 10:28:31 UTC
> Doing so would allow us to
> identify if this is a duplicate of bug #969463.

Stef means "duplicate of bug 960230".

I believe everyone having reported intermittent failures is likely to be affected by bug 960230.

The remaining issue is: Are there situations where it breaks immediately?

Stef, I propose to provide to prepare an RPM build with the fix for bug 960230 and ask everyone to install it. Maybe it will solve Dan's problem already.

Comment 19 Stef Walter 2013-06-05 14:33:09 UTC
Could you test this update, and post positive karma if it fixes the issue?

https://admin.fedoraproject.org/updates/p11-kit-0.18.3-1.fc19

It fixes other (minor) things as well, so no need to leave negative karma unless it breaks.

Thank you.

Comment 20 Daniel Walsh 2013-06-20 17:48:41 UTC
Sorry just back from the Summit.  I have the p11-kit installed and have not noticed the problem recently.

I am on Rawhide now.

rpm -q p11-kit
p11-kit-0.18.3-1.fc20.x86_64
 rpm -q firefox
firefox-21.0-3.fc19.x86_64

Comment 21 Stef Walter 2013-06-21 14:30:44 UTC
Thanks Dan. Closing as a duplicate.

*** This bug has been marked as a duplicate of bug 960230 ***