Bug 1752303 - Firefox shows DigiCert error if homepage set to https://duckduckgo.com
Summary: Firefox shows DigiCert error if homepage set to https://duckduckgo.com
Keywords:
Status: ON_QA
Alias: None
Product: Fedora
Classification: Fedora
Component: nss
Version: 30
Hardware: x86_64
OS: Other
unspecified
medium
Target Milestone: ---
Assignee: Daiki Ueno
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1761161 1766895 1767688 1773327 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-09-15 21:16 UTC by Daniel Bowling
Modified: 2019-12-05 18:46 UTC (History)
60 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:


Attachments (Terms of Use)
Screen cap of error (179.56 KB, image/png)
2019-09-15 21:16 UTC, Daniel Bowling
no flags Details
nss log (32.83 KB, text/plain)
2019-10-30 09:39 UTC, Martin Stransky
no flags Details
p11-kit log (370.25 KB, text/plain)
2019-10-30 10:21 UTC, Gerald Zehetner
no flags Details
requested log with P11_KIT_DEBUG=trust (433.51 KB, text/plain)
2019-10-30 10:31 UTC, Martin Stransky
no flags Details
p11-kit log test-build firefox-70.0-2.fc30.x86_64 (422.17 KB, text/plain)
2019-11-04 12:13 UTC, Gerald Zehetner
no flags Details
Log with P11_KIT_DEBUG=trust MOZ_LOG="certverifier:5" (530.98 KB, text/plain)
2019-12-05 09:09 UTC, Cezary Zemis
no flags Details


Links
System ID Priority Status Summary Last Updated
Mozilla Foundation 1593167 '--' 'NEW' 'Intermittent mis-reporting potential security risk SEC_ERROR_UNKNOWN_ISSUER' 2019-12-04 12:58:40 UTC

Description Daniel Bowling 2019-09-15 21:16:38 UTC
Created attachment 1615400 [details]
Screen cap of error

Description of problem: Every other time I open Firefox (that is half the time, but occurs reliably every other time) I get a warning stating the site is not safe and "This issue is caused by DigiCert Global Root CA, which is either software on your computer or your network." If I quit Firefox completely then re-open it will work fine (loads up https://duckduckgo.com as this is my homepage). If I then quit Firefox completely then re-open it will occur again. I've tried this several times and it seems to reliably happen every other time I open Firefox as long as the homepage is set to https://duckduckgo.com. I have other machines running Slackware and one Mac with Firefox ESR and Firefox respectively and they do not exhibit this issue so I believe it may be a setting in about:config or some sort of build option used with the Fedora Firefox package


Version-Release number of selected component (if applicable): Fedora 30/Firefox 69


How reproducible: Every other time


Steps to Reproduce:
1. Set homepage to "https://duckduckgo.com"
2. Close Firefox
3. Open and close Firefox several times in a row

Actual results:
Every other time a warning is displayed

Expected results:
https://duckduckgo.com should load with no warnings

Additional info:
If you continue to browse to other HTTPS sites signed by DigiCert a similar warning is showed unless you (again) completely quit Firefox and re-open with no error; then DigiCert signed certificates will work as expected

Comment 1 Raphael Fuchs 2019-10-26 09:07:11 UTC
I can confirm this issue (Fedora 30/Firefox 70). It also happens with bugzilla.redhat.com, probably because it uses a DigiCert certificate.

Creating a new Firefox profile did not help. My workaround is to repeatedly launch Firefox. In roughly 20% of cases, I will not get a (persistent) warning.

Comment 2 red-hat-bugzilla 2019-10-26 18:38:45 UTC
The specific error reported by Firefox when this occurs for me is MOZILLA_PKIX_ERROR_MITM_DETECTED.

Comment 3 Martin Stransky 2019-10-26 20:11:22 UTC
Yes, I do see that too, on bugzilla.mozilla.org page.

Comment 4 Martin Stransky 2019-10-26 20:13:30 UTC
Ueno, any idea here?

Comment 5 Matthew F. 2019-10-29 05:25:48 UTC
Would like to add in I'm running Fedora 30 (KDE spin though I don't believe that matters) as well with Firefox version 70 now and it has the same issue when I resume a session with firefox open to trello.com which also uses digicert. The way I was able to fix this for myself for the time being it looks like was to actually go and manually install the Digicert Global Root CA certificate listed by the error from their site and add it within Firefox itself. Couldn't figure out a reason why this specifically was causing such an issue.

Comment 6 Daiki Ueno 2019-10-29 10:56:45 UTC
(In reply to Martin Stransky from comment #4)
> Ueno, any idea here?

No idea, I'm afraid. But I suspect that the MOZILLA_PKIX_ERROR_MITM_DETECTED error might be a red herring; the actual problem is that sometimes Firefox couldn't find the root cert from the system trust store, something similar to:
https://bugzilla.mozilla.org/show_bug.cgi?id=1530429

Maybe good to ask on the upstream bugzilla (if it is not done).

Comment 7 John 2019-10-29 13:50:33 UTC
Happens in Fedora 31 / Firefox 70.0 too when visiting duckduckgo.com. 

Just when I was editing this post, I refreshed the duckduckgo tab. The problem was gone.  What happened?

Comment 8 John 2019-10-29 13:55:19 UTC
Something interesting also happened. I am not sure it's related. I did a "dnf update" and a gpg check also failed.

Problem opening package rpmfusion-nonfree-appstream-data-31-1.fc31.noarch.rpm
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: GPG check FAILED

Now the cert problem is gone, I clean the dnf cache, run "dnf update" again, and gpg check succeeded.

Comment 9 Martin Stransky 2019-10-30 09:30:00 UTC
*** Bug 1766895 has been marked as a duplicate of this bug. ***

Comment 10 Martin Stransky 2019-10-30 09:39:41 UTC
Created attachment 1630467 [details]
nss log

Comment 11 Martin Stransky 2019-10-30 09:41:31 UTC
Ueno, there's a nss log attached. Can you please briefly check that if it's something Fedora specific/related to our system nss or if we should report that upstream? Thanks.

Comment 12 Gerald Zehetner 2019-10-30 10:21:22 UTC
Created attachment 1630507 [details]
p11-kit log

I just ran env P11_KIT_DEBUG=all /usr/bin/firefox >& debug.log and reproduced the error.

See log.

Comment 13 Martin Stransky 2019-10-30 10:31:20 UTC
Created attachment 1630515 [details]
requested log with P11_KIT_DEBUG=trust

Comment 14 Daiki Ueno 2019-10-30 12:08:57 UTC
Thank you for the logs, but I still can't tell where is the problem.

After comparing the p11-kit logs between the successful case and the unsuccessful case, the difference seems to be the following queries missing in the latter case (in terms of DigiCert SHA2 Secure Server CA):

  sys_C_FindObjectsInit: in: 18, (4) [ { CKA_TOKEN = (1) "\x01" }, { CKA_CLASS = CKO_CERTIFICATE }, { CKA_ISSUER = (79) "0M1\x0b0\t\x06\x03U\x04\x06\x13\x02US1\x150\x13\x06\x03U\x04\n\x13\x0cDigiCert Inc1'0%\x06\x03U\x04\x03\x13\x1eDigiCert SHA2 Secure Server CA" }, { CKA_SERIAL_NUMBER = (18) "\x02\x10\x0c\xa9\xc6Ca\xbf\xc9*y\xb1\xdd\x9c\xfb\x9eH\xec" } ]
  sys_C_FindObjectsInit: in: 18, (4) [ { CKA_TOKEN = (1) "\x01" }, { CKA_CLASS = CKO_CERTIFICATE }, { CKA_ISSUER = (79) "0M1\x0b0\t\x06\x03U\x04\x06\x13\x02US1\x150\x13\x06\x03U\x04\n\x13\x0cDigiCert Inc1'0%\x06\x03U\x04\x03\x13\x1eDigiCert SHA2 Secure Server CA" }, { CKA_SERIAL_NUMBER = (16) "\x0c\xa9\xc6Ca\xbf\xc9*y\xb1\xdd\x9c\xfb\x9eH\xec" } ]
  sys_C_FindObjectsInit: in: 18, (4) [ { CKA_TOKEN = (1) "\x01" }, { CKA_CLASS = CKO_NSS_TRUST }, { CKA_ISSUER = (79) "0M1\x0b0\t\x06\x03U\x04\x06\x13\x02US1\x150\x13\x06\x03U\x04\n\x13\x0cDigiCert Inc1'0%\x06\x03U\x04\x03\x13\x1eDigiCert SHA2 Secure Server CA" }, { CKA_SERIAL_NUMBER = (16) "\x0c\xa9\xc6Ca\xbf\xc9*y\xb1\xdd\x9c\xfb\x9eH\xec" } ]

while it has a query without CKA_SERIAL_NUMBER:

  sys_C_FindObjectsInit: in: 18, (3) [ { CKA_CERTIFICATE_TYPE = CKC_X_509 }, { CKA_CLASS = CKO_CERTIFICATE }, { CKA_SUBJECT = (79) "0M1\x0b0\t\x06\x03U\x04\x06\x13\x02US1\x150\x13\x06\x03U\x04\n\x13\x0cDigiCert Inc1'0%\x06\x03U\x04\x03\x13\x1eDigiCert SHA2 Secure Server CA" } ]

I suppose the next step would be to check how/where these queries are constructed, which I suggest hearing from upstream.

Comment 15 Martin Stransky 2019-10-31 22:40:43 UTC
I'm going to create a test build with in-tree nss to check if that fixes this issue.

Comment 16 Martin Stransky 2019-10-31 22:45:03 UTC
(In reply to Martin Stransky from comment #15)
> I'm going to create a test build with in-tree nss to check if that fixes
> this issue.

Added to firefox-70.0-2 builds.

Comment 17 Jonathan Haas 2019-11-01 07:39:40 UTC
I'm also experiencing this error on firefox-70.0-1.fc31.x86_64 after upgrading to Fedora 31

Sires that randomly fail to load are for example reddit.com, bugzilla.mozilla.org

Comment 18 Martin Stransky 2019-11-01 09:58:47 UTC
New test builds are available here:

F30: https://koji.fedoraproject.org/koji/taskinfo?taskID=38697234
F31: https://koji.fedoraproject.org/koji/taskinfo?taskID=38697225

please check them if you see any issue. It uses in-tree nss so you loose nss system certificate integration which causes the bug we see here.

Comment 19 Martin Stransky 2019-11-01 10:01:31 UTC
*** Bug 1767688 has been marked as a duplicate of this bug. ***

Comment 20 Martin Stransky 2019-11-01 10:08:19 UTC
I guess you may see bug 1766340 with the builds above.

Comment 21 Martin Stransky 2019-11-01 10:10:53 UTC
Bug 1648617 may be a dupe.

Comment 22 Scott Young 2019-11-02 02:40:11 UTC
I'm not a Red Hat user, so apologies if this comment is not helpful, but I thought this might confirm that the issue is not Red Hat-specific. I experienced the same bug on Arch Linux today, in firefox-70.0.1-1. I previously experienced it on 25 October 2019 in firefox-70.0-1. In both cases the error message referred to the DigiCert Global Root CA, did not affect other browsers, and disappeared after restarting Firefox, meaning that I could not reproduce it for further diagnosis. I had not noticed the bug before upgrading to firefox-70.0-1 on 22 October 2019.

Comment 23 Jonathan Haas 2019-11-02 06:35:51 UTC
See also https://bbs.archlinux.org/viewtopic.php?id=247067

Comment 24 Gerald Zehetner 2019-11-04 12:11:38 UTC
I upgraded to the test-build and I still can reproduce the error.

rpm -q firefox
firefox-70.0-2.fc30.x86_64

Comment 25 Gerald Zehetner 2019-11-04 12:13:43 UTC
Created attachment 1632523 [details]
p11-kit log test-build firefox-70.0-2.fc30.x86_64

Comment 26 Gerald Zehetner 2019-11-04 12:35:12 UTC
Also see: https://github.com/clearlinux/distribution/issues/1006

Comment 27 Michael Catanzaro 2019-11-04 15:04:44 UTC
FWIW I've noticed this bug occasionally breaks Firefox Sync, because accounts.mozilla.com also uses DigiCert. I noticed because my connected devices did not appear under "Send Tab to Device". Of course it's very sporadic and usually works fine, just like with DuckDuckGo. Certainly it has nothing to do with the homepage setting.

Comment 28 Michael Catanzaro 2019-11-04 15:05:51 UTC
(In reply to Michael Catanzaro from comment #27)
> FWIW I've noticed this bug occasionally breaks Firefox Sync, because
> accounts.mozilla.com also uses DigiCert.

Sorry, I meant accounts.firefox.com.

Comment 29 Martin Stransky 2019-11-05 11:34:21 UTC
*** Bug 1761161 has been marked as a duplicate of this bug. ***

Comment 30 Gerald Zehetner 2019-11-05 11:50:03 UTC
After reboot I can't reproduce the bug with the test-build firefox-70.0-2.fc30.x86_64. Maybe it didn't load the new version without reboot.

Comment 31 Martin Stransky 2019-11-07 12:56:08 UTC
(In reply to Gerald Zehetner from comment #30)
> After reboot I can't reproduce the bug with the test-build
> firefox-70.0-2.fc30.x86_64. Maybe it didn't load the new version without
> reboot.

Okay, moving to nss as it can't be reproduced with Firefox in-tree nss.

Comment 32 Jan Alexander Steffens 2019-11-07 13:03:40 UTC
NSS recently added two new fields to the certificate DB: https://hg.mozilla.org/projects/nss/rev/52024949df95c7147c4995bb8ddc094027636001

I guess p11-kit needs to be updated to support these fields, much like it was for CKA_NSS_MOZILLA_CA_POLICY.

Comment 33 Daiki Ueno 2019-11-07 13:53:16 UTC
(In reply to Jan Alexander Steffens from comment #32)
> NSS recently added two new fields to the certificate DB:
> https://hg.mozilla.org/projects/nss/rev/
> 52024949df95c7147c4995bb8ddc094027636001
> 
> I guess p11-kit needs to be updated to support these fields, much like it
> was for CKA_NSS_MOZILLA_CA_POLICY.

Certainly, but I don't think it is related to this bug as there is no released version that "uses" those attributes.

Comment 34 Lars S. Jensen 2019-11-07 19:28:24 UTC
Until yesterday the firefox-70.0-2.fc31.x86_64 worked without problems with DigiCert's CA for a week.
Yesterday I was force to accept exception to access DigiCert CA's sites.
I have many page preloaded (>100) and each time firefox starts, it change the loading sequence of the pages and the CA's. 
I think the problem is related sequence of CA chain loading in firefox maybe it is GC/allocation bug.

I use uBlocker Origin and NoScript add-on in firefox.

Comment 35 Michael Catanzaro 2019-11-07 20:20:15 UTC
(In reply to Lars S. Jensen from comment #34)
> I have many page preloaded (>100) and each time firefox starts, it change
> the loading sequence of the pages and the CA's. 
> I think the problem is related sequence of CA chain loading in firefox

Interesting. What changes based on the order the pages are loaded is the state of your NSS certificate cache. Based on this, I think I might have found a reproducer: remove all DigiCert intermediate certificates from the certificate cache. Normally this should be safe to do, but due to this bug it causes Firefox to break. I have no clue what is going wrong here between NSS and p11-kit, but hopefully this will help any developers unable to reproduce.

Since loading any webpage could affect this, best to first close all tabs except duckduckgo.com and set Firefox to restore previous tabs on startup to ensure a clean environment. Then: go to Preferences -> Privacy & Security -> scroll to the bottom -> View Certificates... -> Authorities. Scroll down to DigiCert Inc. Delete or Distrust every DigiCert certificate that is labeled Software Security Device. (You should *probably* only need to delete one, DigiCert SHA2 Secure Server CA, but deleting them all is what I just tested.) When doing this, be very careful not to Delete or Distrust any certificate labeled Default Trust, because those are trust roots that should not be messed with. The UI here is very confusing, but I believe the Software Security Device label indicates an intermediate certificate that is not included in ca-certificates and which can safely be deleted, because it will just be cached again next time it is encountered. The button will only delete these certificates from the cache, not cause these certificates to be distrusted, as would happen if deleting a root. (Thus, deleting the intermediate certificate is normally almost pointless. Only in combination with this bug does cause trust errors. I think.) Finally, restart Firefox. Firefox will initially display duckduckgo.com with no issues, but only because it is being restored from previous session state cache. Now, refresh the page and you should encounter the certificate error.

One caveat: I don't know how to get my Firefox out of this broken state, and that means I can't test to be certain that following those steps once again will successfully break my Firefox. I just know that it seems to have worked for me the first time I tried it.

Comment 36 Lars S. Jensen 2019-11-07 21:53:12 UTC
Removing all the windows and tabs and then loading duckduckgo.com trigger the error:
duckduckgo.com is most likely a safe site, but a secure connection could not be established. This issue is caused by DigiCert Global Root CA, which is either software on your computer or your network.

Comment 37 Lars S. Jensen 2019-11-07 22:00:56 UTC
And the certificate that triggers the error is always of the type: Software Security Device.

Comment 38 Lars S. Jensen 2019-11-07 22:10:12 UTC
But not all, this works: DigiCert ECC Secure Server CA

Comment 39 Martin Stransky 2019-11-08 06:45:03 UTC
New test builds with in-tree nss (firefox-70.0.1-3.nss) are here:
https://koji.fedoraproject.org/koji/packageinfo?packageID=37

Comment 40 Lars S. Jensen 2019-11-08 13:52:08 UTC
(In reply to Martin Stransky from comment #39)
> New test builds with in-tree nss (firefox-70.0.1-3.nss) are here:
> https://koji.fedoraproject.org/koji/packageinfo?packageID=37

The version restore duckduckgo.com (DigiCert SHA2 Secure Server CA) and lenovo.com (DigiCert ECC Secure Server CA).
I will continue testing firefox-70.0.1-3.nss

Comment 41 aGgM6kTti 2019-11-09 03:51:57 UTC
Using firefox-30.0.1-3.nss.fc30 I had success browsing reddit.com for a while, but suddenly I am getting the "Software is Preventing Firefox From Safely Connecting to This Site" error referencing DigiCert Global Root CA again. It seems it's not a complete fix.

Comment 42 aGgM6kTti 2019-11-09 03:52:33 UTC
Correction: firefox-70.0.1-3.nss.fc30

Comment 43 Lars S. Jensen 2019-11-10 12:46:20 UTC
It is not fixed with firefox-70.0.1-3.nss -- www.samsung.com(GeoTrust RSA CA 2018):
Firefox does not trust www.samsung.com because its certificate issuer is unknown, the certificate is self-signed, or the server is not sending the correct intermediate certificates.

Comment 44 charles profitt 2019-11-10 20:19:03 UTC
Added link to Mozilla Bug
https://bugzilla.mozilla.org/show_bug.cgi?id=1593167

Comment 45 Jason Montleon 2019-11-12 15:36:38 UTC
I have firefox-70.0-1.fc31.x86_64

What I am observing is that if I open Firefox and imemdiately try to browse to a site like github.com or koji.fedoraproject.org I will get an error. If I wait for few seconds I can usually browse to the site normally.

If instead after opening Firefox I wait several seconds I will not see an error at all.

Comment 46 Stefan Dj 2019-11-12 20:34:38 UTC
Hi,

This issue is easily reproducible with following packages in Fedora 30.

[root@localhost ~]# rpm -qa|grep -E '^firefox|^nss|^p11'
firefox-70.0-1.fc30.x86_64
nss-sysinit-3.47.0-2.fc30.x86_64
nss-mdns-0.14.1-3.fc30.x86_64
nss-3.47.0-2.fc30.x86_64
p11-kit-0.23.16.1-1.fc30.x86_64
p11-kit-trust-0.23.16.1-1.fc30.x86_64
nss-util-3.47.0-2.fc30.x86_64
nss-softokn-3.47.0-2.fc30.x86_64
nss-softokn-freebl-3.47.0-2.fc30.x86_64
p11-kit-server-0.23.16.1-1.fc30.x86_64

Am using 'Tab Session Manager' add-on so it instantly open session for few sites with DigiCert CA when run firefox, and all of them are 'prevented' from
opening.

As Jason mention if I wait a little after firefox is started, then start session all sites loads and everything works as expected.

Comment 47 Brian Lane 2019-11-12 23:40:12 UTC
(In reply to Jason Montleon from comment #45)
> I have firefox-70.0-1.fc31.x86_64
> 
> What I am observing is that if I open Firefox and imemdiately try to browse
> to a site like github.com or koji.fedoraproject.org I will get an error. If
> I wait for few seconds I can usually browse to the site normally.
> 
> If instead after opening Firefox I wait several seconds I will not see an
> error at all.

My experience was different, even after waiting a bit (while scrambling to figure out what was going on) they would not load for me.
Downgrading to firefox-69 fixed it for me on my F30 system.

Comment 48 Ankur Sinha (FranciscoD) 2019-11-13 18:14:36 UTC
(In reply to Jason Montleon from comment #45)
> I have firefox-70.0-1.fc31.x86_64
> 
> What I am observing is that if I open Firefox and imemdiately try to browse
> to a site like github.com or koji.fedoraproject.org I will get an error. If
> I wait for few seconds I can usually browse to the site normally.
> 
> If instead after opening Firefox I wait several seconds I will not see an
> error at all.

+1, this seems to be happening here too. If I start FF and then wait for the "timer" thing to finish in the gnome-shell top panel (where it says "firefox") and then load duckduckgo.com or netflix.com, everything works just fine. If, however, something tries to open before the timer is finished---a tab from a previous session, for example, the the cert error thing keeps coming up for the rest of the session. So, ff is trying to load something here? How can one get some information on what it is doing maybe?

Comment 49 Florian Apolloner 2019-11-15 14:28:33 UTC
(In reply to Stefan Dj from comment #46)
> Am using 'Tab Session Manager' add-on so it instantly open session for few
> sites with DigiCert CA when run firefox, and all of them are 'prevented' from
> opening.

Pinning a tab (rightclick -> "Pin Tab") makes it easy to trigger this without an extra addon. Fwiw, when running into this issue https://accounts.firefox.com will show http key pinning issues :)

Comment 50 Andy Lutomirski 2019-11-17 16:29:32 UTC
*** Bug 1773327 has been marked as a duplicate of this bug. ***

Comment 51 Andy Lutomirski 2019-11-17 16:32:24 UTC
stransky, I'm getting this too, but it's erratic enough that I can't easily tell if your test build fixes it.

BTW, this is a better link:

https://koji.fedoraproject.org/koji/buildinfo?buildID=1406664

Comment 52 Andy Lutomirski 2019-11-17 23:48:16 UTC
Nope, not quite fully fixed.

$ rpm -qa firefox
firefox-70.0-2.fc31.x86_64

I removed cert8.db and cert9.db and restarted Firefox, and the first few webpages I tried did not work.  (news.ycombinator.com gave the MITM error and www.wayfair.com gave a different error.)  Then I loaded bugzilla.redhat.com in a different tab in the same browser instance and it worked.  Then I loaded bugzilla.mozilla.org and *that* worked.  Then I reloaded the other two tabs, and they reloaded successfully, too.

Something is wonky with certificate handling,

Comment 53 Rudolf E. Steiner 2019-11-18 12:30:36 UTC
Same problem/behavior on Fedora 30 with these packages:

     firefox-70.0.1-4.fc30.x86_64
     nss-3.47.0-3.fc30.x86_64
     nss-mdns-0.14.1-3.fc30.x86_64
     nss-pem-1.0.5-1.fc30.x86_64
     nss-softokn-3.47.0-3.fc30.x86_64
     nss-softokn-freebl-3.47.0-3.fc30.i686
     nss-softokn-freebl-3.47.0-3.fc30.x86_64
     nss-sysinit-3.47.0-3.fc30.x86_64
     nss-tools-3.47.0-3.fc30.x86_64
     nss-util-3.47.0-3.fc30.x86_64
     p11-kit-0.23.16.1-1.fc30.x86_64
     p11-kit-server-0.23.16.1-1.fc30.x86_64
     p11-kit-trust-0.23.16.1-1.fc30.x86_64

Comment 54 Daiki Ueno 2019-11-18 15:47:36 UTC
(In reply to Andy Lutomirski from comment #52)

> Something is wonky with certificate handling,

I have been investigating this for weeks but little progress so far.  What I'm almost sure are:
- This is caused by some behavior difference between the stock libnssckbi.so and p11-kit-trust.so, possibly a race condition, or output order
- When the page is successfully loaded, CKA_NSS_MOZILLA_CA_POLICY is queried, while it isn't in the failed cases (as seen in the log if you run P11_KIT_DEBUG=trust firefox ...)
- The problem can be reproduced even with Firefox 69, but with much lower frequency

For the second point, I started scratch-build of Firefox with debug output; I also tried to build the trunk locally but couldn't reproduce the problem with it.

Comment 55 Andy Lutomirski 2019-11-18 20:05:37 UTC
Is there a Fedora Firefox build that simply disables system certificate store integration entirely?  Being able to reliably use the web seems more important than having system certificates be trusted.

Comment 56 Martin Stransky 2019-11-19 12:20:59 UTC
(In reply to Andy Lutomirski from comment #55)
> Is there a Fedora Firefox build that simply disables system certificate
> store integration entirely?  Being able to reliably use the web seems more
> important than having system certificates be trusted.

I fired firefox-70.0.1-5.nss builds for that, will be available at http://koji.fedoraproject.org/koji/packageinfo?packageID=37

Comment 57 Villy Kruse 2019-11-20 11:38:30 UTC
There is something strange about the "DigiCert SHA2 Secure Server CA" certificate.  It is supposed to be signed by "DigiCert Global Root CA",
but if you in Firefox look in the Certificate Manager Page and find  the "DigiCert SHA2 Secure Server CA" certificate you wil see in the View->Details page you see the Certificate Hierarchy does not point pack to the signing root certificate.

Could that have something to do with the problem:


Also on that certificate, you can Edit Trust and enable the trust settings.  This seems to make the problem go away for that certificate.

Comment 58 Geoff 2019-11-24 14:23:11 UTC
On a clean install of the KDE Plasma spin of Fedora 31, I see this error consistently. Looking at `about:config` on my freshly installed system, I see that security.pki.mitm_canary_issuer is set to "CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US". Both of the user accounts on the system have two firefox profiles. One is "default" and one is "default-release". For some reason, on one of the accounts, prefs.js in the "default-release" profile contains that and the other doesn't.

```
./fhx3n5ez.default-release/prefs.js:user_pref("security.pki.mitm_canary_issuer", "CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US");
```

Clearing that value resolves the issue.

I've got no idea how it could've gotten set in the first place, though, nor why it would be different on one account than the other. I am quite certain that I did not go into about:config and set it myself, though.

Comment 59 charles profitt 2019-11-25 03:39:15 UTC
It appears the Mozilla folks have made some progress on this.
https://bugzilla.mozilla.org/show_bug.cgi?id=1593167

Comment 60 Jonathan Haas 2019-11-25 10:13:53 UTC
TLDR: It looks like this is an actual upstream Firefox issue, where Firefox incorrectly caches temporary invalid certificates, and not an NSS issue. The issue seems to just appear more often on Fedora, because NSS certificates load slower and thus the chance is higher that a temporary certificate gets loaded instead of the real one.

Comment 61 Villy Kruse 2019-11-25 13:30:46 UTC
(In reply to Jonathan Haas from comment #60)
> TLDR: It looks like this is an actual upstream Firefox issue, where Firefox
> incorrectly caches temporary invalid certificates, and not an NSS issue. The
> issue seems to just appear more often on Fedora, because NSS certificates
> load slower and thus the chance is higher that a temporary certificate gets
> loaded instead of the real one.

How it actually works (if it works correctly):

The website will include its own certificates and the signing certificates as well except the top level CA certificate:

For the bugzilla page, which just showed the proble the certificate chain is:

openssl x509 -noout -subject -issuer -in 1
subject=C = US, ST = North Carolina, L = Raleigh, O = "Red Hat, Inc.", OU = Information Technology, CN = *.redhat.com
issuer=C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 High Assurance Server CA

openssl x509 -noout -subject -issuer -in 2
subject=C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 High Assurance Server CA
issuer=C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA

openssl x509 -noout -subject -issuer -in 3
subject=C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA
issuer=C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA

The files 1, 2, and 3 are the involved certificates exported from Firefox.

"CN = DigiCert SHA2 High Assurance Server CA" comes from the website and is saved in cert9.db (or cert8.db).
"CN = *.redhat.com" also arrives from the website and not stored anywhere.
"CN = DigiCert High Assurance EV Root CA" comes from a local certificate database and it is used to verify "CN = DigiCert SHA2 High Assurance Server CA" which in turn verifies "CN = *.redhat.com".

Some race condition causes the top level certificate not to be found in time when the first verification is done, thus the broken certificate chain, and the web pages therefore marked as "not trusted".

Comment 62 Fedora Update System 2019-11-29 06:14:19 UTC
FEDORA-2019-ff27bbf69a has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-ff27bbf69a

Comment 64 Eric L. 2019-11-29 16:42:39 UTC
@Jan Hutař, installing the beta nss packages didn't help and downgrading to Firefox 69.0.3-2 did not solve this problem for me.

@Geoff I have the same setting in my about:config and profile.js. Clearing the value and ensuring it's deleted from profiles.js... does nothing. It gets added back as soon as Firefox is launched.

Something I updated in the past two days on my Fedora 30 system causes this issue. I had already been running Firefox 70 for a while, so I'm flummoxed as to why I can't access Fastmail or DuckDuckGo this morning.

Comment 65 Eric L. 2019-11-29 16:52:38 UTC
Edit: Both the Firefox 71 beta and Firefox 72 nightly resolve the issue. The DigiCert CA is no longer mentioned as the mitm_canary_issuer in about:config and browsing the DigiCert sites (including Mozilla.org!) works as intended.

Comment 66 Daiki Ueno 2019-11-29 17:32:50 UTC
(In reply to Eric L. from comment #64)
> @Jan Hutař, installing the beta nss packages didn't help and downgrading to
> Firefox 69.0.3-2 did not solve this problem for me.

Interesting. Could you provide the exact versions of packages you are using, with: rpm -qa firefox nss p11-kit?

Comment 67 Eric L. 2019-11-29 18:10:11 UTC
Hey, Daiki. I just read your comment on https://bugzilla.mozilla.org/show_bug.cgi?id=1593167#c27 , coincidentally.

Here are my current packages:
```
[root@schwarz eric]# rpm -qa firefox nss p11-kit
p11-kit-0.23.16.1-1.fc30.x86_64
firefox-70.0.1-4.fc30.x86_64
p11-kit-0.23.16.1-1.fc30.i686
nss-3.47.1-2.fc30.i686
nss-3.47.1-2.fc30.x86_64
```

Because Firefox now uses separate profiles for beta and nightly installs, I spent a load of time fiddling with it to try and get the beta install functioning like my normal install. LastPass doesn't work on the beta, for some reason, so I gave up and switched back to 70.0.1-4. I stumbled across a fix on the Manjaro forums https://forum.manjaro.org/t/digicert-invalid-certificate-in-firefox-after-system-upgrade-potential-rogue-certificate/109242/13

If you set the relevant cert to have the "This certificate can identify websites" setting, Firefox 70 can again launch a session with tabs behind Digicert CAs.

Comment 68 Fedora Update System 2019-11-30 01:19:42 UTC
nss-3.47.1-2.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-ff27bbf69a

Comment 69 Eric L. 2019-12-02 18:37:49 UTC
Installed a fresh copy of Fedora 31 on a different laptop. DuckDuckGo and all other DigiCert sites worked on the Firefox 69 + whatever NSS versions shipped with the FC31 XFCE spin. Doing a dnf update brings the same errors as the Firefox 70 + NSS 3.47.1-2 on Fedora 30.

Updated that Fedora 30 to Fedora 31, where the bug obviously persists. The nss-3.47.1-2.fc31 testing update did not fix the problem in FC31.

Comment 70 charles profitt 2019-12-04 13:02:00 UTC
Mozilla has updated the upstream bug with the following details:

Bug 1593167, certdb: propagate trust information if trust module is loaded afterwards 
When the builtin trust module is loaded after some temp certs being created, these temp certs are usually not accompanied by trust information. This causes a problem in Firefox as it loads the module from a separate thread while accessing the network cache which populates temp certs.

This change makes it properly roll up the trust information, if a temp cert doesn't have trust information.

link: https://phabricator.services.mozilla.com/D54726

----

Bug 1593167, ensure that loadable certs are loaded when TransportSecurityInfo::Read is called 
Previously, there was a race condition: when the first time
CERT_NewTempCertificate is called, the nssckbi module might not be
loaded, and thus the lookup fails. In that case the function creates a
temp certificate without any trust and put it in the cache, which
later interferes with certificate chain validation.

link: https://phabricator.services.mozilla.com/D54135

Comment 71 Fedora Update System 2019-12-04 15:29:43 UTC
FEDORA-2019-ff27bbf69a has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-ff27bbf69a

Comment 72 Daiki Ueno 2019-12-04 15:37:22 UTC
(In reply to Eric L. from comment #69)
> Installed a fresh copy of Fedora 31 on a different laptop. DuckDuckGo and
> all other DigiCert sites worked on the Firefox 69 + whatever NSS versions
> shipped with the FC31 XFCE spin. Doing a dnf update brings the same errors
> as the Firefox 70 + NSS 3.47.1-2 on Fedora 30.
> 
> Updated that Fedora 30 to Fedora 31, where the bug obviously persists. The
> nss-3.47.1-2.fc31 testing update did not fix the problem in FC31.

We've come up with a simpler (and possibly more comprehensive) fix; please try 3.47.1-4 from:
https://bodhi.fedoraproject.org/updates/FEDORA-2019-8fbc65ef9e

If it doesn't work, provide logs by running firefox under: P11_KIT_DEBUG=trust MOZ_LOG="certverifier:5"

Comment 73 Fedora Update System 2019-12-05 01:23:22 UTC
nss-3.47.1-4.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-ff27bbf69a

Comment 74 Fedora Update System 2019-12-05 02:01:05 UTC
nss-3.47.1-4.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-8fbc65ef9e

Comment 75 Cezary Zemis 2019-12-05 09:01:57 UTC
The problem persists with the following packages installed:

Installed Packages
firefox.x86_64                                                                   70.0.1-4.fc30                                                    @updates        
nss.i686                                                                         3.47.1-4.fc30                                                    @updates-testing
nss.x86_64                                                                       3.47.1-4.fc30                                                    @updates-testing
nss-mdns.i686                                                                    0.14.1-3.fc30                                                    @fedora         
nss-mdns.x86_64                                                                  0.14.1-3.fc30                                                    @fedora         
nss-pem.x86_64                                                                   1.0.5-1.fc30                                                     @fedora         
nss-softokn.i686                                                                 3.47.1-4.fc30                                                    @updates-testing
nss-softokn.x86_64                                                               3.47.1-4.fc30                                                    @updates-testing
nss-softokn-freebl.i686                                                          3.47.1-4.fc30                                                    @updates-testing
nss-softokn-freebl.x86_64                                                        3.47.1-4.fc30                                                    @updates-testing
nss-sysinit.x86_64                                                               3.47.1-4.fc30                                                    @updates-testing
nss-tools.x86_64                                                                 3.47.1-4.fc30                                                    @updates-testing
nss-util.i686                                                                    3.47.1-4.fc30                                                    @updates-testing
nss-util.x86_64                                                                  3.47.1-4.fc30                                                    @updates-testing

Tested on https://www.mozilla.org/ (Error code: MOZILLA_PKIX_ERROR_MITM_DETECTED) and https://www.17track.net/ (Error code: SEC_ERROR_UNKNOWN_ISSUER)

Comment 76 Cezary Zemis 2019-12-05 09:09:00 UTC
Created attachment 1642301 [details]
Log with P11_KIT_DEBUG=trust MOZ_LOG="certverifier:5"

Comment 77 Cezary Zemis 2019-12-05 09:14:29 UTC
The same result with firefox-71.0-8.npgo.fc30.x86_64

Comment 78 Daiki Ueno 2019-12-05 14:37:54 UTC
(In reply to Cezary Zemis from comment #76)
> Created attachment 1642301 [details]
> Log with P11_KIT_DEBUG=trust MOZ_LOG="certverifier:5"

The log looks quite different from what we had seen; the trust information is correctly queried. Are you able to reproduce it with a fresh new profile (firefox -P) or on a different network?

Comment 79 Cezary Zemis 2019-12-05 18:30:59 UTC
Cannot reproduce on another account with new profile.

Comment 80 Cezary Zemis 2019-12-05 18:46:04 UTC
Solved by re-creating cert9.db in existing profile.


Note You need to log in before you can comment on or make changes to this bug.