Bug 2058661

Preamble:
I fully realize that it's very likely that this is part server configuration issue and/or part NSS, but I am a bit at loss as to where the exact problem lies and I'd love if you guys could throw some cluebones my way :)

Description of problem:
Starting with Firefox 94, opening the http url of an openshift operator called "openshift-gitops" inside an OCP cluster returns "Error code: SEC_ERROR_BAD_DER."

Chrome works without issues and so does Firefox 93 and curl. openssl seems to be fine with it (barring the unknown CA), so "openssl s_client -connect openshift-gitops-server-openshift-gitops.apps.bandini-dc.blueprints.rhecoeng.com:443" complains about the verification bits but nothing close to the DER-encoded message issue.

I sort of bisected the breakage between nightlies to have happened here:
* 94.0a1 (https://ftp.mozilla.org/pub/firefox/nightly/2021/09/2021-09-30-14-38-54-mozilla-central/firefox-94.0a1.en-US.linux-x86_64.tar.bz2) -> WORKS
* 94.0a1 (https://ftp.mozilla.org/pub/firefox/nightly/2021/10/2021-10-04-09-53-42-mozilla-central/firefox-94.0a1.en-US.linux-x86_64.tar.bz2) -> BROKEN

So the change must have happened (in the nightlies) between 2021-09-30 and 2021-10-04. I cloned the mercurial mozilla-central tree and since the nightly folders above contain the hg revisions. I went ahead and tried my luck with: hg log -r "85b7b3d04d9f249418325f02d4b93046724b496b:06e67beeafc265ff1aef7d033706a67d91ef0186"

The only change that sort of made sense was:
changeset:   593811:96a66f46f801
user:        Benjamin Beurdouche <bbeurdouche>
date:        Thu Sep 30 12:57:34 2021 +0000
summary:     Bug 1729163 - land NSS NSS_3_71_RTM UPGRADE_NSS_RELEASE, r=djackson DONTBUILD

Looking at the bug I see this changeset: https://hg.mozilla.org/mozilla-central/rev/330c22fc463e which seems potentially related so, as suspected, likely more of an NSS thing.

After some googling here and there I ran firefox with the following:
export NSPR_LOG_MODULES="nsHttp:3,nsHostResolver:3,pkix:5,nss_mod_log:4" NSS_STRICT_SHUTDOWN="x" NSS_OUTPUT_FILE=/tmp/tls.log NSPR_LOG_FILE=/tmp/firefox.log  SSLTRACE=127

The firefox log (which does not really have much in terms of ssl errors/warnings) is here: https://acksyn.org/files/tls-der-issue/firefox.log.moz_log

The cert pem file itself is here: https://acksyn.org/files/tls-der-issue/gitops.pem
The corresponding CA is here: https://acksyn.org/files/tls-der-issue/ca.pem

I sniffed the traffic for this specific failure here:
https://acksyn.org/files/tls-der-issue/filtered.pcap

Note that the cert is not only signed by a private ephemeral CA, its dns/cname entries are also not matching the FQDN:
$ openssl x509 -in gitops.pem -text |grep -i -e dns -e cname
DNS:openshift-gitops, DNS:openshift-gitops-grpc, DNS:openshift-gitops.openshift-gitops.svc.cluster.local

I also tried a couple of ssl checker websites https://www.sslshopper.com/ssl-checker.htm and https://www.digicert.com/help/ and barring the CA thing, they were okay with the cert. Interestingly SSLLabs keeps having internal server errors on my url.

To check the NSS parts I tried the following:
certutil -N -d /tmp/ssl/db
certutil -d /tmp/ssl/db -A -t "C,," -n ocp-ca -i ca.pem
certutil -d /tmp/ssl/db -A -t ",," -n gitops -i gitops.pem

I see that "certutil -d /tmp/ssl/db -L -n gitops" spits out the cert just fine. Maybe the problem here is that the Issuer is empty?:
$ certutil -d /tmp/ssl/db -L -n gitops |head -n10
Certificate:
  ¦ Data:
  ¦ ¦ ¦ Version: 3 (0x2)
  ¦ ¦ ¦ Serial Number:
  ¦ ¦ ¦ ¦ ¦ 6f:e2:33:d6:c0:54:e0:d4
  ¦ ¦ ¦ Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
  ¦ ¦ ¦ Issuer: "(empty)"
  ¦ ¦ ¦ Validity:
  ¦ ¦ ¦ ¦ ¦ Not Before: Fri Feb 25 09:07:04 2022
  ¦ ¦ ¦ ¦ ¦ Not After : Sat Feb 25 09:07:05 2023

Interestingly the CA verification (If I got the command right) seems to work:
$ certutil -d /tmp/ssl/db -V -n gitops -u Y
certutil: certificate is valid


The expectation is though that we get prompted if we're okay with this self-signed/internal cert, just like FF93 did and chrome still does.

Any tips for me to debug this further? What's the right incantation to get firefox to log all tls/pki bits to a file? Or should I move this to NSS component and try my luck there?