Bug 1204670

Epiphany connects without delay, after warning about the certificate.

This is not Cockpit specific. If I take the same certificate and put it on my local apache httpd 2.4.10 instance with mod_ssl ... I get identical behavior.

Created attachment 1005307 [details]
Example certificate

This is an example certificate that causes the problem. Here is how it is configured with httpd and mod_ssl:

LoadModule ssl_module modules/mod_ssl.so

Listen 443 https
<VirtualHost _default_:443>
DocumentRoot /var/www/html
SSLEngine on
SSLCertificateFile /etc/cockpit/ws-certs.d/~self-signed.cert
SSLCertificateKeyFile /etc/cockpit/ws-certs.d/~self-signed.cert
</VirtualHost>

After Firefox connects, clicking 'View Certificate' makes firefox hang solid ... it seems like it hangs forever. It doesn't respond to clicks, nor am I able to relaunch Firefox until I kill Firefox completely.

firefox-36.0-1.fc22.x86_64

Also seen on F21

Firefox falls over because of the "CN=localhost.localdomain" in the certificate subject. This generates a certificate that causes Firefox problems:

$ openssl req -x509 -days 36500 -newkey rsa:2048 -keyout $key_file  -keyform PEM -nodes -out $out_file -outform PEM -subj /CN=localhost.localdomain

Whereas this works fine:

$ openssl req -x509 -days 36500 -newkey rsa:2048 -keyout $key_file  -keyform PEM -nodes -out $out_file -outform PEM -subj /CN=localhost

Cockpit workaround for our self-signed certificate generation: 

https://github.com/cockpit-project/cockpit/pull/1984

Work around here for Cockpit: https://admin.fedoraproject.org/updates/cockpit-0.45-1.fc22

It's not just 'CN=localhost.localdomain' - I can reproduce the hang on viewing the cert with a self-signed cert with a different CN here.

Stef - do you happen to have another self-signed localhost.localdomain cert installed?  Where I saw this hang was when I had multiple self-signed serts with the same info (but different keys, as they were generated at different times.)

Yes, this browser has seen many certificates with CN=localhost.localdomain ... all self-signed certificates generated by various cockpit instances.

(In reply to Bill Nottingham from comment #15)
> Where I saw this hang was when I had multiple self-signed serts
> with the same info (but different keys, as they were generated at different
> times.)

Hmm, does that mean that changing the subject to /CN=localhost is not a permanent fix?

It might be that changing the subject helps initially, but as soon as a second certificate with the same subject shows up, Firefox will hang again.  True?

How would one create self-signed certs for machines without domain name?  No subject, or a random subject?

> Hmm, does that mean that changing the subject to /CN=localhost is not a permanent fix?

I guess so. Firefox is really the thing that needs to be fixed.

Firefox/NSS has a requirement that {issuer-name,serial-number} must be unique.

If you have added many overrides for certificates, and you a duplicate {issuer-name,serial-number} certificate, then firefox will report sec_error_reused_issuer_and_serial. However, firefox shouldn't hang.

You didn't report that error. Also, the serial number on the example you've provided appears to be random.

I guess you aren't affected by the duplicate issuer/serial issue.

Side note:

Although unrelated to this bug, your server certificate is really out of spec. Please, at least add a basic constraint extension which says "this isn't a CA".

> Although unrelated to this bug, your server certificate is really out of spec. Please, at least add a basic constraint extension which says "this isn't a CA".

I guess OpenSSL shouldn't generate certificates that are out of spec? The certificate generation command is derived from:

$ rpm -qf /etc/ssl/certs/make-dummy-cert 
openssl-1.0.1k-2.fc22.x86_64

You probably want to patch 'openssl req' or its make-dummp-cert to fix that.

My reading of the relevant RFC notes that this is optional, and is treated as cA = FALSE when not present.

https://tools.ietf.org/html/rfc5280#section-4.2.1.9

If your browser profile really have seen many certificates with the same subject name, and if you had added many permanent overrides: If you connect to the server and the cert uses the same subject name, then I wonder: Maybe firefox searches your cert database for a trusted certificate with that name?

That would explain why:
- the bug cannot be seen with a fresh profile
- I cannot see the bug with my old profile
- you can fix it by using a different cert for the new name
  (because you don't have many other certs with the same name)

If this theory were right, you could try to:
- open firefox certificate manager
- open the server tab
- delete all entries for localhost.localdomain
- restart firefox and try again

Does that fix it?

If it fixed it, it would mean that firefox didn't detect that it's a self signed cert right away and that searching for a potential issuer cert won't help.

(In reply to Stef Walter from comment #23)
> 
> I guess OpenSSL shouldn't generate certificates that are out of spec? 

It's out of spec, because you use it as a server cert.


> My reading of the relevant RFC notes that this is optional, and is treated
> as cA = FALSE when not present.

The basic constraint extension is present in your cert, and it says ca=true

Created attachment 1006266 [details]
Screenshot of all relevant self-signed certificates in 'Certificate Manager'

> The basic constraint extension is present in your cert, and it says ca=true

Aha. Good catch.

> Does that fix it?
> 
> If it fixed it, it would mean that firefox didn't detect that it's a self signed cert right away and that searching for a potential issuer cert won't help.

No I don't see it fixed. I removed all the entries in the screenshot above, and still see the same issue.

Ok, thanks for trying. And there weren't that many entries.

Right now I'm out of ideas. I wish I could reproduce.

I've tried to install cockpit on my f21 machine, but I can connect to localhost:9090 without problems.

> The basic constraint extension is present in your cert, and it says ca=true

Posted a pull request to fix this: https://github.com/cockpit-project/cockpit/pull/2018

> Right now I'm out of ideas. I wish I could reproduce.

Do you want my profile?

(In reply to Stef Walter from comment #29)
> > Right now I'm out of ideas. I wish I could reproduce.
> 
> Do you want my profile?

Maybe files cert8.db / cert9.db and cert_override.txt would be sufficient for the first step.

(In reply to Stef Walter from comment #18)
> > Hmm, does that mean that changing the subject to /CN=localhost is not a permanent fix?
> 
> I guess so. Firefox is really the thing that needs to be fixed.

Yes, sure, sorry.  What I meant is that /CN=localhost might not be a effective workaround because it fundamentally doesn't change anything.  Changing the subject just resets things enough to work for a while, until they break again when Firefox has gotten into the wedged state.

(In reply to Kai Engert (:kaie) from comment #28)

> I've tried to install cockpit on my f21 machine, but I can connect to
> localhost:9090 without problems.

Try deleting /etc/cockpit/ws-certs.d/~self-signed.cert followed by "systemctl restart cockpit".

If this bug is really caused by Firefox accumulating multiple certificates/exceptions, then you might be able to trigger it by regenerating the Cockpit cert.

Thanks, I can reproduce now.

I confirm the time to connect takes longer with each additional permanent override.

Firefox stores each cert of the permanent override. And it makes a mistake, which is probably responsible for the effect you're seeing.

It uses the same "nickname" for each new certificat, although the certificates are different.

I think Firefox should use a unique nickname each time it imports a new certificate.

The fact that there are different certs with identical nicknames might confuse NSS. Either NSS shouldn't get confused by that, or NSS should refuse to import them with the conflicting nickname.

I'll debug more and file an upstream bug.

For the time being, here is a workaround for you:

Please ensure you quit all Firefox windows before you do you the following:

cd into the Firefox profile directory, $HOME/.mozilla/firefox/*.default (or the directory of the profile you're using).

ls -l cert*

Do you see cert8.db?
====================
Then run
  certutil -d dbm:. -L
You'll see many localhost.localdomain entries.

Then run
  certutil -d dbm:. -D -n localhost.localdomain

multiple times, it will delete on cert with each execution.

You can run the -L command again to check how many you have left.

When you have deleted all of them, try firefox again, and it should be quick again.


Do you see cert9.db?
====================
In the above command, instead of dbm:. use sql:.

I found that Firefox/PSM code deliberately reuses the same nickname, because the certificate has the same subject name. It might not be the cause of the issue.

Using the NSS utility tstclnt to connect to the server, while using the NSS database with multiple entries (that hangs firefox) works fine and quick.

Using the NSS utility certutil to validate the certificate is also quick.

I suspect that the issue is with the mozilla::pkix certificate validation code. The fact that you're using a CA certificate with a CAID parameter might contribute to the confusion of the mozilla::pkix code. But it shouldn't be confused. It sounds like it's looping. Anyway, it shouldn't behave like that.

I will be interesting if you can still reproduce this issue after you've fixed the server cert to not be a CA any more.

7 permanent overrides, meaning 7 stored certs, was sufficient to trigger a long delay already.

(In reply to Kai Engert (:kaie) from comment #33)
> Thanks, I can reproduce now.
> 
> I confirm the time to connect takes longer with each additional permanent
> override.

Awesome, thanks a lot for figuring this out!

This means that this issue is far less severe than we thought it would be.

I indeed see a lot of looping in mozilla::pkix

I see recursive attempts to find issuer certs, and I'll prepare a test case for upstream to debug and get this fixed.

Once you have a build of cockpit that uses non-ca server cert, we could test if that solves the issue already. Could you try to add 7 overrides, each time with a different cert, and check if it's still slow?

Kai is mozilla::pkix trying to find issuers of a self-signed certificate? We should report that back to mozilla. I think that they should short circuit self-signed certs. Can we reproduce the problem with the NSS PKIX validator?

bob

Oh, I bet it is. I bet it's trying to build all possible chains. Each new cert doubles the number of possible chains, so by the time you get to 7 you need to check 2^6 chains.

The problem is 2 fold: 1) cockpit is using selfsigned certs. The pathological case is with selfsigned certs. 2) cockpit is marking their selfsigned certs as CA's (therefore you can make chains our of them). 3) Mozilla should short-circuilt self-signed tests in chain building (don't build chains past a self-signed cert).

Cockpit should be able to make this issue go away with the any of the following: Issue new certificates from a common root (better security characteristics. Self-signed certs a fine hack to get things going, but if you are trying to actually deploy a product, you need to be able to get roots from a common infrastructure. I think the IPA guys can help. They have spent a fair amount of time figuring out how to distribute new and manage root certificates for their infrastructure.

In the meantime, don't issue roots with CA=True in them. If the roots had CA=False, mozilla:pkix wouldn't try to build chains with them.

You can also fix this by varying your the DN for your cert. You don't need to make a unique CN, changing any component of the DN would work. For production, I would actually recommend not putting the hostname in the DN, but put it in the subject Alt extension. The latter is the preferred way anyway, but that's not what is causing the issue.

bob

I've reported the issue upstream and also provided 8 test services to make it easy for people to reproduce the issue.
https://bugzilla.mozilla.org/show_bug.cgi?id=1147544

> Cockpit should be able to make this issue go away with the any of
> the following: Issue new certificates from a common root (better
> security characteristics. Self-signed certs a fine hack to get
> things going, but if you are trying to actually deploy a product,
> you need to be able to get roots from a common infrastructure. I
> think the IPA guys can help. They have spent a fair amount of
> time figuring out how to distribute new and manage root
> certificates for their infrastructure.

Yes, we have that on our roadmap: Make it easy to use domain (as in FreeIPA) issued certificates with the various hosts.

However because Cockpit heavily targets the "out of the box" use case, self-signed certificates will be a staple of Cockpit usage for the forseeable future.

This is analogous to how most people use SSH keys verified by fingerprint, rather than SSH certificates signed by a common authority.

> In the meantime, don't issue roots with CA=True in them. If the
> roots had CA=False, mozilla:pkix wouldn't try to build chains with them.

Done. Thanks for confirming this.

(In reply to Kai Engert (:kaie) from comment #40)
> I've reported the issue upstream and also provided 8 test services to make
> it easy for people to reproduce the issue.
> https://bugzilla.mozilla.org/show_bug.cgi?id=1147544

Thanks Kai!

> However because Cockpit heavily targets the "out of the box" use case,
> self-signed certificates will be a staple of Cockpit usage for the forseeable
> future.

So any instructions that include 'override this security warning' isn't 'out of the box ready'. I'm sure we can come up with something that will make this unnecessary, and still have reasonable 'out of the box' access.

> This is analogous to how most people use SSH keys verified by fingerprint,
> rather than SSH certificates signed by a common authority.

As soon as you make the browser your engine, you need to respect the infrastructure. The browser security overrides are likely to get more and more difficult, we should not depend on them for any shipping product. It's considered unfriendly and poor security within the browser community. Doing so make us look like we don't know what we are doing.

bob

(In reply to Bob Relyea from comment #42)
> > This is analogous to how most people use SSH keys verified by fingerprint,
> > rather than SSH certificates signed by a common authority.
> 
> As soon as you make the browser your engine, you need to respect the
> infrastructure. The browser security overrides are likely to get more and
> more difficult, we should not depend on them for any shipping product. It's
> considered unfriendly and poor security within the browser community. Doing
> so make us look like we don't know what we are doing.

Cockpit is not a product. The product is "Fedora Server", the product is RHEL, the product is  ... Cockpit is the admin interface. 

So yes, if we're at the point where Fedora or RHEL can trivially acquire have a signed Web server certificate out of the box upon installation ... then that would be great. However, it needs to meet Cockpit's ideals and criteria if it's going to replace self-signed certificates. In particular the following:

 * Cockpit requires no configuration or infrastructure

http://stef.thewalter.net/ideals-of-cockpit.html

Kai, is there anything needed on Firefox/PSM side or is that just NSS issue? Or do we need any local (Fedora/RHEL) fix for that or it can be moved upstream?

This message is a reminder that Fedora 21 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 21. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '21'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 21 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Fedora 21 changed to end-of-life (EOL) status on 2015-12-01. Fedora 21 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Bill, why is this one reopened and what fix do you expect here?

... it's reopened because it's still an open issue. See upstream bug at https://bugzilla.mozilla.org/show_bug.cgi?id=1056341

This message is a reminder that Fedora 23 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 23. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '23'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 23 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

This message is a reminder that Fedora 25 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 25. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '25'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 25 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

This message is a reminder that Fedora 27 is nearing its end of life.
On 2018-Nov-30  Fedora will stop maintaining and issuing updates for
Fedora 27. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora  'version' of '27'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 27 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Fedora 27 changed to end-of-life (EOL) status on 2018-11-30. Fedora 27 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.