Bug 1491053

Summary: Firefox reports insecure TLS configuration when visiting FreeIPA web UI after standard server deployment
Product: [Fedora] Fedora Reporter: Adam Williamson <awilliam>
Component: freeipaAssignee: IPA Maintainers <ipa-maint>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 27CC: abokovoy, cheimes, dueno, ipa-maint, jcholast, jhrozek, jswensso, kengert, kparal, mkosek, pvoborni, rcritten, robatino, slaznick, ssorce, tkrizek
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: AcceptedFreezeException AcceptedBlocker
Fixed In Version: freeipa-4.6.1-3.fc27 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-18 15:23:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1502882, 1502890    

Description Adam Williamson 2017-09-12 22:43:10 UTC
In current Fedora 27 and Rawhide, after standard FreeIPA deployment via rolekit, visiting the FreeIPA web UI with firefox results in a 'Your connection is not secure' screen:

https://openqa.stg.fedoraproject.org/tests/159242#step/freeipa_webui/4

The funny thing is, this didn't happen 6 days ago when running tests on the FreeIPA update we pushed stable:

https://openqa.stg.fedoraproject.org/tests/155282

I haven't figured out what changed between the two runs yet, but so far it doesn't seem to be obvious - not FreeIPA, not Firefox, and not NSS. I'll look into it some more soon, if no-one else gets to it first.

Proposing as a blocker bug per Alpha criterion "The core functional requirements for all Featured Server Roles must be met, but it is acceptable if moderate workarounds are necessary to achieve this" - 'domain controller' is a release-blocking role, and one of its core requirements is "The FreeIPA configuration web UI must be available and allow at least basic configuration of user accounts and permissions" (per https://fedoraproject.org/wiki/Domain_controller_role_requirements ).

Comment 1 Petr Vobornik 2017-09-13 13:52:00 UTC
Would be good to get info from next page when click on "advanced" button. 

Atm it can be e.g. not trusting IPA CA. Which might indicate e.g. not adding IPA CA to system store.

Or it can be completely different error.

Comment 2 Alexander Bokovoy 2017-09-13 14:03:05 UTC
From ipa-client-install.log:

2017-09-12T20:03:35Z DEBUG Starting external process
2017-09-12T20:03:35Z DEBUG args=/usr/bin/certutil -d /etc/ipa/nssdb -N -f /etc/ipa/nssdb/pwdfile.txt -f /etc/ipa/nssdb/pwdfile.txt
2017-09-12T20:03:35Z DEBUG Process finished, return code=0
2017-09-12T20:03:35Z DEBUG stdout=
2017-09-12T20:03:35Z DEBUG stderr=
2017-09-12T20:03:35Z DEBUG Adding CA certificates to the IPA NSS database.
2017-09-12T20:03:35Z DEBUG Starting external process
2017-09-12T20:03:35Z DEBUG args=/usr/bin/certutil -d /etc/ipa/nssdb -A -n DOMAIN.LOCAL IPA CA -t CT,C,C -a -f /etc/ipa/nssdb/pwdfile.txt
2017-09-12T20:03:35Z DEBUG Process finished, return code=0
2017-09-12T20:03:35Z DEBUG stdout=
2017-09-12T20:03:35Z DEBUG stderr=
2017-09-12T20:03:35Z DEBUG Starting external process
2017-09-12T20:03:35Z DEBUG args=/usr/bin/update-ca-trust
2017-09-12T20:03:36Z DEBUG Process finished, return code=0
2017-09-12T20:03:36Z DEBUG stdout=
2017-09-12T20:03:36Z DEBUG stderr=
2017-09-12T20:03:36Z INFO Systemwide CA database updated.

At least, it claims CA cert was correctly updated in the system-wide CA store.

Comment 3 Adam Williamson 2017-09-13 16:26:40 UTC
petr: yeah, I know that's an obvious next step, I just didn't have time for it yesterday and figured I'd report the bug in case anyone wanted to look into it during my night cycle, or already knew what the issue was.

Comment 4 Adam Williamson 2017-09-14 20:23:49 UTC
So the error is:

https://openqa.stg.fedoraproject.org/tests/161126#step/freeipa_webui/7

ipa001.domain.local uses an invalid security certificate.

"The certificate is not trusted because the issuer certificate is unknown..."

Comment 5 Adam Williamson 2017-09-15 02:05:35 UTC
Discussed at 2017-09-14 Beta Go/No-Go meeting, acting as a blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-meeting-2/2017-09-14/f27-beta-go-no-go-meeting.2017-09-14-17.00.html . We agreed to delay the blocker decision on this while we figure out how hard it is to work around (if you can just grant an exception in Firefox, that's probably okay).

Comment 6 Adam Williamson 2017-09-15 15:39:49 UTC
So I fiddled with the test a bit to give us more data, which you can find here:

https://openqa.stg.fedoraproject.org/tests/161565#downloads

freeipa_webui-ipa.p11-kit is the contents of the 'ipa.p11-kit' file that freeipa drops in /etc/pki/ca-trust/source/ .

freeipa_webui-x509.log is the result of `openssl x509 -in /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt -noout -text > /tmp/x509.log` (which basically shows the last certificate in the bundle; in this case we can see that it appears to be the FreeIPA CA cert).

s_client.log is the result of `openssl s_client -connect ipa001.domain.local:443 -showcerts > /tmp/s_client.log` - which, note, fails. The claimed reason for failure is '(self signed certificate in certificate chain)', but openssl error reasons can be more or less flat out lies at times.

I'm *somewhat* interested in the 'trusted' stuff - it looks a bit like the certificate is marked as a 'trusted' one, but no specific trusted uses are listed. But I'm not 100% up on this '[p11-kit-object-v1]' format, so I may be missing something there.

Comment 7 Alexander Bokovoy 2017-09-15 15:51:19 UTC
Thanks, Adam. I think this is https://pagure.io/freeipa/issue/7119

Comment 8 Stephen Gallagher 2017-09-15 15:53:04 UTC
I want to add to this that the CA cert *does* seem to be added at least partially to Firefox, as it shows up in the list of Authorities: https://sgallagh.fedorapeople.org/ipa-cert.png

(The domain is BROKEN.IPA)

So I'm guessing it's as Adam says: it's being imported but not getting the appropriate trust set.

Comment 9 Adam Williamson 2017-09-15 16:03:30 UTC
AB: having read through it, I concur, that looks like exactly what we're seeing. In an 'of course!' moment I woke up and realized I should have the test check /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem as well as ca-bundle-trusted.crt, since the former will *only* include certificates p11-kit decides are actually trusted; I've modified the test to do that as well now, I expect it'll show us that the cert is not in that file, which would be strong confirmation these are the same bug.

Have we figured out why this is happening yet? I guess it's most likely to do with b5732efda6d27f588adbc3f41259bc4511716f43 .

Comment 10 Alexander Bokovoy 2017-09-15 16:39:59 UTC
The problem I see is that there are no EKUs in the actual CA certificate because caCACert.cfg profile in Dogtag does not include them at all -- check yourself in /usr/share/pki/ca/profiles/ca/caCACert.cfg.

Looking at the rewrite you pointed to (b5732efda..), the extraction of EKU bits was replaced from

-def get_ext_key_usage(certificate, datatype=PEM):
-    cert = load_certificate(certificate, datatype)
-    try:
-        eku = cert.extensions.get_extension_for_oid(
-            cryptography.x509.oid.ExtensionOID.EXTENDED_KEY_USAGE).value
-    except cryptography.x509.ExtensionNotFound:
-        return None
-
-    return set(oid.dotted_string for oid in eku)

to

+    @property
+    def extended_key_usage(self):
+        try:
+            ext_key_usage = self._cert.extensions.get_extension_for_oid(
+                crypto_x509.oid.ExtensionOID.EXTENDED_KEY_USAGE).value
+        except crypto_x509.ExtensionNotFound:
+            return None
+
+        return set(oid.dotted_string for oid in ext_key_usage)
+
+    @property
+    def extended_key_usage_bytes(self):
+        ekurfc = rfc2459.ExtKeyUsageSyntax()
+        eku = self.extended_key_usage or {EKU_PLACEHOLDER}
+        for i, oid in enumerate(eku):
+            ekurfc[i] = univ.ObjectIdentifier(oid)
+        ekurfc = encoder.encode(ekurfc)
+        return self.__encode_extension('2.5.29.37', EKU_ANY not in eku, ekurfc)
+

As you can see, at the core it is still the same call to python-cryptography's x509 cert.extensions.get_extension_for_oid(). If EKUs were missing before b5732efda.. they would not be there with old code too.

Comment 11 Adam Williamson 2017-09-15 19:17:42 UTC
One possibility occurs to me: what if, before, we were not writing a correct EKU section, but *no EKU section at all*?

I guess I should run my modified test on F26 and see what things look like there...

Comment 12 Adam Williamson 2017-09-15 19:23:30 UTC
Oh, turns out it already *has* run, so we can compare the results...

https://openqa.stg.fedoraproject.org/tests/161672/file/freeipa_webui-ipa.p11-kit is from an F26 test, https://openqa.stg.fedoraproject.org/tests/161694/file/freeipa_webui-ipa.p11-kit from F27. So it does look like F26 gets the correct EKU block, rather than just having no EKU block at all.

Comment 13 Alexander Bokovoy 2017-09-15 20:20:50 UTC
Very interesting, indeed. Thanks.

Christian, any comments? You know python-cryptography best.

Comment 14 Adam Williamson 2017-09-16 03:16:14 UTC
Alex: the other thought that occurs to me is that so far we only looked at *one end* of the process - when (if I'm understanding it correctly) the client retrieves the server cert via LDAP and puts it into the client's system-wide store. We haven't yet (or at least I haven't, and you haven't said that you have) looked at whether the server cert is actually being *created* correctly, i.e. whether it has any EKUs in it in the first place. If it doesn't, there could be no bugs at all in the code for reading the EKUs from the retrieved cert, and we'd still get this result...

Comment 15 Alexander Bokovoy 2017-09-16 09:10:17 UTC
Adam, as I wrote in the comment 10, caCACert.cfg has no EKU definitions, thus I'm at loss why it worked before. I'd like Christian's view on the issue as he is one of python-cryptography folks.

Comment 16 Adam Williamson 2017-09-16 14:03:16 UTC
ab: note ipalib/x509.py defines several EKU constants (e.g. x509.EKU_SERVER_AUTH) and there are several points in the code at which these are used, including ipalib/install/certstore.py in a function called 'make_compat_ca_certs'...

Comment 17 Christian Heimes 2017-09-16 14:34:50 UTC
Correct, this upstream issue 7119. I posted a detailed analysis of the issue three weeks ago, see comment https://pagure.io/freeipa/issue/7119#comment-460202

In comment 10 you said that Dogtag does not include a EKU in its CA profile. This is completely fine and actually a good idea for a root CA. Most root CAs do not have a EKU extension, just a KU extension for cert and CRL signing. For example the root CA "VeriSign Class 3 Public Primary Certification Authority - G5" for www.redhat.com does not have an EKU extension. In a matter of fact some TLS libraries even ignore EKU extensions on trust anchors, see #1301683 for my OpenSSL 1.0.2 bug report.

It's not the lack of EKU but the presence of EKU with OID 1.3.6.1.4.1.3319.6.10.16  that causes this bug. FreeIPA adds OID 1.3.6.1.4.1.3319.6.10.16. According to FreeIPA documentation "A value of 1.3.6.1.4.1.3319.6.10.16 means no extended key usages are allowed.". The documentation fails to mention that this OID taints the chain and affects all sub certificates.

As a consequence, commands like "/usr/bin/p11-kit extract --format=pem-bundle --filter=ca-anchors --overwrite --comment --purpose server-auth" (used by update-ca-trust) no longer see the certificate as trusted for server auth.

Solution: Either FreeIPA must not include an EKU in CA certs or include all supported EKUs.

Comment 18 Christian Heimes 2017-09-16 14:39:48 UTC
PS: No EKU is probably a better idea. A strict EKU may block future use cases (e.g. code signing or new, not yet invited EKUs). The benefits of EKU on root CAs is limited, too. As I mentioned, some libraries ignore EKU on trust anchors when they verify a chain.

Comment 19 Alexander Bokovoy 2017-09-18 07:18:11 UTC
So the code to deal with OID 1.3.6.1.4.1.3319.6.10.16 (called EKU_PLACEHOLDER in FreeIPA code) was added with de695e688e390deef5510dca0daef133f50f490f and is part of FreeIPA releases since 4.1.0.

Did the handling of this OID change in python-cryptography in Fedora 27? Looking at p11-kit, it does allow EKU_PLACEHOLDER to be present due to RFC5280 requiring to have at least one EKU on the cert if EKU extension is present.

On my system which was installed on July 19th using Fedora 26 bits, I have all extensions available in the CA certificate:

# ldapsearch -Y GSSAPI 'cn=L.IPA.COOL IPA CA'l.ipa.cool
dn: cn=L.IPA.COOL IPA CA,cn=certificates,cn=ipa,cn=etc,dc=l,dc=ipa,dc=cool
ipaKeyExtUsage: 1.3.6.1.5.5.7.3.1
ipaKeyExtUsage: 1.3.6.1.5.5.7.3.2
ipaKeyExtUsage: 1.3.6.1.5.5.7.3.3
ipaKeyExtUsage: 1.3.6.1.5.2.3.4
ipaKeyExtUsage: 1.3.6.1.5.2.3.5
ipaKeyExtUsage: 1.3.6.1.5.5.7.3.4
cn: L.IPA.COOL IPA CA
objectClass: ipaCertificate
objectClass: pkiCA
objectClass: ipaKeyPolicy
objectClass: top
ipaCertSubject: CN=Certificate Authority,O=L.IPA.COOL
ipaKeyTrust: trusted
ipaCertIssuerSerial: CN=Certificate Authority,O=L.IPA.COOL;1
ipaConfigString: ipaCa

And the corresponding ipa.p11-kit file has a correct value for object-id: 2.5.29.37. This install was done with FreeIPA git master at the time, so it definitely included all the code that should have placed EKU_PLACEHOLDER in case no other EKU was present.

The same correct result comes with up to date Fedora 26 too, as Adam has demonstrated in the comment 12. It is interesting that both F26 and F27 versions of python-cryptography are the same, essentially.

At this point, I'm trying to understand what did change and where. Given the code did not change in FreeIPA with regards to EKU_PLACEHOLDER processing between F26 and F27, I'm currently at a loss why this is happening.

Comment 20 Christian Heimes 2017-09-18 09:37:47 UTC
After further investigation I'm sure that the root CA certs are completely fine. It has to be a bug in FreeIPA's code that generates the p11-kit file. It contains EKU_PLACEHOLDER instead of {x509.EKU_SERVER_AUTH, x509.EKU_CLIENT_AUTH, x509.EKU_EMAIL_PROTECTION, x509.EKU_CODE_SIGNING}.

The bug in the p11-kit file only affects programs that use IPA's p11-kit file. The presence of an EKU OID in the p11-kit file cause programs like update-ca-trust to ignore the absence of EKU from the cert and use the EKU override instead. It looks like NSS and Firefox also use the p11-kit extended information.

Most FreeIPA commands do not rely on the system trust store or the p11-kit file. They use /etc/ipa/ca.crt as sole provider for trust anchors. I guess that is the reasons FreeIPA's tests have not picked up the problem yet. The problem was noticed and reported as https://pagure.io/freeipa/issue/7119 when Michal tested kinit over KDC proxy. kinit did not trust the certificate because the root CA was not present in /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem.

I'd concentrate on the code that creates the p11-kit file.

Comment 21 Alexander Bokovoy 2017-09-18 11:48:22 UTC
I tried to remove the whole extension section from ipa.p11-kit and it seems that this approach did help with update-ca-trust. Standa is working on a proper pull request right now.

Comment 22 Standa Laznicka 2017-09-18 14:40:37 UTC
Upstream ticket:
https://pagure.io/freeipa/issue/7159

Comment 23 Alexander Bokovoy 2017-09-18 16:03:17 UTC
Here is a pull request that attempts to fix this issue: https://github.com/freeipa/freeipa/pull/1090

Comment 24 Kamil Páral 2017-09-18 16:53:13 UTC
Discussed at blocker review meeting [1]:

RejectedBlocker (beta) AcceptedFreezeException (beta) AcceptedBlocker (final) - There should be an easy workaround - just allow the invalid cert in Firefox. For final we'll block because this bug violates the criterion: "The FreeIPA configuration web UI must be available and allow at least basic configuration of user accounts and permissions "

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2017-09-18

Comment 25 Standa Laznicka 2017-09-19 07:58:46 UTC
Fixed upstream
master:
https://pagure.io/freeipa/c/e537686bcc8248bc0216ce634ae7707fa65e70ba

Comment 26 Tomas Krizek 2017-09-19 15:53:37 UTC
Fixed upstream
ipa-4-6:
https://pagure.io/freeipa/c/7da5187987799210b596fe638e98ed26a8563b11

Comment 27 Fedora Update System 2017-09-26 15:55:46 UTC
freeipa-4.6.1-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-7806cebb23

Comment 28 Fedora Update System 2017-09-27 15:53:06 UTC
freeipa-4.6.1-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-7806cebb23

Comment 29 Fedora Update System 2017-10-04 14:21:14 UTC
freeipa-4.6.1-1.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Comment 30 Adam Williamson 2017-10-13 00:48:26 UTC
Only, this still seems to be happening as of Fedora-27-20171012.n.0:

https://openqa.fedoraproject.org/tests/156778#step/freeipa_webui/4

Comment 31 Adam Williamson 2017-10-13 00:56:29 UTC
Note, the change that landed upstream *does* seem to have taken effect; the EKU section no longer appears in the ipa.p11-kit file. But Firefox still reports an insecure certificate.

The openssl s_client test also reports the cert as OK, from the logs:

https://openqa.stg.fedoraproject.org/tests/177575/file/freeipa_webui-s_client.log

so...really not sure why Firefox still doesn't like it. But it definitely doesn't; the test is still failing every time in all openQA runs I've checked. Firefox uses NSS, so there must be some difference there...

Comment 32 Adam Williamson 2017-10-13 00:58:54 UTC
Unfortunately I threw away the openQA test code that makes it click the Advanced button, so I don't know for sure what reason Firefox is giving yet. I'll recreate all that diagnostic code tomorrow so we can hopefully figure it out.

Comment 33 Adam Williamson 2017-10-13 01:01:34 UTC
To compare the diagnostic logs from before and after the change, here's a failure from before the change:

https://openqa.stg.fedoraproject.org/tests/161565

And here's a failure from after the change but before I removed the debugging code (it still doesn't click on Advanced in Firefox, though :/):

https://openqa.stg.fedoraproject.org/tests/177575

you can find the logs as always on the Logs & Assets tab.

Comment 34 Adam Williamson 2017-10-13 23:00:34 UTC
So I found a slightly easier way to play with this than running the openQA test over and over. You can just set up a clean VM, dump files in the relevant format - which apparently is called the p11-kit 'persistence format', documented at https://github.com/p11-glue/p11-kit/blob/master/doc/internal/persist-format.txt - in /etc/pki/ca-trust/source , run update-ca-trust , reload Firefox, and see what it says about them.

I've been messing around with the file as FreeIPA currently produces it - that is, https://openqa.stg.fedoraproject.org/tests/177575/file/freeipa_webui-ipa.p11-kit - and a file I produced by using the 'trust dump' command, which dumps the contents of an existing trust store into the same 'persistence' format. That file looks like this:

# This file was created by IPA. Do not edit.

[p11-kit-object-v1]
class: certificate
certificate-type: x-509
certificate-category: authority
label: "DOMAIN.LOCAL%20IPA%20CA"
subject: "071%150%13%06%03U%04%0A%0C%0CDOMAIN.LOCAL1%1E0%1C%06%03U%04%03%0C%15Certificate%20Authority"
issuer: "071%150%13%06%03U%04%0A%0C%0CDOMAIN.LOCAL1%1E0%1C%06%03U%04%03%0C%15Certificate%20Authority"
serial-number: "1"
x-public-key-info: "0%82%01%220%0D%06%09%2A%86H%86%F7%0D%01%01%01%05%00%03%82%01%0F%000%82%01%0A%02%82%01%01%00%AD%A1CQF%1F%9C%98%AC%EDt%EC%D0%3E%2C7%92q%D8y%F7KI%E2%DF9%9C%C8%C0%1B%91T%C0%C4v%C2%C4%AB%02n/%CA%C1K%91n%18yU_%C9%84%CFi%2A%E7t%03%F6%DE%FA%23%F4%95%B8%0A%BBh%90%B9%C7%86%8FWBF%16%5DS%84z%15%B4%8D%F8%C3J%3A%F5%9E%A1W/%12M%94%97lH%9F%E4%29%C8%C0%FDI%8AOK%BF%0F%2B%9A%BE%EE%AB%9Fco%E7%23%1A%EF%E4%0E%EB%ED%FB%09%85%F3%7C%7F%A4%25%98%DA%B3%3Br0%92i%91aU9%F5%11%7C%C8%85m%28%13%02%98%19%ACjjY%8C%3BE%B8%02%E3Kg%BE%E4%22%81%C2%D6%C7%CD%C1%00H%7E5%C2%0F%E4%3C%D5%9DLYa%BF%EB%3B%824%7B%0C%B2%80%97%91O%B3%1C%E9%D0%1Ca%3C%DA9%EDZ%90%92%8C%86%D5%09Q%12%E0%DA%95%EBpf%89%83PC%0E%00%C0Sk%C1%9D%D2%FD%BC%A2%17%EBK%B7%AA%BA%E1%C8%5DTL%3B%02%03%01%00%01"
trusted: true
private: false
modifiable: false
x-distrusted: false
-----BEGIN CERTIFICATE-----
MIIDjjCCAnagAwIBAgIBATANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQKDAxET01B
SU4uTE9DQUwxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xNzEw
MTAyMzI5MjJaFw0zNzEwMTAyMzI5MjJaMDcxFTATBgNVBAoMDERPTUFJTi5MT0NB
TDEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MIIBIjANBgkqhkiG9w0B
AQEFAAOCAQ8AMIIBCgKCAQEAraFDUUYfnJis7XTs0D4sN5Jx2Hn3S0ni3zmcyMAb
kVTAxHbCxKsCbi/KwUuRbhh5VV/JhM9pKud0A/be+iP0lbgKu2iQuceGj1dCRhZd
U4R6FbSN+MNKOvWeoVcvEk2Ul2xIn+QpyMD9SYpPS78PK5q+7qufY2/nIxrv5A7r
7fsJhfN8f6QlmNqzO3IwkmmRYVU59RF8yIVtKBMCmBmsampZjDtFuALjS2e+5CKB
wtbHzcEASH41wg/kPNWdTFlhv+s7gjR7DLKAl5FPsxzp0BxhPNo57VqQkoyG1QlR
EuDaletwZomDUEMOAMBTa8Gd0v28ohfrS7equuHIXVRMOwIDAQABo4GkMIGhMB8G
A1UdIwQYMBaAFFQ9o0rzeUFe4kxt5CzBbu341e51MA8GA1UdEwEB/wQFMAMBAf8w
DgYDVR0PAQH/BAQDAgHGMB0GA1UdDgQWBBRUPaNK83lBXuJMbeQswW7t+NXudTA+
BggrBgEFBQcBAQQyMDAwLgYIKwYBBQUHMAGGImh0dHA6Ly9pcGEtY2EuZG9tYWlu
LmxvY2FsL2NhL29jc3AwDQYJKoZIhvcNAQELBQADggEBAJyeKpoLvAmgjYDlzYaI
ItMVhnE547zuphLTUDiTyXySpj9ag93pR6Ncl+pSjvykRBlyeROhkPXPTaFfsBDU
PbKoGi5kNvMKnHP3IcDV4adkgTBk2Wv1uqut5dw0K71PVMqR03Q1o2ZS143f5x6K
RxYfrs9AFZQiboRVNSsl38sl3bfQ9JBPmNeBGBz69XdiVg6iZ5+4+miLcKlxiPxV
idY6cqujg9GNX3gjphFhK3KBuEcbPExf418pCLk1t9myHZVl1FB84SyFXth7/2hR
8N7LZz/8qG9ViYYgl32g8voP5xoRSkcS7jMF8KscUHPRNmUTS6OD9zJIasilvUoU
rNw=
-----END CERTIFICATE-----

as you can probably guess, it's the CA cert for my own FreeIPA domain (HAPPYASSASSIN.NET). But the various fields are rather different to the ones in the FreeIPA-produced file; some different ones are present in each file.

My test is to put both .p11-kit files into /etc/pki/ca-trust/source and run 'update-ca-trust', then run Firefox. Go to Edit / Preferences / Privacy & Security , scroll down to Certificates, and click View Certificates. Now scroll down to DOMAIN.LOCAL and see there's a certificate named 'Certificate Authority'. Select it and click 'View...', and Firefox shows the cert details, but with a bolded text that says "Could not verify this certificate because the issuer is unknown".

Close that cert, and scroll on down to HAPPYASSASSIN.NET. Again, select the certificate named 'Certificate Authority', and click 'View'''. This time, you see the cert details and bolded text "This certificate has been verified for the following uses: SSL Certificate Authority".

This is ultimately the difference, here - if we were getting the format right, the DOMAIN.LOCAL cert should show up the same. But it doesn't. I haven't yet figured out what specific difference between the two p11-kit files causes one cert to wind up being trusted by Firefox and the other not, though. Let's pull in Daiki and Kai and see if they can help...

Comment 35 Adam Williamson 2017-10-13 23:02:09 UTC
Daiki, Kai, the short version of the latest problem we're dealing with here is that FreeIPA tries to add its own CA cert to the system trust store. It does this by creating a file in p11-kit persistence format, putting it in /etc/pki/ca-trust/source , and running 'update-ca-trust' (more or less). However, currently in Fedora 27, Firefox winds up not trusting the cert for any purpose (though OpenSSL seems happy with it, at least after the most recent changes). You can find the file generated by FreeIPA at https://openqa.stg.fedoraproject.org/tests/177575/file/freeipa_webui-ipa.p11-kit , and test just importing it to the system trust store with update-ca-trust then examine what Firefox thinks of it, as I detailed in the previous comment - Firefox doesn't like it. But that's as far as I've got.

Comment 36 Daiki Ueno 2017-10-16 08:46:47 UTC
(In reply to Adam Williamson from comment #34)

> [p11-kit-object-v1]
> class: certificate
> certificate-type: x-509
> certificate-category: authority
> label: "DOMAIN.LOCAL%20IPA%20CA"
> subject:
> "071%150%13%06%03U%04%0A%0C%0CDOMAIN.
> LOCAL1%1E0%1C%06%03U%04%03%0C%15Certificate%20Authority"
> issuer:
> "071%150%13%06%03U%04%0A%0C%0CDOMAIN.
> LOCAL1%1E0%1C%06%03U%04%03%0C%15Certificate%20Authority"
> serial-number: "1"

I think this line is the culprit; all values in the p11-kit persistent format must be encoded in DER, but the serial number here is not encoded and thus does not match the one decoded from the certificate.  Try changing this to:

serial-number: "%02%01%01"

Comment 37 Alexander Bokovoy 2017-10-16 09:44:09 UTC
Great discovery, Daiki! Thanks!

I think this is part of the commit b5732efda6 which refactored x509 certificates internally as Python objects in FreeIPA and in particular had these chunks in ipaplatform/redhat/tasks.py:

@@ -266,10 +265,10 @@ class RedHatTaskNamespace(BaseTaskNamespace):
         has_eku = set()
         for cert, nickname, trusted, ext_key_usage in ca_certs:
             try:
-                subject = x509.get_der_subject(cert, x509.DER)
-                issuer = x509.get_der_issuer(cert, x509.DER)
-                serial_number = x509.get_der_serial_number(cert, x509.DER)
-                public_key_info = x509.get_der_public_key_info(cert, x509.DER)
+                subject = cert.subject_bytes
+                issuer = cert.issuer_bytes
+                serial_number = cert.serial_number
+                public_key_info = cert.public_key_info_bytes
             except (PyAsn1Error, ValueError, CertificateError) as e:
                 logger.warning(
                     "Failed to decode certificate \"%s\": %s", nickname, e)
@@ -278,12 +277,9 @@ class RedHatTaskNamespace(BaseTaskNamespace):
             label = urllib.parse.quote(nickname)
             subject = urllib.parse.quote(subject)
             issuer = urllib.parse.quote(issuer)
-            serial_number = urllib.parse.quote(serial_number)
+            serial_number = urllib.parse.quote(str(serial_number))
             public_key_info = urllib.parse.quote(public_key_info)
 
-            cert = base64.b64encode(cert)
-            cert = x509.make_pem(cert)
-
             obj = ("[p11-kit-object-v1]\n"
                    "class: certificate\n"
                    "certificate-type: x-509\n"


Previously serial number was taken from the cert by decoding it every time:

def _get_der_field(cert, datatype, dbdir, field):
    cert = normalize_certificate(cert)
    cert = decoder.decode(cert, rfc2459.Certificate())[0]
    field = cert['tbsCertificate'][field]
    field = encoder.encode(field)
    return field

def get_der_serial_number(cert, datatype=PEM, dbdir=None):
    return _get_der_field(cert, datatype, dbdir, 'serialNumber')


now the code relies on Python-Cryptography's Certificate object which returns a serial number for the certificate as an integer.

I guess a fix here would be to explicitly encode with DER before writing p11-kit file.

Comment 38 Standa Laznicka 2017-10-16 11:27:09 UTC
Upstream ticket:
https://pagure.io/freeipa/issue/7210

Comment 39 Standa Laznicka 2017-10-16 11:32:29 UTC
I caused this regression, sorry. Fix at https://patch-diff.githubusercontent.com/raw/freeipa/freeipa/pull/1156.patch

Comment 40 Adam Williamson 2017-10-16 18:44:18 UTC
Can we get a package build / update with the fix ASAP? Thanks.

Comment 41 Alexander Bokovoy 2017-10-16 18:55:07 UTC
I'm on it.

Comment 42 Alexander Bokovoy 2017-10-16 19:01:19 UTC
https://koji.fedoraproject.org/koji/taskinfo?taskID=22491295 is building.

Comment 43 Fedora Update System 2017-10-16 20:50:23 UTC
freeipa-4.6.1-2.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-6afbf6070a

Comment 44 Adam Williamson 2017-10-16 22:19:06 UTC
Fix looks good, according to https://openqa.stg.fedoraproject.org/tests/overview?groupid=2&version=27&build=Update-FEDORA-2017-6afbf6070a&distri=fedora - note I had to hack up the test a bit because it seems like Cockpit installs the older freeipa-client from the official repos , not the updated one from the update repo, if I leave Cockpit to install freeipa-client. I hacked the test to install freeipa-client with dnf before starting Cockpit, which causes the one from the update repo to get installed. The test failed in production openQA because of this.

Comment 45 Adam Williamson 2017-10-17 00:12:13 UTC
Moving to Server-specific FE tracker.

Comment 46 Adam Williamson 2017-10-17 00:34:00 UTC
Moving to Server-specific blocker tracker.

Comment 47 Fedora Update System 2017-10-17 02:25:47 UTC
freeipa-4.6.1-2.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-6afbf6070a

Comment 48 Fedora Update System 2017-10-17 11:05:58 UTC
freeipa-4.6.1-3.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-fd9748d8f1

Comment 49 Tomas Krizek 2017-10-17 14:48:15 UTC
Fixed upstream
master:
https://pagure.io/freeipa/c/9b8b7afeb406f042c8c6d46f84cbb04126ac5204

Comment 50 Fedora Update System 2017-10-17 18:51:55 UTC
freeipa-4.6.1-3.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-fd9748d8f1

Comment 51 Tomas Krizek 2017-10-18 10:19:22 UTC
Fixed upstream
ipa-4-6:
https://pagure.io/freeipa/c/07630799b65721f2d3543d33a8ede4cf9064be71

Comment 52 Fedora Update System 2017-10-18 15:23:00 UTC
freeipa-4.6.1-3.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.