Bug 2118284
| Summary: | mtls CRL not working when using an intermediate CA | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Maria Alonso <malonso> |
| Component: | Networking | Assignee: | Ryan Fredette <rfredette> |
| Networking sub component: | router | QA Contact: | Hongan Li <hongli> |
| Status: | CLOSED DEFERRED | Docs Contact: | |
| Severity: | high | ||
| Priority: | unspecified | CC: | akanse, andbartl, dpateriy, jackevans43, linux78910, mabajodu, misalunk, mmasters, nagarjuna.gorantla, openshift-bugs-escalate, pjagtap, rfredette, ttaluh |
| Version: | 4.10 | ||
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-03-09 01:27:38 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Maria Alonso
2022-08-15 11:42:22 UTC
(In reply to Maria Alonso from comment #0) [...] > mtls connection is not working when using an intermetiate CA appart from the > root CA, both with CRL defined. > The Intermediate CA Cert had a published CDP which directed to a CRL issued > by the root CA. In the "ca-bundle.crt" data value of the "router-client-ca-default" config map from the must-gather archive, only the intermediate CA specifies a distribution point. > The config map in the openshift-ingress namespace contains the CRL as issued > by the root CA. The CRL issued by the Intermediate CA is not present since > that CDP is in the user cert and so not in the bundle. What do "CDP" and "user cert" refer to? I would have guessed that "CDP" referred to the X509v3 CRL distribution point and that "user cert" referred to the client certificate, but that doesn't make sense because the client certificate doesn't specify the CRL (it would be silly and useless for the client certificate to specify whether or not it had been revoked). > When attempting to connect using a user certificate issued by the > Intermediate CA it fails with an error of unknown CA. > > When attempting to connect using a user certificate issued by the to Root CA > the connection is successful. [...] > Steps to Reproduce: > > 1. Configure CA and intermediate CA with CRL > 2. Sign client certificate with the intermediate CA > 3. Configure mtls in openshift-ingress > > Actual results: > > When attempting to connect using a user certificate issued by the > Intermediate CA it fails with an error of unknown CA. > When attempting to connect using a user certificate issued by the to Root > CA the connection is successful. Are you saying that the router rejects the connection even if the client certificate and intermediate CA have not been revoked? (In reply to Miciah Dashiel Butler Masters from comment #2) > (In reply to Maria Alonso from comment #0) > [...] > > mtls connection is not working when using an intermetiate CA appart from the > > root CA, both with CRL defined. > > The Intermediate CA Cert had a published CDP which directed to a CRL issued > > by the root CA. > > In the "ca-bundle.crt" data value of the "router-client-ca-default" config > map from the must-gather archive, only the intermediate CA specifies a > distribution point. > That's right. The Root certificate doesn't contain a CDP (it's the root). The intermediate CA has a CDP to a location where the CRL from the Root CA is published. > > The config map in the openshift-ingress namespace contains the CRL as issued > > by the root CA. The CRL issued by the Intermediate CA is not present since > > that CDP is in the user cert and so not in the bundle. > > What do "CDP" and "user cert" refer to? I would have guessed that "CDP" > referred to the X509v3 CRL distribution point and that "user cert" referred > to the client certificate, but that doesn't make sense because the client > certificate doesn't specify the CRL (it would be silly and useless for the > client certificate to specify whether or not it had been revoked). > Correct CDP = X509v3 CRL distribution point User Cert = Client Cert Why is it silly for the client cert to have the CDP to identify the location of the CRL for checking the client certificate is not revoked? The CDP doesn't specify if the client certificate is revoked or not, only where to find the CRL as issued and signed by a trusted certificate authority to check if the client certificate status. For clarity The root certificate authority publishes a Certificate Revocation List (CRL), and certificates issued from the root authority contain an X509v3 CRL distribution point (CDP) so that the location of the CRL might be know to confirm the status for any certificates it has issued. The intermediate certificate authority, having a certificate issued from the root has a CDP identifying the CRL as issued by the root authority. A CRL is issued by the Intermediate Authority to identify any certificates that it issued that have now been revoked. Any certificates issued by the Intermediate Certificate Authority contain a CDP identifying the location of the CRL published by the Intermediate Certificate Authority. The "ca-bundle.crt" data value of the "router-client-ca-default" therefore contains the certificate for both the root and the intermediate certificate authorities. The intermediate certificate authority containing the CDP so that the operator may find the CRL issued by the root authority. In order to fully validate the chain the CRL published by the Intermediate Authority will also be needed. This can be identified by the CDP in the client certificate. The Validity of the CRL can be determined because the CRL is signed by the Intermediate authority, for which we have the certificate. > > When attempting to connect using a user certificate issued by the > > Intermediate CA it fails with an error of unknown CA. > > > > When attempting to connect using a user certificate issued by the to Root CA > > the connection is successful. > [...] > > Steps to Reproduce: > > > > 1. Configure CA and intermediate CA with CRL > > 2. Sign client certificate with the intermediate CA > > 3. Configure mtls in openshift-ingress > > > > Actual results: > > > > When attempting to connect using a user certificate issued by the > > Intermediate CA it fails with an error of unknown CA. > > When attempting to connect using a user certificate issued by the to Root > > CA the connection is successful. > > Are you saying that the router rejects the connection even if the client > certificate and intermediate CA have not been revoked? The client certificate is rejected even if no certificates in the chain are revoked. This is most likely because one of the required CRLs is missing so the chain can't be fully validated. If a client certificate issue by the root certificate authority is used then all the required components are present i.e. certificate authority certificate and CRL, and so the client is accepted. However client certificates should not be issued by the root authority. (In reply to linux78910 from comment #4) > Why is it silly for the client cert to have the CDP to identify the location > of the CRL for checking the client certificate is not revoked? The CDP > doesn't specify if the client certificate is revoked or not, only where to > find the CRL as issued and signed by a trusted certificate authority to > check if the client certificate status. Ah, you're right—because the CRL is signed by the issuing CA, so as long as we verify that the intermediate CA hasn't been revoked, we can trust that the CRL referenced by the client certificate and signed by the intermediate CA is valid. > For clarity > > The root certificate authority publishes a Certificate Revocation List > (CRL), and certificates issued from the root authority contain an X509v3 CRL > distribution point (CDP) so that the location of the CRL might be know to > confirm the status for any certificates it has issued. > > The intermediate certificate authority, having a certificate issued from the > root has a CDP identifying the CRL as issued by the root authority. > > A CRL is issued by the Intermediate Authority to identify any certificates > that it issued that have now been revoked. Any certificates issued by the > Intermediate Certificate Authority contain a CDP identifying the location of > the CRL published by the Intermediate Certificate Authority. > > The "ca-bundle.crt" data value of the "router-client-ca-default" therefore > contains the certificate for both the root and the intermediate certificate > authorities. The intermediate certificate authority containing the CDP so > that the operator may find the CRL issued by the root authority. > > In order to fully validate the chain the CRL published by the Intermediate > Authority will also be needed. This can be identified by the CDP in the > client certificate. The Validity of the CRL can be determined because the > CRL is signed by the Intermediate authority, for which we have the > certificate. I see, that makes sense. This is a use-case that we hadn't anticipated, and unfortunately the current design does not accommodate it. Only the operator downloads CRLs (which it provides to HAProxy), and the operator doesn't ever receive the client certificate; HAProxy receives the client certificate certificate, but HAProxy doesn't download CRLs. > > Are you saying that the router rejects the connection even if the client > > certificate and intermediate CA have not been revoked? > > The client certificate is rejected even if no certificates in the chain are > revoked. This is most likely because one of the required CRLs is missing so > the chain can't be fully validated. > > If a client certificate issue by the root certificate authority is used then > all the required components are present i.e. certificate authority > certificate and CRL, and so the client is accepted. > > However client certificates should not be issued by the root authority. Thanks for explaining. It seems like what is required here is some way to tell the operator where to find the CRL that the cluster administrator expects client certificates to use, so that the operator can download the CRL and provide it to HAProxy. By the way, did you use mTLS and CRLs on OpenShift 3? If so, can you describe the configuration that you used there? (In reply to Miciah Dashiel Butler Masters from comment #6) > By the way, did you use mTLS and CRLs on OpenShift 3? If so, can you > describe the configuration that you used there? I'll need to confirm but I'm pretty sure that the 3.11 solution was https://github.com/RedHatOfficial/ose-pivproxy (In reply to Miciah Dashiel Butler Masters from comment #5) <snip> > Thanks for explaining. It seems like what is required here is some way to > tell the operator where to find the CRL that the cluster administrator > expects client certificates to use, so that the operator can download the > CRL and provide it to HAProxy. Agreed. Some parameter to override the automated download from the CDP in the certificate would seem sensible, a list of required CRLs maybe. For the 3.11 cluster we used a pod on a cron to populate a configmap with a collection of CRLs from our environment. I thought also it might be useful to describe some workarounds we tried to fool the operator into downloading the 2nd crl. * We included a client cert in the bundle with the relevant CDP set. That resulted in two copies of the root crl and none of the intermediate CRL. * We added a certificate for an unused Certificate Authority with the relevant CDP set. This also resulted in two copies of the root CRL with none from the intermediate. * We added both CRLs into one file in the CDP. The operator discards the 2nd CRL and just takes whichever is first int the file, and I think duplicates it. @mmasters Thank you Miciah for your help on this issue! I understand that we need to open an RFE for this issue, it's ok? Regards MarÃa Hey Miciah - for 3.11 we deployed mTLS with haproxy as per the docs [1], setting --mutual-tls-auth-crl, then had a support ticket which gave us a CronJob to pull all the CRLs and put into a ConfigMap mounted into HAproxy. Would you be able to have a look into why our workaround above, of including a leaf certificate/sample client certificate with the CDP set in the CA bundle resulted in two copies of the root CRL and not the root CRL plus the intermediate CRL? I've had a look at the go code for the operator and it looks as it should obtain the CRL from the CDP in every certificate included in the bundle (regardless of if it's a root CA) and merge them into a single ConfigMap. [1] https://docs.openshift.com/container-platform/3.11/install_config/router/default_haproxy_router.html#mutual-tls-auth (In reply to linux78910 from comment #8) > Agreed. Some parameter to override the automated download from the CDP in > the certificate would seem sensible, a list of required CRLs maybe. [...] > I thought also it might be useful to describe some workarounds we tried to > fool the operator into downloading the 2nd crl. > > * We included a client cert in the bundle with the relevant CDP set. That > resulted in two copies of the root crl and none of the intermediate CRL. > * We added a certificate for an unused Certificate Authority with the > relevant CDP set. This also resulted in two copies of the root CRL with none > from the intermediate. > * We added both CRLs into one file in the CDP. The operator discards the 2nd > CRL and just takes whichever is first int the file, and I think duplicates > it. I don't understand why the first two workarounds don't work (the third one probably failed because the library that the operator uses to parse the CRL data only parses the first "X509 CRL" block that it finds). I haven't been able to reproduce whatever issue is causing those workarounds to fail, but I believe that resolving that issue and using the workaround would be the more expedient option (as opposed to adding a new API field to specify CRLs or CDPs). Does each certificate in the bundle have a unique subject key identifier? Are you able to provide operator logs (`oc -n openshift-ingress-operator logs deploy/ingress-operator -c ingress-operator`)? I'm particularly interested in lines containing "operator.crl". Any chance you could provide the public certificate itself, or sanitized output of `openssl x509 -noout -text -in intermediate-ca.pem`? I'm mostly interested in the "X509v3 extensions" section. (In reply to Miciah Dashiel Butler Masters from comment #14) > (In reply to linux78910 from comment #8) > > Agreed. Some parameter to override the automated download from the CDP in > > the certificate would seem sensible, a list of required CRLs maybe. > [...] > > I thought also it might be useful to describe some workarounds we tried to > > fool the operator into downloading the 2nd crl. > > > > * We included a client cert in the bundle with the relevant CDP set. That > > resulted in two copies of the root crl and none of the intermediate CRL. > > * We added a certificate for an unused Certificate Authority with the > > relevant CDP set. This also resulted in two copies of the root CRL with none > > from the intermediate. > > * We added both CRLs into one file in the CDP. The operator discards the 2nd > > CRL and just takes whichever is first int the file, and I think duplicates > > it. > > I don't understand why the first two workarounds don't work (the third one > probably failed because the library that the operator uses to parse the CRL > data only parses the first "X509 CRL" block that it finds). I haven't been > able to reproduce whatever issue is causing those workarounds to fail, but I > believe that resolving that issue and using the workaround would be the more > expedient option (as opposed to adding a new API field to specify CRLs or > CDPs). > > Does each certificate in the bundle have a unique subject key identifier? > > Are you able to provide operator logs (`oc -n openshift-ingress-operator > logs deploy/ingress-operator -c ingress-operator`)? I'm particularly > interested in lines containing "operator.crl". > > Any chance you could provide the public certificate itself, or sanitized > output of `openssl x509 -noout -text -in intermediate-ca.pem`? I'm mostly > interested in the "X509v3 extensions" section. Hi Miciah, apologies for delay in relying... Yes, each certificate has a unique subject key identifier. Please see below for the x509 output of each root/intermediate/user certificate.. ***Root Cert*** Certificate: Data: Version: 3 (0x2) Serial Number: 36:61:e1:ca:e2:12:49:8d:f0:ce:d6:bd:34:bb:2b:5c:b4:86:c0:28 Signature Algorithm: sha256WithRSAEncryption Issuer: C = GB, ST = England, O = Bluebird, OU = IT, CN = Bluebird Root CA, emailAddress = admin Validity Not Before: Jul 26 09:16:35 2022 GMT Not After : Jul 21 09:16:35 2042 GMT Subject: C = GB, ST = England, O = Bluebird, OU = IT, CN = Bluebird Root CA, emailAddress = admin Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public-Key: (4096 bit) Modulus: 00:a7:b1:35:c0:9b:e4:71:82:f2:ed:4d:ce:a9:a5: d3:54:5b:1a:c7:0c:e4:b4:ea:78:57:e2:d1:10:ba: 50:38:7b:fa:26:c0:7b:09:e4:52:30:ae:49:52:c9: ed:d3:a3:8b:62:af:82:5a:73:c6:49:c3:13:f1:6a: bf:63:a1:65:39:dc:5d:0c:ac:02:42:a8:b9:ad:c7: 25:bd:74:2d:a8:cb:36:76:73:70:3f:ac:7e:68:83: be:cb:27:7a:03:6d:b3:39:dd:e4:e1:be:2d:88:0e: 99:c7:23:27:06:be:b5:29:85:0f:47:36:cd:5e:17: b6:85:59:16:d5:40:8a:e0:83:f2:72:f9:99:4d:d6: 2b:d2:ad:d0:8e:15:37:f7:83:75:4c:3f:1f:aa:5b: 58:e4:74:72:6d:be:77:8a:b1:65:af:75:70:7a:33: a5:f2:3d:7f:1a:56:5c:5b:ac:6c:13:00:6e:41:dc: b6:95:6d:18:9e:f0:da:db:2d:06:6b:1d:dc:7d:ac: f2:9c:cd:aa:ae:dc:38:94:84:0f:77:07:f8:55:12: e0:f0:53:93:f2:15:3d:0c:b7:5c:31:60:14:4c:30: 22:6f:ac:f6:ea:23:9f:ee:de:1f:36:2e:32:16:ae: b1:24:38:79:04:c9:fc:6a:3e:24:6b:78:e8:d7:86: 8f:e4:0b:1f:d4:68:fd:9a:2f:42:82:7d:d6:52:05: 40:7e:68:f9:50:5a:62:8e:61:fb:96:32:ce:84:44: b7:ca:ba:95:fc:e6:17:eb:be:b9:79:96:da:cf:27: 51:ac:1e:d1:63:7b:8c:76:f9:ba:18:b8:3e:cd:cb: 42:95:76:79:e5:16:44:81:42:96:9a:e8:fc:0b:47: ea:22:ad:2b:eb:ce:24:a3:22:c1:28:63:c9:25:f4: 24:7b:9f:7f:40:9d:0d:16:0a:1e:96:cf:e8:3b:70: bd:b2:a9:fb:0f:a6:68:1c:05:d8:97:60:e7:3e:df: bf:78:39:2d:ff:77:53:eb:27:09:7c:65:8b:ae:56: 96:3b:94:b1:dc:09:a3:b5:4e:e9:3e:02:eb:a8:d2: 9e:70:18:6a:69:f8:c7:fa:42:1a:ed:13:4d:21:f9: 00:ac:ea:17:f5:11:b7:04:06:9c:12:7c:a7:69:32: c2:d4:67:13:3b:28:33:02:7c:f4:c1:89:10:f3:2b: 6c:9f:b6:ec:7f:3b:d8:4b:fc:2c:3a:85:80:7a:5f: 11:57:92:99:b3:3b:1e:ed:75:fa:da:32:8c:b9:2e: 51:c2:d0:a5:ea:e1:b2:df:fc:6c:8e:33:8e:62:46: 9a:60:1c:2a:71:2e:d2:78:be:c2:aa:17:7e:c3:de: 5a:23:2f Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Key Identifier: 35:CF:5F:8C:5A:20:DF:0D:B3:73:5F:C9:B2:00:B0:8A:03:A2:8E:BF X509v3 Authority Key Identifier: keyid:35:CF:5F:8C:5A:20:DF:0D:B3:73:5F:C9:B2:00:B0:8A:03:A2:8E:BF X509v3 Basic Constraints: critical CA:TRUE X509v3 Key Usage: critical Digital Signature, Certificate Sign, CRL Sign Signature Algorithm: sha256WithRSAEncryption 54:45:87:8e:2f:02:7c:39:e1:3f:e2:d2:50:55:d1:77:f3:d3: 66:19:52:65:7c:94:7d:99:26:ce:31:db:23:8a:8e:a3:18:bc: c5:89:8f:ce:26:ea:c3:b4:82:74:d7:7d:75:05:70:d3:03:ae: 92:26:26:86:5c:6b:8f:2d:82:86:37:5c:6e:f0:dd:40:70:0f: 0f:c3:9d:67:55:c8:15:1c:40:d5:48:81:99:b8:96:1d:57:2d: d7:80:59:b9:55:83:7c:56:bc:a3:f5:96:6c:1e:2a:b1:2c:12: 41:5a:74:14:c4:53:65:78:c2:2d:a5:2e:84:08:bb:fd:d6:28: c3:70:2e:71:8c:14:82:23:86:ab:b4:fc:09:6c:43:38:96:02: 8f:82:88:db:1e:8b:7d:58:95:7a:a2:7a:9e:99:d7:9b:8e:2b: e5:87:45:e3:34:9b:33:87:9c:52:ca:f3:0f:93:1c:58:8d:a1: ed:1a:87:7f:52:4f:62:a1:df:52:b6:6b:fc:cb:44:c1:af:89: d1:8e:16:f1:67:c9:b9:cb:e0:4f:84:f7:91:fa:c4:11:17:02: e1:2f:22:6d:1a:93:69:94:48:a2:e3:34:72:a8:1f:11:43:01: 36:bf:4d:a8:71:c3:43:58:de:4d:42:77:ad:c1:b4:6c:bf:2a: 0d:b1:b8:56:f5:ee:ac:5d:7b:37:38:60:6c:71:55:e7:0d:54: 02:4f:35:60:0d:bc:37:1d:af:14:e2:e1:6c:57:8e:cb:b4:3e: ab:c9:cf:ff:8f:17:15:dc:65:92:93:59:bf:66:3a:67:ef:f2: 82:46:78:c0:0b:24:af:ac:95:1c:2c:e6:99:69:ce:16:8d:18: d5:f2:2b:f5:48:1b:34:f5:bb:52:49:05:ca:37:23:ee:3b:f3: 1d:ac:23:44:0c:8e:c7:c4:b8:9e:89:0a:97:6d:1d:0a:75:95: 40:ee:88:99:97:6a:55:48:d6:07:b3:d3:68:bc:52:9e:41:b6: 1e:27:05:69:6b:8c:cb:7d:43:05:ef:37:61:ee:d9:22:66:8d: 26:48:12:d6:3d:74:1e:6f:32:c3:74:fb:5c:37:8f:e8:3f:c2: ea:c5:cd:f0:c3:a5:48:99:b1:38:af:e5:3d:c2:4f:bb:c7:4d: 1e:9b:0b:86:42:0a:6e:43:ad:bd:b1:f5:03:b1:1b:c8:0e:31: 69:e8:20:9b:44:4c:99:28:68:41:0b:54:02:bd:94:05:b6:ba: 2f:e3:db:69:51:f0:76:51:f9:a8:72:59:f0:7a:aa:ea:32:a0: 3d:98:af:ef:61:f5:b7:00:b7:9e:ff:8c:0f:62:f3:c8:97:1d: bc:d9:c2:e5:f6:20:01:f3 ***Intermediate Cert*** Certificate: Data: Version: 3 (0x2) Serial Number: 4096 (0x1000) Signature Algorithm: sha256WithRSAEncryption Issuer: C = GB, ST = England, O = Bluebird, OU = IT, CN = Bluebird Root CA, emailAddress = admin Validity Not Before: Jul 26 09:47:22 2022 GMT Not After : Jul 23 09:47:22 2032 GMT Subject: C = GB, ST = England, O = Bluebird, OU = IT, CN = Bluebird Intermediate CA, emailAddress = admin Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public-Key: (4096 bit) Modulus: 00:b8:63:18:bb:78:fe:00:d0:59:fe:08:a2:33:e5: 0c:50:f7:cc:c9:92:0e:ca:68:cd:f4:93:5a:75:59: 32:23:78:58:41:fa:6f:eb:2f:86:3a:c5:1b:0a:c0: ad:f9:38:8c:03:19:92:c8:18:f4:d5:2c:0a:14:4a: db:2f:06:f3:9b:a0:6f:0a:35:80:46:38:f3:08:6f: 22:fd:e0:05:45:15:7c:b8:28:57:0a:c8:81:8e:3e: 2a:3e:10:29:07:16:c2:3f:05:69:f1:32:3e:e7:aa: 69:06:78:ac:af:91:55:99:e5:3a:99:6f:b9:25:95: 18:1e:6f:4e:f6:c6:09:3c:8e:45:ed:fa:84:d3:ef: 09:b9:d2:e4:e2:c6:a2:56:20:92:81:1c:17:6c:cf: e7:c2:6e:2d:05:5a:9a:a2:ad:a3:7b:ef:98:db:2e: 77:27:c7:65:5d:20:c1:f4:cf:7e:cd:05:db:e8:ed: 9f:96:94:4b:f2:1a:8d:c0:f2:5c:72:b6:50:95:79: 0d:75:87:ad:8d:9b:bb:36:9e:8f:07:34:7f:ae:74: 08:e9:c8:97:01:b5:dd:98:ed:d6:6b:83:fe:03:62: 19:70:db:ec:65:05:43:94:2f:00:5f:4d:ef:4c:9b: 89:4e:99:4f:b5:dc:41:a5:3f:09:52:0d:af:68:9a: 78:98:cb:03:72:a3:d0:67:97:1d:04:28:87:8a:cf: cc:01:5d:33:b2:e3:8e:8a:23:a6:74:c3:be:f6:da: a8:e7:d2:90:21:d9:1b:77:1b:f1:56:35:ec:9a:09: 0c:32:0f:60:89:c7:3d:45:d3:c3:1c:76:97:d1:01: 42:57:09:80:91:a5:c2:ad:95:59:9d:80:0c:a3:b1: ce:21:3b:e4:fe:30:42:e6:62:3d:40:37:2f:93:2e: ed:16:86:19:92:95:42:ad:22:be:80:9d:48:8a:bc: 76:7c:89:de:15:7a:72:bb:73:8d:08:cd:94:10:56: 6a:91:cb:d5:b6:a8:54:87:e2:e5:8e:97:21:0c:c9: 81:00:c2:c3:f9:9e:56:33:26:15:60:9e:8d:29:ec: 25:56:03:88:7e:30:a5:73:50:2a:ab:8f:8f:c2:8f: 88:59:9e:ad:cd:9c:d5:2d:30:b5:ad:1d:9c:84:cd: 39:0f:74:cf:8c:80:28:8c:6a:c9:6b:8f:d9:4e:97: ce:66:a6:ca:51:a3:84:c0:a0:5b:a7:be:7f:d2:4e: 26:a5:ed:7a:fd:95:f5:c2:22:34:1f:b6:e7:ee:ac: 80:3a:a1:f5:15:ff:23:ea:1b:ae:d2:f6:8b:ff:b1: be:42:dc:c1:bf:a7:53:65:62:54:66:76:0f:4b:aa: 5f:49:d3 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Key Identifier: 12:47:E8:5C:8D:02:18:C3:51:89:3C:C9:E8:AB:91:A8:FF:D4:8A:1D X509v3 Authority Key Identifier: keyid:35:CF:5F:8C:5A:20:DF:0D:B3:73:5F:C9:B2:00:B0:8A:03:A2:8E:BF X509v3 Basic Constraints: critical CA:TRUE, pathlen:0 X509v3 Key Usage: critical Digital Signature, Certificate Sign, CRL Sign X509v3 CRL Distribution Points: Full Name: URI:http://bastion.opt3.io/crl/ca-root.crl.pem Signature Algorithm: sha256WithRSAEncryption 39:f7:3d:a6:6d:bf:12:90:04:0c:60:91:0d:9b:63:84:36:f1: 94:93:8d:e1:ce:b5:79:b3:02:79:5b:e3:08:d6:46:5c:03:fe: ee:4a:c3:c9:55:aa:58:99:09:eb:79:65:73:28:32:7c:ad:a3: 7e:8a:57:e1:1e:05:3c:64:cd:2a:9e:94:e6:a4:27:12:04:f7: 17:7e:4f:98:ac:86:86:b4:68:84:47:cf:fa:cb:2f:b5:b3:bb: 8a:24:04:ad:37:c8:42:d4:79:94:67:68:63:03:2f:ec:0d:47: 40:ba:45:0b:7f:87:27:8a:c5:59:95:5a:b0:66:03:40:2b:0d: d9:94:4a:16:2f:72:f4:37:cf:7a:56:86:b1:a9:e0:14:04:3a: 9d:5d:bf:9f:8d:42:96:0d:0b:2e:34:40:2f:67:16:45:f2:29: b2:25:ab:0c:06:a1:f5:65:c3:44:a0:3a:1e:d9:04:ef:7b:f2: 50:1f:1d:63:75:d6:91:2f:ff:c5:8e:59:41:e4:61:d2:0f:d2: c1:cb:68:de:8c:a4:de:e3:4d:93:91:1d:3b:db:07:54:85:c5: 01:b2:7e:88:33:fd:5c:24:6c:52:82:d7:75:03:03:c2:62:29: d6:c5:d8:a2:ca:b8:2e:a3:fc:c9:77:b4:67:f6:c9:73:4e:4b: 8d:4f:70:55:02:fc:c9:5a:67:e9:e4:f8:53:77:e4:26:2d:a2: 64:5d:70:4c:ac:64:93:83:fc:b9:74:10:b9:00:57:16:fa:27: f7:5f:f2:a1:dc:ed:c6:cd:85:1b:60:f1:fe:19:c5:5c:82:cc: 51:00:dd:d1:87:1f:9b:a2:72:5d:f6:71:b3:fa:ef:9e:2c:1a: d6:6f:61:3d:da:5d:3b:71:c8:0b:2c:90:88:c2:da:36:1e:ac: 42:b7:49:0f:de:4b:d0:c1:db:f9:3d:ed:ad:60:a5:11:80:74: b2:dd:71:cf:3b:ab:e1:6f:28:e1:43:da:da:42:d3:e3:a4:70: 9f:f9:f5:55:3e:26:5c:7e:56:8f:9c:6f:86:a2:58:39:0a:ca: 3e:d3:16:39:c2:1e:40:18:6f:ab:9f:9c:09:47:6b:39:49:80: 4f:3b:59:ca:0c:e4:22:02:bf:fe:e5:dd:c1:d2:86:8c:21:47: 57:f6:e9:4b:86:f7:9d:1b:30:86:cd:44:00:6f:78:bb:b2:50: 56:9a:c6:57:96:19:ac:da:d8:43:c4:f9:d1:1e:b5:fa:fb:c3: 8b:3e:34:2f:c4:95:f5:b6:8d:3a:aa:78:58:ec:4b:28:c9:ea: 58:dc:2f:1a:6a:cc:c9:fa:01:39:99:73:71:64:ce:b4:8a:f7: 27:09:06:de:9a:71:50:5e ***User Cert*** Certificate: Data: Version: 3 (0x2) Serial Number: 4096 (0x1000) Signature Algorithm: sha256WithRSAEncryption Issuer: C = GB, ST = England, O = Bluebird, OU = IT, CN = Bluebird Intermediate CA, emailAddress = admin Validity Not Before: Jul 26 10:02:18 2022 GMT Not After : Aug 5 10:02:18 2023 GMT Subject: C = GB, ST = England, O = Bluebird, OU = IT, CN = user1, emailAddress = user1 Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public-Key: (2048 bit) Modulus: 00:a1:65:a9:f3:7e:40:78:fa:a4:57:b4:f0:ef:b2: 35:85:0a:d9:18:02:9f:9c:b6:ee:a3:82:71:b5:b0: 96:02:c2:b3:1d:4f:38:95:c7:4e:3e:1b:17:98:3c: 51:a4:a5:ed:85:b9:ec:14:fb:d2:94:40:37:f8:bb: 94:be:20:0d:73:45:49:7f:53:6e:fd:df:4e:0e:c4: 4b:2e:3c:91:f9:e4:f2:86:c2:43:5c:87:41:78:74: 96:9f:5e:60:80:4a:5f:ce:0c:42:12:f8:ca:90:6d: b1:b3:99:36:e6:73:35:59:bd:a1:8b:fe:83:7b:79: 01:85:c7:f6:2c:0b:34:73:f4:cb:c2:90:5a:80:22: 1a:e7:c2:77:1b:bf:f6:17:e6:02:18:c5:6c:a7:45: b8:7e:94:8d:52:7d:79:89:9e:cd:02:b1:83:b5:56: b2:2f:3d:26:e0:d2:ff:9f:48:f0:a1:c8:84:84:0e: 9c:ba:50:db:9c:c0:8b:c8:3a:97:4b:7f:40:29:d4: 09:ca:58:c6:a5:17:9b:c5:80:d8:a8:65:85:ab:d8: 9a:39:50:a4:57:b6:e7:39:89:86:0e:71:aa:f9:d4: 9c:8b:a2:e4:4b:1a:2d:2f:99:71:59:2b:0c:70:dd: 4d:c6:75:99:af:50:f0:bf:66:78:90:70:8a:c3:25: 83:65 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Basic Constraints: CA:FALSE Netscape Cert Type: SSL Client, S/MIME Netscape Comment: OpenSSL Generated Client Certificate X509v3 Subject Key Identifier: 1D:42:04:61:91:F2:16:C2:F0:24:AF:91:A2:4D:90:92:83:52:C3:32 X509v3 Authority Key Identifier: keyid:12:47:E8:5C:8D:02:18:C3:51:89:3C:C9:E8:AB:91:A8:FF:D4:8A:1D X509v3 Key Usage: critical Digital Signature, Non Repudiation, Key Encipherment X509v3 Extended Key Usage: TLS Web Client Authentication, E-mail Protection X509v3 CRL Distribution Points: Full Name: URI:http://bastion.opt3.io/crl/ca-intermediate.crl.pem Signature Algorithm: sha256WithRSAEncryption 9e:60:20:68:a0:03:c1:fd:9c:4c:e2:85:0f:09:bb:60:f1:b9: 03:e8:08:12:d2:79:28:32:56:26:cb:40:62:62:6a:43:07:68: ec:fa:97:44:16:82:73:4c:99:4b:b1:40:64:9a:96:e4:df:22: f1:ab:ce:12:7b:8a:e7:4e:63:a2:c7:62:4d:b8:e7:09:2c:5b: a9:f7:3f:12:fd:70:dc:65:76:17:8b:02:97:aa:8b:eb:3f:f1: ac:6d:58:fd:38:27:e2:a1:e9:0e:e7:b9:6f:51:9d:58:99:0f: 02:59:4c:71:ab:f8:ad:10:1a:55:29:cb:77:50:c8:3b:61:c2: bb:77:a0:62:01:d3:ba:01:81:48:e6:a6:39:04:a1:06:c6:4e: e8:be:b0:65:98:f9:88:00:7f:ca:54:b2:82:df:89:82:5f:5c: 52:d7:13:7e:cb:87:18:6d:1f:e7:be:0a:f9:10:fa:56:d4:c2: b1:23:98:0a:f1:9d:b6:13:0b:cb:f3:e1:43:cd:9b:01:a8:ea: 7d:f4:9d:74:9b:9f:03:6c:bc:06:ef:4f:51:2a:a7:86:67:72: 7c:b6:17:51:ad:7c:8b:3a:9b:57:38:8a:51:d8:21:9f:95:f6: 40:19:bf:47:8f:05:2a:40:67:ab:26:c8:db:e4:0d:f1:1a:08: 89:2b:5d:5b:ab:b2:ba:93:a8:3f:c1:84:db:d9:25:03:36:ad: 47:39:37:94:1e:a3:64:f2:70:85:4e:02:f7:61:e8:83:d2:7b: 7b:34:f6:98:15:37:ef:a6:eb:09:03:b3:61:3c:b5:af:f7:80: d7:28:ef:6c:de:ec:bb:f3:f1:2c:1b:8e:41:47:5b:85:f1:27: 4a:a0:b4:29:27:cc:bb:f1:bb:5c:93:71:a8:19:58:d6:35:6b: 16:f7:76:f4:d2:ef:75:86:48:18:5f:7e:4a:29:72:2e:24:10: 45:9c:b8:64:5d:3b:b4:fa:51:d8:aa:87:50:d0:d2:c0:6c:2f: 56:3c:b8:b9:9e:8b:68:9e:87:09:5c:9e:87:32:18:2f:4b:b2: 8b:4d:5d:9d:f6:9d:4a:d6:b3:25:00:e0:8d:56:16:11:c9:7e: d9:9f:ef:21:42:95:e0:13:a2:67:3b:05:bc:25:6f:a6:ba:b1: 04:92:fd:72:88:e9:8d:84:86:9d:f0:8c:c9:3a:07:07:2e:d9: 88:83:b7:48:4e:1d:8d:3d:bb:a5:ec:c6:3e:9c:9e:a3:02:58: f7:41:6c:d4:57:60:9e:a9:3b:d0:37:d9:da:d7:f5:c6:83:62: 37:50:bd:70:7d:ea:da:77:1a:bc:70:4b:c1:31:2a:ae:34:3b: 12:4a:cc:88:1d:e1:a5:e4 I have previously provided the operator logs in the support case but I can if required reproduce these. Thanks I was able to reproduce the issue. I created a root CA with no CDP, an intermediate CA with the root CA's CDP, and several client certificates signed by the intermediate CA, which include the intermediate CA's CDP. By configuring the ingress controller's clientCA to only have the intermediate and root CA certificates, the intermediate CA's CRL is missing, and mTLS fails for the certs signed by the intermediate CA, although certs signed by the root CA work.
However, the workaround here *did* work for me:
> * We included a client cert in the bundle with the relevant CDP set. That resulted in two copies of the root crl and none of the intermediate CRL.
by adding the client cert before the 2 CA certs in the client CA bundle, the operator does download both CRLs, and my certs signed by the intermediate CA work.
I wonder if the order in the CA bundle is relevant? I had the order as client > intermediate > root, which worked. I was about to try intermediate > root > client to see if it made any difference, but unfortunately I lost access to my cluster right before I could test that out. I will try first thing tomorrow and report back.
I know it has been a while since that workaround was tried, but do you recall what order you had the certificates in for the client CA bundle?
I tested out changing where in the client CA bundle I placed the client certificate containing the intermediate CA's CDP, but it didn't seem to matter. With that though, I'm not sure what other differences there could be that would result in the workaround working on my cluster but not another's. @rfredette the proposed solution has been tested and worked at the beginning with both crls appearing in the configmap. But after renewing the crls as they had expired and deleting the config map to refresh it, the behavior has returned when it contains 2 duplicate crls from the intermediate CA. The order of certs in the configmap are `client>intermediate>root`. Could you try to test this scenario? Thank you! > But after renewing the crls as they had expired and deleting the config map to refresh it, the behavior has returned when it contains 2 duplicate crls from the intermediate CA.
So far I haven't been able to reproduce this particular part of the issue, but I will see what I can do. I have a couple clarifying questions that may help me:
* Does it seem to be only happening when the CRLs expire and get updated? Or can it happen when the ingress controller is newly set up?
* Does this happen every time the CRLs are updated on this cluster, or is it intermittent?
* As a sanity check, does each certificate in the client CA bundle have a unique Subject Key Identifier?
@rfredette Talking to customer who is running into this same issue. Customer mentioned that adding client certificate (leaf cert) in CA bundle in the order client->intermediate->root works only for Public CA signed certificates. It doesn't work for customer signed certificates. Customer is really waiting for this fix. If there is any information needed from customer or any testing need to be performed on fix we can engage customers. Please see CASE # https://access.redhat.com/support/cases/#/case/03392443 > client->intermediate->root works only for Public CA signed certificates I was able to use that workaround with a cert chain where the "root" CA is self signed, so I don't think it necessarily needs to be a public CA for it to work. If adding a leaf cert to the bundle doesn't fix the issue, I wonder if the root cause of their failure is different to the one that this bug describes. Here's some steps you should be able to run to tell if this is the same issue or a different one: 1. Check if the client certificate includes a CRL Distribution Point: --- $ openssl x509 -text -noout -in artifacts/signed-by-intermediate.pem|grep -A0 -e "Issuer:" -e "Subject:" -e "X509v3 CRL Distribution Points:" -A 5 Issuer: C = US, ST = NC, L = Raleigh, O = OS4, OU = Eng, CN = Test Intermediate CA ... Subject: C = US, ST = NC, L = Raleigh, O = OS4, OU = Eng, CN = Another Test Client ... X509v3 CRL Distribution Points: Full Name: URI:http://crl-host.crl-test.svc/intermediate/intermediate.crl CRL Issuer: DirName:C = US, ST = NC, L = Raleigh, O = OS4, OU = Eng, CN = Test Intermediate CA ... --- The "X509v3 CRL Distribution Points" section may include multiple URIs. These should all point to different ways to acquire the same CRL. If no CRL Distribution Point is listed, then the connection issues are due to a different issue than this one. 2. Get the ca bundle for your router, and check if there is a source for the same CRL already in the bundle: --- $ oc -n openshift-ingress get configmap/router-client-ca-default -o jsonpath='{.data.ca-bundle\.pem}' > /tmp/ca-bundle.pem; openssl crl2pkcs7 -nocrl -certfile /tmp/ca-bundle.pem|openssl pkcs7 -print_certs -text -noout|grep -e "Issuer:" -e "Subject:" -e "X509v3 CRL Distribution Points:" -A 5 Issuer: C=US, ST=NC, L=Raleigh, O=OS4, OU=Eng, CN=Test Root CA ... Subject: C=US, ST=NC, L=Raleigh, O=OS4, OU=Eng, CN=Test Intermediate CA ... X509v3 CRL Distribution Points: Full Name: URI:http://crl-host.crl-test.svc/root/root.crl CRL Issuer: DirName:C = US, ST = NC, L = Raleigh, O = OS4, OU = Eng, CN = Test Root CA ... Issuer: C=US, ST=NC, L=Raleigh, O=OS4, OU=Eng, CN=Test Root CA ... Subject: C=US, ST=NC, L=Raleigh, O=OS4, OU=Eng, CN=Test Root CA ... --- My example is using the default router, but if you are using a custom router, the configmap will be named configmap/router-client-ca-<router-name>. In this example, there's a CRL Distribution Point for "CN=Test Root CA", but none for "CN=Test Intermediate CA". This will cause mTLS connections to fail. This is the use-case where adding a client certificate to the clientCA bundle will help, because it will include the CRL Distribution Point that is missing from the bundle. Here is what the output looks like after the workaround is applied: --- $ oc -n openshift-ingress get configmap/router-client-ca-default -o jsonpath='{.data.ca-bundle\.pem}' > /tmp/ca-bundle.pem; openssl crl2pkcs7 -nocrl -certfile /tmp/ca-bundle.pem|openssl pkcs7 -print_certs -text -noout|grep -e "Issuer:" -e "Subject:" -e "X509v3 CRL Distribution Points:" -A 5 Issuer: C=US, ST=NC, L=Raleigh, O=OS4, OU=Eng, CN=Test Intermediate CA ... Subject: C=US, ST=NC, L=Raleigh, O=OS4, OU=Eng, CN=Another Test Client ... X509v3 CRL Distribution Points: Full Name: URI:http://crl-host.crl-test.svc/intermediate/intermediate.crl CRL Issuer: DirName:C = US, ST = NC, L = Raleigh, O = OS4, OU = Eng, CN = Test Intermediate CA ... Issuer: C=US, ST=NC, L=Raleigh, O=OS4, OU=Eng, CN=Test Root CA ... Subject: C=US, ST=NC, L=Raleigh, O=OS4, OU=Eng, CN=Test Intermediate CA ... X509v3 CRL Distribution Points: Full Name: URI:http://crl-host.crl-test.svc/root/root.crl CRL Issuer: DirName:C = US, ST = NC, L = Raleigh, O = OS4, OU = Eng, CN = Test Root CA ... Issuer: C=US, ST=NC, L=Raleigh, O=OS4, OU=Eng, CN=Test Root CA ... Subject: C=US, ST=NC, L=Raleigh, O=OS4, OU=Eng, CN=Test Root CA ... --- This bundle includes the same CRL that the client lists (the one for "CN= Test Intermediate CA"). 3. Once you're sure the clientCA bundle includes all necessary CRL Distribution Points, check that the ingress operator was able to download all the CRLs: --- $ oc -n openshift-ingress get configmap/router-client-ca-crl-default -o jsonpath='{.data.crl\.pem}'|csplit -s -z -f /tmp/crls- - '/-----BEGIN X509 CRL-----/' '{*}' $ for file in $(ls /tmp/crls-*); do openssl crl -text -noout -in $file|grep "Issuer:"; done Issuer: C = US, ST = NC, L = Raleigh, O = OS4, OU = Eng, CN = Test Intermediate CA Issuer: C = US, ST = NC, L = Raleigh, O = OS4, OU = Eng, CN = Test Root CA --- If 'csplit' is not available on your system, you may need to manually split the CRL bundle into individual CRL files to run through 'openssl crl'. Each file should include just one of the blocks starting with "-----BEGIN X509 CRL-----" and ending with "-----END X509 CRL-----" If there are any issues with missing CRLs at this point, check the ingress operator logs to see if it failed to download any CRLs. The relevant log lines should be from "operator.crl": --- $ oc -n openshift-ingress-operator logs ingress-operator-<id> ingress-operator|grep "operator\.crl" --- Once the clientCA bundle is patched, it may take several minutes for the update to propagate to the router pods, but once it's complete there should no longer be issues completing mTLS connections from the openshift side. If you're still seeing issues at that point, check the logs and output of your client to look for TLS errors OpenShift has moved to Jira for its defect tracking! This bug can now be found in the OCPBUGS project in Jira. https://issues.redhat.com/browse/OCPBUGS-9464 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days |