Bug 2029023
| Summary: | ipa: ERROR: 'Certificate operation cannot be completed: Unable to communicate with CMS (403) | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Vinay Mishra <vmishra> | |
| Component: | pki-core | Assignee: | Endi Sukma Dewata <edewata> | |
| Status: | CLOSED ERRATA | QA Contact: | PKI QE <bugzilla-pkiqe> | |
| Severity: | unspecified | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 8.5 | CC: | aakkiang, abokovoy, apeddire, ckelley, edewata, hakon.gislason, ksiddiqu, matthew.lesieur, mharmsen, msauton, pgm-rhel-tools, prisingh, rhcs-maint, sjansen, skhandel, ssidhaye, sumenon, tmihinto, toneata, tscherf, wrydberg | |
| Target Milestone: | rc | Keywords: | Triaged, ZStream | |
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | pki-core-10.6-8050020220111200158.3246ec52 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 2041399 2061458 (view as bug list) | Environment: | ||
| Last Closed: | 2022-02-01 21:19:58 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 2041399, 2061458 | |||
|
Comment 4
sjansen
2021-12-16 22:15:42 UTC
(In reply to sjansen from comment #4) > Today we updated severeal ipa servers with the latest ipa-server errata on > our RHEL 8.5 machines with several replicas. My private single node ipa > server also got affacted after updating. After updating the ipa packages the > CA completely stops working, web gui show "unable to communicate with CMS > (403)", every cert command is unable to communicate with the rest API. ipa > cert-show 1 throw "ERROR: Certificate operation cannot be completed: Request > failed with status 403: Non-2xx response from CA REST API: 403" > > In apache error log i can see errors about failed to set perms (3140) on > /run/ipa/ccaches/myusername but nothing else that can > help. > > On my private machine i tried a package rollback (i know i should not do > that), but the issues persist, only a rollback of my machine and avoiding > the latest errata works for me. Could you please tell your package versions for ipa and pki? latest ipa update 4.9.6-10 only adds code to harden installation for CVE-2020-25717 which has nothing to do with CA operations. Most likely, it is an issue with PKI side that keeps modifying AJP connector secret in tomcat configuration without coordinating it with IPA change in httpd configuration. This would be visible with egrep "secret|requiredSecret" /etc/httpd/conf.d/ipa-pki-proxy.conf /etc/pki/pki-tomcat/server.xml in tomcat's server.xml there should only be 'requiredSecret' field, not 'secret', while in ipa-pki-proxy.conf there should be 'secret' field with the same value as 'requiredSecret'. Err, I explained it the other way around -- there should be no 'requiredSecret' in server.xml, only 'secret' in both. Hi Alexander, i just grepped my config and i can see that that both config files contain secret="somesecret" but /etc/pki/pki-tomcat/server.xml also contain two entries for requiredSecret="somesecret", one in "<connector port="8009...." and one in <connector address="localhost6" name="Connector1...". Sorry for not pasting the content, these system are now airgaped because we had to rollback our ipa servers and i just keep one machine for logs/debugging in disconnected mode over vm console. These are all the updates we installed that lead to this issue, i can repeat this by installing them again and rolling the machine back to get my working config again, so this may help for further debugging. Upgrading: ipa-client x86_64 4.9.6-10.module+el8.5.0+13587+92118e57 rhel-8-for-x86_64-appstream-rpms 281 k ipa-client-common noarch 4.9.6-10.module+el8.5.0+13587+92118e57 rhel-8-for-x86_64-appstream-rpms 184 k ipa-common noarch 4.9.6-10.module+el8.5.0+13587+92118e57 rhel-8-for-x86_64-appstream-rpms 796 k ipa-selinux noarch 4.9.6-10.module+el8.5.0+13587+92118e57 rhel-8-for-x86_64-appstream-rpms 176 k ipa-server x86_64 4.9.6-10.module+el8.5.0+13587+92118e57 rhel-8-for-x86_64-appstream-rpms 530 k ipa-server-common noarch 4.9.6-10.module+el8.5.0+13587+92118e57 rhel-8-for-x86_64-appstream-rpms 612 k ipa-server-dns noarch 4.9.6-10.module+el8.5.0+13587+92118e57 rhel-8-for-x86_64-appstream-rpms 192 k libwbclient x86_64 4.14.5-7.el8_5 rhel-8-for-x86_64-baseos-rpms 121 k python3-ipaclient noarch 4.9.6-10.module+el8.5.0+13587+92118e57 rhel-8-for-x86_64-appstream-rpms 688 k python3-ipalib noarch 4.9.6-10.module+el8.5.0+13587+92118e57 rhel-8-for-x86_64-appstream-rpms 756 k python3-ipaserver noarch 4.9.6-10.module+el8.5.0+13587+92118e57 rhel-8-for-x86_64-appstream-rpms 1.6 M samba-client-libs x86_64 4.14.5-7.el8_5 rhel-8-for-x86_64-baseos-rpms 5.4 M samba-common noarch 4.14.5-7.el8_5 rhel-8-for-x86_64-baseos-rpms 221 k samba-common-libs x86_64 4.14.5-7.el8_5 rhel-8-for-x86_64-baseos-rpms 174 k selinux-policy noarch 3.14.3-80.el8_5.2 rhel-8-for-x86_64-baseos-rpms 636 k selinux-policy-targeted noarch 3.14.3-80.el8_5.2 rhel-8-for-x86_64-baseos-rpms 15 M systemd x86_64 239-51.el8_5.3 rhel-8-for-x86_64-baseos-rpms 3.6 M systemd-libs x86_64 239-51.el8_5.3 rhel-8-for-x86_64-baseos-rpms 1.1 M systemd-pam x86_64 239-51.el8_5.3 rhel-8-for-x86_64-baseos-rpms 477 k systemd-udev x86_64 239-51.el8_5.3 rhel-8-for-x86_64-baseos-rpms 1.6 M Thanks. As I said, IPA update is not a reason for these failures. It looks like it is part of PKI internal upgrade code. You can mitigate this issue by replacing 'requiredSecret' with 'secret' in all connectors and make sure all of them are using the same value as in ipa-pki-proxy.conf. Removing requiredSecret did the trick, i just did a quick "ipa cert-show 1" and the cert are listed again. Do you know if i have to report this somewhere else to get the pki packages fixed or is this already handled with this report? Alexander, you really saved my last day before holiday, thank you very much! This is the bug against pki-core so it will be handled here. Enjoy your holidays. Searching for "Unable to communicate with CMS" sent me to a Red Hat solution, which didn't resolve the problem. Removing requiredSecret as outlined above did fix the problem for me. Should the solution be updated? https://access.redhat.com/solutions/4796941 the HTTP error code 403 is kind of generic and can happen in various situations. for this issue, we need this article: https://access.redhat.com/solutions/6632811 *** Bug 2041399 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (pki-core:10.6 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:0357 This issue came back with latest pki update from this week, FreeIPA again show CMS error 403 in the certificate area.
Upgrade pki-base-10.11.2-5.module+el8.5.0+14437+bc030dcc.noarch @rhel-8-for-x86_64-appstream-rpms
Upgraded pki-base-10.11.2-4.module+el8.5.0+13827+5b1d191d.noarch @@System
Upgrade pki-server-10.11.2-5.module+el8.5.0+14437+bc030dcc.noarch @rhel-8-for-x86_64-appstream-rpms
Upgraded pki-server-10.11.2-4.module+el8.5.0+13827+5b1d191d.noarch @@System
Upgrade pki-acme-10.11.2-5.module+el8.5.0+14437+bc030dcc.noarch @rhel-8-for-x86_64-appstream-rpms
Upgraded pki-acme-10.11.2-4.module+el8.5.0+13827+5b1d191d.noarch @@System
Upgrade python3-pki-10.11.2-5.module+el8.5.0+14437+bc030dcc.noarch @rhel-8-for-x86_64-appstream-rpms
Upgraded python3-pki-10.11.2-4.module+el8.5.0+13827+5b1d191d.noarch @@System
Upgrade pki-base-java-10.11.2-5.module+el8.5.0+14437+bc030dcc.noarch @rhel-8-for-x86_64-appstream-rpms
Upgraded pki-base-java-10.11.2-4.module+el8.5.0+13827+5b1d191d.noarch @@System
Upgrade pki-kra-10.11.2-5.module+el8.5.0+14437+bc030dcc.noarch @rhel-8-for-x86_64-appstream-rpms
Upgraded pki-kra-10.11.2-4.module+el8.5.0+13827+5b1d191d.noarch @@System
Upgrade pki-symkey-10.11.2-5.module+el8.5.0+14437+bc030dcc.x86_64 @rhel-8-for-x86_64-appstream-rpms
Upgraded pki-symkey-10.11.2-4.module+el8.5.0+13827+5b1d191d.x86_64 @@System
Upgrade pki-tools-10.11.2-5.module+el8.5.0+14437+bc030dcc.x86_64 @rhel-8-for-x86_64-appstream-rpms
Upgraded pki-tools-10.11.2-4.module+el8.5.0+13827+5b1d191d.x86_64 @@System
Upgrade pki-ca-10.11.2-5.module+el8.5.0+14437+bc030dcc.noarch @rhel-8-for-x86_64-appstream-rpms
Upgraded pki-ca-10.11.2-4.module+el8.5.0+13827+5b1d191d.noarch @@System
Little update: i solved the issue by applying the previous workaround but this time a new problem came up. Again this update changed the connector settings in "/etc/pki/pki-tomcat/server.xml" from "secret=" to "requiredsecret=", but this time it also changed the secret key itself. So setting the connector back from "requiredSecret=" to "secret=" solve "CMS Error 403" but now i got something with Error 50x, then i compared the key from "/etc/httpd/conf.d/ipa-pki-proxy.conf" and found that is so longer the same, so i took over the key (like it was before the update) and now after a restart the IPA certificate section is working again. pre update: /etc/pki/pki-tomcat/server.xml: <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="localhost4" name="Connector1" secret="Oldkey"/> /etc/pki/pki-tomcat/server.xml: <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="localhost6" name="Connector2" secret="Oldkey"/> after update: /etc/pki/pki-tomcat/server.xml: <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="localhost4" name="Connector1" requiredSecret="Newkey"/> /etc/pki/pki-tomcat/server.xml: <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="localhost6" name="Connector2" requiredSecret="NewKey"/> my fix: /etc/pki/pki-tomcat/server.xml: <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="localhost4" name="Connector1" secret="Oldkey"/> /etc/pki/pki-tomcat/server.xml: <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="localhost6" name="Connector2" secret="Oldkey"/> The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days |