Today we updated severeal ipa servers with the latest ipa-server errata on our RHEL 8.5 machines with several replicas. My private single node ipa server also got affacted after updating. After updating the ipa packages the CA completely stops working, web gui show "unable to communicate with CMS (403)", every cert command is unable to communicate with the rest API. ipa cert-show 1 throw "ERROR: Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403"

In apache error log i can see errors about failed to set perms (3140) on /run/ipa/ccaches/myusername but nothing else that can help.

On my private machine i tried a package rollback (i know i should not do that), but the issues persist, only a rollback of my machine and avoiding the latest errata works for me.

(In reply to sjansen from comment #4)
> Today we updated severeal ipa servers with the latest ipa-server errata on
> our RHEL 8.5 machines with several replicas. My private single node ipa
> server also got affacted after updating. After updating the ipa packages the
> CA completely stops working, web gui show "unable to communicate with CMS
> (403)", every cert command is unable to communicate with the rest API. ipa
> cert-show 1 throw "ERROR: Certificate operation cannot be completed: Request
> failed with status 403: Non-2xx response from CA REST API: 403"
> 
> In apache error log i can see errors about failed to set perms (3140) on
> /run/ipa/ccaches/myusername but nothing else that can
> help.
> 
> On my private machine i tried a package rollback (i know i should not do
> that), but the issues persist, only a rollback of my machine and avoiding
> the latest errata works for me.

Could you please tell your package versions for ipa and pki?
latest ipa update 4.9.6-10 only adds code to harden installation for CVE-2020-25717 which has nothing to do with CA operations. 

Most likely, it is an issue with PKI side that keeps modifying AJP connector secret in tomcat configuration without coordinating it with IPA change in httpd configuration.

This would be visible with
egrep "secret|requiredSecret" /etc/httpd/conf.d/ipa-pki-proxy.conf /etc/pki/pki-tomcat/server.xml

in tomcat's server.xml there should only be 'requiredSecret' field, not 'secret', while in ipa-pki-proxy.conf there should be 'secret' field with the same value as 'requiredSecret'.

Err, I explained it the other way around -- there should be no 'requiredSecret' in server.xml, only 'secret' in both.

Hi Alexander,

i just grepped my config and i can see that that both config files contain secret="somesecret" but /etc/pki/pki-tomcat/server.xml also contain two entries for requiredSecret="somesecret", one in "<connector port="8009...." and one in <connector address="localhost6" name="Connector1...". Sorry for not pasting the content, these system are now airgaped because we had to rollback our ipa servers and i just keep one machine for logs/debugging in disconnected mode over vm console.

These are all the updates we installed that lead to this issue, i can repeat this by installing them again and rolling the machine back to get my working config again, so this may help for further debugging.

Upgrading:
 ipa-client                              x86_64                 4.9.6-10.module+el8.5.0+13587+92118e57                   rhel-8-for-x86_64-appstream-rpms                 281 k
 ipa-client-common                       noarch                 4.9.6-10.module+el8.5.0+13587+92118e57                   rhel-8-for-x86_64-appstream-rpms                 184 k
 ipa-common                              noarch                 4.9.6-10.module+el8.5.0+13587+92118e57                   rhel-8-for-x86_64-appstream-rpms                 796 k
 ipa-selinux                             noarch                 4.9.6-10.module+el8.5.0+13587+92118e57                   rhel-8-for-x86_64-appstream-rpms                 176 k
 ipa-server                              x86_64                 4.9.6-10.module+el8.5.0+13587+92118e57                   rhel-8-for-x86_64-appstream-rpms                 530 k
 ipa-server-common                       noarch                 4.9.6-10.module+el8.5.0+13587+92118e57                   rhel-8-for-x86_64-appstream-rpms                 612 k
 ipa-server-dns                          noarch                 4.9.6-10.module+el8.5.0+13587+92118e57                   rhel-8-for-x86_64-appstream-rpms                 192 k
 libwbclient                             x86_64                 4.14.5-7.el8_5                                           rhel-8-for-x86_64-baseos-rpms                    121 k
 python3-ipaclient                       noarch                 4.9.6-10.module+el8.5.0+13587+92118e57                   rhel-8-for-x86_64-appstream-rpms                 688 k
 python3-ipalib                          noarch                 4.9.6-10.module+el8.5.0+13587+92118e57                   rhel-8-for-x86_64-appstream-rpms                 756 k
 python3-ipaserver                       noarch                 4.9.6-10.module+el8.5.0+13587+92118e57                   rhel-8-for-x86_64-appstream-rpms                 1.6 M
 samba-client-libs                       x86_64                 4.14.5-7.el8_5                                           rhel-8-for-x86_64-baseos-rpms                    5.4 M
 samba-common                            noarch                 4.14.5-7.el8_5                                           rhel-8-for-x86_64-baseos-rpms                    221 k
 samba-common-libs                       x86_64                 4.14.5-7.el8_5                                           rhel-8-for-x86_64-baseos-rpms                    174 k
 selinux-policy                          noarch                 3.14.3-80.el8_5.2                                        rhel-8-for-x86_64-baseos-rpms                    636 k
 selinux-policy-targeted                 noarch                 3.14.3-80.el8_5.2                                        rhel-8-for-x86_64-baseos-rpms                     15 M
 systemd                                 x86_64                 239-51.el8_5.3                                           rhel-8-for-x86_64-baseos-rpms                    3.6 M
 systemd-libs                            x86_64                 239-51.el8_5.3                                           rhel-8-for-x86_64-baseos-rpms                    1.1 M
 systemd-pam                             x86_64                 239-51.el8_5.3                                           rhel-8-for-x86_64-baseos-rpms                    477 k
 systemd-udev                            x86_64                 239-51.el8_5.3                                           rhel-8-for-x86_64-baseos-rpms                    1.6 M

Thanks.

As I said, IPA update is not a reason for these failures. It looks like it is part of PKI internal upgrade code.

You can mitigate this issue by replacing 'requiredSecret' with 'secret' in all connectors and make sure all of them are using the same value as in ipa-pki-proxy.conf.

Removing requiredSecret did the trick, i just did a quick "ipa cert-show 1" and the cert are listed again. Do you know if i have to report this somewhere else to get the pki packages fixed or is this already handled with this report?

Alexander, you really saved my last day before holiday, thank you very much!

This is the bug against pki-core so it will be handled here.

Enjoy your holidays.

Searching for "Unable to communicate with CMS" sent me to a Red Hat solution, which didn't resolve the problem.  Removing requiredSecret as outlined above did fix the problem for me.  Should the solution be updated?

https://access.redhat.com/solutions/4796941

the HTTP error code 403 is kind of generic and can happen in various situations.

for this issue, we need this article: https://access.redhat.com/solutions/6632811

*** Bug 2041399 has been marked as a duplicate of this bug. ***

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (pki-core:10.6 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0357

This issue came back with latest pki update from this week, FreeIPA again show CMS error 403 in the certificate area.

   Upgrade  pki-base-10.11.2-5.module+el8.5.0+14437+bc030dcc.noarch         @rhel-8-for-x86_64-appstream-rpms
    Upgraded pki-base-10.11.2-4.module+el8.5.0+13827+5b1d191d.noarch         @@System
    Upgrade  pki-server-10.11.2-5.module+el8.5.0+14437+bc030dcc.noarch       @rhel-8-for-x86_64-appstream-rpms
    Upgraded pki-server-10.11.2-4.module+el8.5.0+13827+5b1d191d.noarch       @@System
    Upgrade  pki-acme-10.11.2-5.module+el8.5.0+14437+bc030dcc.noarch         @rhel-8-for-x86_64-appstream-rpms
    Upgraded pki-acme-10.11.2-4.module+el8.5.0+13827+5b1d191d.noarch         @@System
    Upgrade  python3-pki-10.11.2-5.module+el8.5.0+14437+bc030dcc.noarch      @rhel-8-for-x86_64-appstream-rpms
    Upgraded python3-pki-10.11.2-4.module+el8.5.0+13827+5b1d191d.noarch      @@System
    Upgrade  pki-base-java-10.11.2-5.module+el8.5.0+14437+bc030dcc.noarch    @rhel-8-for-x86_64-appstream-rpms
    Upgraded pki-base-java-10.11.2-4.module+el8.5.0+13827+5b1d191d.noarch    @@System
    Upgrade  pki-kra-10.11.2-5.module+el8.5.0+14437+bc030dcc.noarch          @rhel-8-for-x86_64-appstream-rpms
    Upgraded pki-kra-10.11.2-4.module+el8.5.0+13827+5b1d191d.noarch          @@System
    Upgrade  pki-symkey-10.11.2-5.module+el8.5.0+14437+bc030dcc.x86_64       @rhel-8-for-x86_64-appstream-rpms
    Upgraded pki-symkey-10.11.2-4.module+el8.5.0+13827+5b1d191d.x86_64       @@System
    Upgrade  pki-tools-10.11.2-5.module+el8.5.0+14437+bc030dcc.x86_64        @rhel-8-for-x86_64-appstream-rpms
    Upgraded pki-tools-10.11.2-4.module+el8.5.0+13827+5b1d191d.x86_64        @@System
    Upgrade  pki-ca-10.11.2-5.module+el8.5.0+14437+bc030dcc.noarch           @rhel-8-for-x86_64-appstream-rpms
    Upgraded pki-ca-10.11.2-4.module+el8.5.0+13827+5b1d191d.noarch           @@System

Little update: i solved the issue by applying the previous workaround but this time a new problem came up.

Again this update changed the connector settings in "/etc/pki/pki-tomcat/server.xml" from "secret=" to "requiredsecret=", but this time it also changed the secret key itself. So setting the connector back from "requiredSecret=" to "secret=" solve "CMS Error 403" but now i got something with Error 50x, then i compared the key from "/etc/httpd/conf.d/ipa-pki-proxy.conf" and found that is so longer the same, so i took over the key (like it was before the update) and now after a restart the IPA certificate section is working again.


pre update:
/etc/pki/pki-tomcat/server.xml:    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="localhost4" name="Connector1" secret="Oldkey"/>
/etc/pki/pki-tomcat/server.xml:    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="localhost6" name="Connector2" secret="Oldkey"/>

after update:
/etc/pki/pki-tomcat/server.xml:    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="localhost4" name="Connector1" requiredSecret="Newkey"/>
/etc/pki/pki-tomcat/server.xml:    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="localhost6" name="Connector2" requiredSecret="NewKey"/>

my fix:
/etc/pki/pki-tomcat/server.xml:    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="localhost4" name="Connector1" secret="Oldkey"/>
/etc/pki/pki-tomcat/server.xml:    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="localhost6" name="Connector2" secret="Oldkey"/>

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days