Bug 2096922

+++ This bug was initially created as a clone of Bug #2092015 +++

Description of problem:
<Continuation of the issue>
https://bugzilla.redhat.com/show_bug.cgi?id=2029023 - this bug tracks the issue with attribute name for pki-tomcat (secret changed to sharedSecret),

https://bugzilla.redhat.com/show_bug.cgi?id=2061458 - this bug fixes the config if old secret is present in  /etc/pki/pki-tomcat/server.xml

Now, as we can see in 
https://github.com/dogtagpki/pki/commit/bbdb82268026821cd6a00edae09cc30079effd30#diff-d448de858fda07d3c3395089190c9259e97541e42204ef3bce21945e4d7d5e61R880

if the correct attributes are in place, their value is not checked. But since it's a new attribute on PKI side, it usually has got different value.

We discussed this issue with Endi Dewata, and, as ipa-pki-proxy.conf is in IPA's responsibility, it's probably a good idea to keep track of modifying it on IPA's team's side.

Version-Release number of selected component (if applicable):
Verified on 8.5,
ipa-server-4.9.6-12.module+el8.5.0+14525+2137cc8f.x86_64
pki-server-10.11.2-5.module+el8.5.0+14437+bc030dcc.noarch

How reproducible:
should every time when sharedSecret is added into /etc/pki/pki-tomcat/server.xml

Steps to Reproduce:
1.
2.
3.

Actual results:
 [wsgi:error] [pid 1884012:tid 139838145005312] [remote 172.22.154.43:34178] ipa: ERROR: ra.find(): Unable to communicate with CMS (403)


Expected results:
httpd can communicate with CA correctly

Additional info:

--- Additional comment from Rob Crittenden on 2022-05-31 14:51:10 UTC ---

Why would PKI change a secret value? Changing the attribute ok, but changing the value? This will affect any AJP user so is not IPA-specific.

--- Additional comment from Aleksandr Sharov on 2022-05-31 15:19:44 UTC ---

Hi Rob!

Thank I don't know. I've got two cases with similar symptoms attached - both configs have correct parameter names, but don't have matching secrets. Happened after update to 8.5 and pki-server-10.11.2-5.

Best regards,
Aleksandr Sharov
Red Hat EMEA

--- Additional comment from Rob Crittenden on 2022-05-31 15:41:27 UTC ---

Endi, same question from comment 1

--- Additional comment from Endi Sukma Dewata on 2022-06-08 15:44:49 UTC ---

Rob, to my understanding this is how it works:

During installation pkispawn generates a random value for AJP
connector's secret:
https://github.com/dogtagpki/pki/blob/master/base/server/python/pki/server/deployment/pkiparser.py#L290

Then pkispawn stores the value in the "secret" attribute in server.xml:
https://github.com/dogtagpki/pki/blob/master/base/server/etc/default.cfg#L266
https://github.com/dogtagpki/pki/blob/master/base/server/python/pki/server/deployment/pkiparser.py#L717-L718
https://github.com/dogtagpki/pki/blob/master/base/server/config/pkislots.cfg#L10
https://github.com/dogtagpki/pki/blob/master/base/tomcat-9.0/conf/server.xml#L149
https://github.com/dogtagpki/pki/blob/master/base/tomcat-9.0/conf/server.xml#L154

Every time the server is started, the startup script will update the
connector according to the Tomcat version:
https://github.com/dogtagpki/pki/blob/master/base/server/python/pki/server/instance.py#L962-L965

If it's an older Tomcat, the "secret" will be removed and the
"requiredSecret" will be added with the same value. However,
if the connector already has a "requiredSecret", its existing
value will not be overwritten:
https://github.com/dogtagpki/pki/blob/master/base/server/python/pki/server/instance.py#L932-L953

Similarly, if it's a newer Tomcat, the "requiredSecret" will be
removed and the "secret" will be added with the same value. However,
if the connector already has a "secret", its existing value will not
be overwritten:
https://github.com/dogtagpki/pki/blob/master/base/server/python/pki/server/instance.py#L878-L899

So if for some reason the connector has both attributes with different
secrets, one of the attributes will be dropped and it will look like
the secret has changed but as explained above PKI doesn't actually
change it.

Since ipa-pki-proxy.conf is owned by IPA, PKI cannot read that file
to determine which secret to use and then update the server.xml, nor
it can update ipa-pki-proxy.conf with the secret from server.xml.

I think there are 2 ways to handle this:

1. Let PKI generate the secret in server.xml, then every time IPA
restarts HTTPD it needs to read the secret from server.xml and update
ipa-pki-proxy.conf. IPA should not assume that the secret will be
constant, and IPA should not modify the connector in server.xml.

2. Let IPA generate the secret for both server.xml and ipa-pki-proxy.conf.
PKI should not change the secret in server.xml, but PKI still can make
other modifications to server.xml (e.g. for upgrades).

I prefer option #1, but either way the changes need to be made on IPA side.

--- Additional comment from Rob Crittenden on 2022-06-08 17:36:08 UTC ---

IPA has similar detection https://github.com/freeipa/freeipa/blob/master/ipaserver/install/dogtaginstance.py#L366

Pre 9.0.31.0 uses requiredSecret. After uses secret.

The code loops through the connectors looking for existing secrets and if found, it maintains them, otherwise a new one is generated. So maybe there is some conflict here because PKI just does a string replace.

If it determines that something has changed then server.xml and ipa-pki-proxy.conf are updated.

I haven't been able to duplicate the problem purely using the IPA upgrade methods but I also don't have a clear view of what server.xml looks like prior to the updates (and sosreport does not capture this file).

How can I trigger the pki upgrade script that would do the replacement?

--- Additional comment from Endi Sukma Dewata on 2022-06-08 18:19:48 UTC ---

You just need to restart PKI server, it will execute some commands
to update the configuration before actually running the server:
https://github.com/dogtagpki/pki/blob/master/base/server/share/lib/systemd/system/pki-tomcatd%40.service

--- Additional comment from Rob Crittenden on 2022-06-09 02:01:14 UTC ---

What we lack is are reproducible steps. It has certainly affected a number of people but I've yet to discover how.

A vanilla 8.5.0 server results in a server.xml with both secrete and requiredSecret values:

    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="127.0.0.1" secret="1HxvHuR5XAJ6Yjp33IpBqG2ZAKIdXcUEzLJOMZlPBbPu" name="Connector1" requiredSecret="0FqBtwc1hL6T6dxiIlQSW1JLY25A93Dt61BQIOfXB1Cp"/>
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="::1" secret="1HxvHuR5XAJ6Yjp33IpBqG2ZAKIdXcUEzLJOMZlPBbPu" name="Connector2" requiredSecret="0FqBtwc1hL6T6dxiIlQSW1JLY25A93Dt61BQIOfXB1Cp"/>

ipa-pki-proxy.conf is using the requiredSecret value.

This incorrect adding of secret was fixed in https://bugzilla.redhat.com/show_bug.cgi?id=2006070 but older installs may still have both.

IPA and the CA work fine in this scenario because the AJP secrets match. What makes them not match I don't know.

IPA will only do the secret swap if tomcat is > 9.0.30.0. RHEL 8 only has 9.0.30.0 AFAIK. The only way I've been able to reproduce the failure is to trick IPA into thinking that tomcat is newer. But that drops the requiredSecret value and it isn't what we've seen, which is both values but they both differ from the IPA proxy.

Unfortunately server.xml is not included in sosreports so I only know that the secrets are different but not when they were generated or by what.

The most sensible fix is to try to catch this condition in the IPA upgrade code and just fix it. This is just additional fallout from 2006070.

--- Additional comment from Rob Crittenden on 2022-06-09 13:34:55 UTC ---

Cloned to https://pagure.io/freeipa/issue/9176

--- Additional comment from Florence Blanc-Renaud on 2022-06-14 14:53:16 UTC ---

Fixed upstream
ipa-4-9:
https://pagure.io/freeipa/c/deaaaaf1492410269c1f66f8d4bb57e41b99d87c

master:
https://pagure.io/freeipa/c/7572174175415a113a95c1094503bb1102257610