Why would PKI change a secret value? Changing the attribute ok, but changing the value? This will affect any AJP user so is not IPA-specific.

Hi Rob!

Thank I don't know. I've got two cases with similar symptoms attached - both configs have correct parameter names, but don't have matching secrets. Happened after update to 8.5 and pki-server-10.11.2-5.

Best regards,
Aleksandr Sharov
Red Hat EMEA

Endi, same question from comment 1

Rob, to my understanding this is how it works:

During installation pkispawn generates a random value for AJP
connector's secret:
https://github.com/dogtagpki/pki/blob/master/base/server/python/pki/server/deployment/pkiparser.py#L290

Then pkispawn stores the value in the "secret" attribute in server.xml:
https://github.com/dogtagpki/pki/blob/master/base/server/etc/default.cfg#L266
https://github.com/dogtagpki/pki/blob/master/base/server/python/pki/server/deployment/pkiparser.py#L717-L718
https://github.com/dogtagpki/pki/blob/master/base/server/config/pkislots.cfg#L10
https://github.com/dogtagpki/pki/blob/master/base/tomcat-9.0/conf/server.xml#L149
https://github.com/dogtagpki/pki/blob/master/base/tomcat-9.0/conf/server.xml#L154

Every time the server is started, the startup script will update the
connector according to the Tomcat version:
https://github.com/dogtagpki/pki/blob/master/base/server/python/pki/server/instance.py#L962-L965

If it's an older Tomcat, the "secret" will be removed and the
"requiredSecret" will be added with the same value. However,
if the connector already has a "requiredSecret", its existing
value will not be overwritten:
https://github.com/dogtagpki/pki/blob/master/base/server/python/pki/server/instance.py#L932-L953

Similarly, if it's a newer Tomcat, the "requiredSecret" will be
removed and the "secret" will be added with the same value. However,
if the connector already has a "secret", its existing value will not
be overwritten:
https://github.com/dogtagpki/pki/blob/master/base/server/python/pki/server/instance.py#L878-L899

So if for some reason the connector has both attributes with different
secrets, one of the attributes will be dropped and it will look like
the secret has changed but as explained above PKI doesn't actually
change it.

Since ipa-pki-proxy.conf is owned by IPA, PKI cannot read that file
to determine which secret to use and then update the server.xml, nor
it can update ipa-pki-proxy.conf with the secret from server.xml.

I think there are 2 ways to handle this:

1. Let PKI generate the secret in server.xml, then every time IPA
restarts HTTPD it needs to read the secret from server.xml and update
ipa-pki-proxy.conf. IPA should not assume that the secret will be
constant, and IPA should not modify the connector in server.xml.

2. Let IPA generate the secret for both server.xml and ipa-pki-proxy.conf.
PKI should not change the secret in server.xml, but PKI still can make
other modifications to server.xml (e.g. for upgrades).

I prefer option #1, but either way the changes need to be made on IPA side.

IPA has similar detection https://github.com/freeipa/freeipa/blob/master/ipaserver/install/dogtaginstance.py#L366

Pre 9.0.31.0 uses requiredSecret. After uses secret.

The code loops through the connectors looking for existing secrets and if found, it maintains them, otherwise a new one is generated. So maybe there is some conflict here because PKI just does a string replace.

If it determines that something has changed then server.xml and ipa-pki-proxy.conf are updated.

I haven't been able to duplicate the problem purely using the IPA upgrade methods but I also don't have a clear view of what server.xml looks like prior to the updates (and sosreport does not capture this file).

How can I trigger the pki upgrade script that would do the replacement?

You just need to restart PKI server, it will execute some commands
to update the configuration before actually running the server:
https://github.com/dogtagpki/pki/blob/master/base/server/share/lib/systemd/system/pki-tomcatd%40.service

What we lack is are reproducible steps. It has certainly affected a number of people but I've yet to discover how.

A vanilla 8.5.0 server results in a server.xml with both secrete and requiredSecret values:

    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="127.0.0.1" secret="1HxvHuR5XAJ6Yjp33IpBqG2ZAKIdXcUEzLJOMZlPBbPu" name="Connector1" requiredSecret="0FqBtwc1hL6T6dxiIlQSW1JLY25A93Dt61BQIOfXB1Cp"/>
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="::1" secret="1HxvHuR5XAJ6Yjp33IpBqG2ZAKIdXcUEzLJOMZlPBbPu" name="Connector2" requiredSecret="0FqBtwc1hL6T6dxiIlQSW1JLY25A93Dt61BQIOfXB1Cp"/>

ipa-pki-proxy.conf is using the requiredSecret value.

This incorrect adding of secret was fixed in https://bugzilla.redhat.com/show_bug.cgi?id=2006070 but older installs may still have both.

IPA and the CA work fine in this scenario because the AJP secrets match. What makes them not match I don't know.

IPA will only do the secret swap if tomcat is > 9.0.30.0. RHEL 8 only has 9.0.30.0 AFAIK. The only way I've been able to reproduce the failure is to trick IPA into thinking that tomcat is newer. But that drops the requiredSecret value and it isn't what we've seen, which is both values but they both differ from the IPA proxy.

Unfortunately server.xml is not included in sosreports so I only know that the secrets are different but not when they were generated or by what.

The most sensible fix is to try to catch this condition in the IPA upgrade code and just fix it. This is just additional fallout from 2006070.

Cloned to https://pagure.io/freeipa/issue/9176

Fixed upstream
ipa-4-9:
https://pagure.io/freeipa/c/deaaaaf1492410269c1f66f8d4bb57e41b99d87c

master:
https://pagure.io/freeipa/c/7572174175415a113a95c1094503bb1102257610

version:
ipa-server-4.9.10-4.module+el8.7.0+15926+daa9f08b.x86_64
ipa-server-trust-ad-4.9.10-4.module+el8.7.0+15926+daa9f08b.x86_64
ipa-server-dns-4.9.10-4.module+el8.7.0+15926+daa9f08b.noarch


Steps:
1. Add some a secret in AJP connector in /etc/pki/pki-tomcat/server.xml
2. ipa-server-upgrade
3. check if added secret in step 1 removed
4. check that the secret in  /etc/pki/pki-tomcat/server.xml matches with secret in /etc/httpd/conf.d/ipa-pki-proxy.conf


[root@master ~]# grep requiredSecret /etc/pki/pki-tomcat/server.xml
[root@master ~]# 
[root@master ~]# grep secret /etc/pki/pki-tomcat/server.xml
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="127.0.0.1" name="Connector1" secret="1HuZ2o436r0ncEinYTWS4xJbf5XIWstdmWmpZAsHXh72"/>
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="::1" name="Connector2" secret="1HuZ2o436r0ncEinYTWS4xJbf5XIWstdmWmpZAsHXh72"/>


after change:

[root@master ~]# grep secret /etc/pki/pki-tomcat/server.xml
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="127.0.0.1" name="Connector1" secret="1HuZ2o436r0ncEinYTWS4xJbf5XIWstdmWmpZAsHXh72" requiredSecret="SomeSecret"/>
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="::1" name="Connector2" secret="1HuZ2o436r0ncEinYTWS4xJbf5XIWstdmWmpZAsHXh72" requiredSecret="SomeSecret"/>
[root@master ~]# 
[root@master ~]# 
[root@master ~]# ipa-server-ip
-bash: ipa-server-ip: command not found
[root@master ~]# ipa-server-upgrade 
Upgrading IPA:. Estimated time: 1 minute 30 seconds
  [1/11]: stopping directory server
  [2/11]: saving configuration
  [3/11]: disabling listeners
  [4/11]: enabling DS global lock
  [5/11]: disabling Schema Compat
  [6/11]: starting directory server
  [7/11]: updating schema
  [8/11]: upgrading server
  [9/11]: stopping directory server
  [10/11]: restoring configuration
  [11/11]: starting directory server
Done.
Update complete
Upgrading IPA services
Upgrading the configuration of the IPA services
Disabled p11-kit-proxy
[Verifying that root certificate is published]
[Migrate CRL publish directory]
Publish directory already set to new location
Forcing update of template /usr/share/ipa/ipa-pki-proxy.conf.template
Upgraded /etc/httpd/conf.d/ipa-pki-proxy.conf to version 17
[Verifying that KDC configuration is using ipa-kdb backend]
[Fix DS schema file syntax]
[Removing RA cert from DS NSS database]
[Enable sidgen and extdom plugins by default]
[Updating HTTPD service IPA configuration]
[Updating HTTPD service IPA WSGI configuration]
Nothing to do for configure_httpd_wsgi_conf
[Migrating from mod_nss to mod_ssl]
Already migrated to mod_ssl
[Moving HTTPD service keytab to gssproxy]
[Removing self-signed CA]
[Removing Dogtag 9 CA]
[Checking for deprecated KDC configuration files]
[Checking for deprecated backups of Samba configuration files]
dnssec-validation yes
[Add missing CA DNS records]
Updating DNS system records
named user config '/etc/named/ipa-ext.conf' already exists
named user config '/etc/named/ipa-options-ext.conf' already exists
named user config '/etc/named/ipa-logging-ext.conf' already exists
[Upgrading CA schema]
CA schema update complete
[Update certmonger certificate renewal configuration]
Certmonger certificate renewal configuration already up-to-date
[Enable PKIX certificate path discovery and validation]
[Authorizing RA Agent to modify profiles]
[Authorizing RA Agent to manage lightweight CAs]
[Ensuring Lightweight CAs container exists in Dogtag database]
[Adding default OCSP URI configuration]
[Disabling cert publishing]
pki-tomcat configuration changed, restart pki-tomcat
[Ensuring CA is using LDAPProfileSubsystem]
[Migrating certificate profiles to LDAP]
[Ensuring presence of included profiles]
[Add default CA ACL]
[Updating ACME configuration]
[Migrating to authselect profile]
[Create systemd-user hbac service and rule]
hbac service systemd-user already exists
[Add root alias to admin account]
Alias already exists
[Setup SPAKE]
[Setup PKINIT]
[Enable server krb5.conf snippet]
[Setup kpasswd_server]
[Adding ipa-ca alias to HTTP certificate]
Certificate is OK; nothing to do
The IPA services were upgraded
The ipa-server-upgrade command was successful
[root@master ~]# 
[root@master ~]# 
[root@master ~]# grep secret /etc/pki/pki-tomcat/server.xml
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="127.0.0.1" name="Connector1" secret="1HuZ2o436r0ncEinYTWS4xJbf5XIWstdmWmpZAsHXh72"/>
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="::1" name="Connector2" secret="1HuZ2o436r0ncEinYTWS4xJbf5XIWstdmWmpZAsHXh72"/>
[root@master ~]# 
[root@master ~]# 
[root@master ~]# grep secret /etc/httpd/conf.d/ipa-pki-proxy.conf
    ProxyPassMatch ajp://localhost:8009 secret=1HuZ2o436r0ncEinYTWS4xJbf5XIWstdmWmpZAsHXh72
    ProxyPassMatch ajp://localhost:8009 secret=1HuZ2o436r0ncEinYTWS4xJbf5XIWstdmWmpZAsHXh72
    ProxyPassMatch ajp://localhost:8009 secret=1HuZ2o436r0ncEinYTWS4xJbf5XIWstdmWmpZAsHXh72
    ProxyPassMatch ajp://localhost:8009 secret=1HuZ2o436r0ncEinYTWS4xJbf5XIWstdmWmpZAsHXh72
[root@master ~]# 


Hence based on the above observation, marking the bug as verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (idm:client and idm:DL1 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:7540