Bug 1999142

This seems to be a regression related to the fix for https://pagure.io/freeipa/issue/8738 / https://bugzilla.redhat.com/show_bug.cgi?id=1934991

When a replica is installed, the installer now tries to add missing profiles to the 1st server. With 8.4, it tries to add the acmeIPAServerCert profile which contains the following definition:
[...]
policyset.serverCertSet.9.default.class_id=sanToCNDefaultImpl
[...]

On a 7.9 server, this sanToCNDefaultImpl does not exist, and the profile import fails. We can see in the master ca debug log:
[30/Aug/2021:07:37:35][http-bio-8443-exec-15]: BasicProfile: createProfilePolicy:  Cannot find sanToCNDefaultImpl

and in the replica:
2021-08-30T11:37:35Z DEBUG response body (decoded): b'{"Attributes":{"Attribute":[]},"ClassName":"com.netscape.certsrv.base.BadRequestException","Code":400,"Message":"Invalid profile data"}'
2021-08-30T11:37:35Z DEBUG Error migrating 'acmeIPAServerCert': Request failed with status 400: Non-2xx response from CA REST API: 400. Invalid profile data

When the profile import fails, the replica assumes that the profile already exists and tries to disable it, then update it. Disable fails because the profile does not exist on the master:
(master ca debug log):
[30/Aug/2021:07:37:35][http-bio-8443-exec-16]: Trying to modify profile: acmeIPAServerCert.  Profile not found.

and on the replica:
2021-08-30T11:37:35Z DEBUG request POST https://gizmo.idmqe.lab.eng.bos.redhat.com:8443/ca/rest/profiles/acmeIPAServerCert?action=disable
2021-08-30T11:37:35Z DEBUG request body ''
2021-08-30T11:37:35Z DEBUG response status 500

Update also fails:
(master ca debug log):
[30/Aug/2021:07:37:35][http-bio-8443-exec-16]: Trying to modify profile: acmeIPAServerCert.  Profile not found.

(replica):
2021-08-30T11:37:36Z DEBUG The ipa-replica-install command failed, exception: HTTPRequestError: Request failed with status 500: Non-2xx response from CA REST API: 500.
2021-08-30T11:37:36Z ERROR Request failed with status 500: Non-2xx response from CA REST API: 500.
2021-08-30T11:37:36Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

Flo's analysis is correct. sanToCNDefaultImpl is only available in newer dogtag and is code, not configuration. The installer communicates with the existing CA and LDAP so is limited to the features that they provide.

When it fails it returns a 500 error which isn't the best thing to catch and ignore:

Request failed with status 500: Non-2xx response from CA REST API: 500.

The most straightforward fix is to catch this and continue. It would be nicer if dogtag returned a 400 Bad Request instead but it is what it is.

ACME will not work after this but if the LDAP profile is deleted and ipa-server-upgrade is run this will fix it on the 8.5 server:

# ipa-acme-manage enable
# ldapdelete -x -D 'cn=directory manager' -W cn=acmeIPAServerCert,cn=certprofiles,cn=ca,dc=example,dc=test
# ipa-server-upgrade

The removal of the profile will allow the upgrade to happen against a RHEL 8 server which has the sanToCNDefaultImpl so the profile will be added correctly.

The workaround of pre-creating the ACME profile prior to installing the first EL8 instance is successful.

Instructions when starting from scratch. If there is already an EL7 deployment then the install step can be skipped.

On the EL7 server:

# ipa-server-install <options>

Add the temporary ACME profile before installing IdM on EL8

# cat acme.ldif 
dn: cn=acmeIPAServerCert,cn=certprofiles,cn=ca,dc=example,dc=test
objectClass: ipacertprofile
objectClass: top
cn: acmeIPAServerCert
description: Temporary ACME IPA service certificate profile
ipaCertProfileStoreIssued: FALSE

# ldapadd -x -D 'cn=directory manager' -W -f acme.ldif 
Enter LDAP Password: 
adding new entry "cn=acmeIPAServerCert,cn=certprofiles,cn=ca,dc=example,dc=test"

On the EL8 server:

# ipa-client-install <options>
# ipa-replica-install --setup-ca

Delete the temporary profile:

# ldapdelete -x -D 'cn=directory manager' -w password cn=acmeIPAServerCert,cn=certprofiles,cn=ca,dc=example,dc=test

Re-run upgrade which will add the missing profile:

# ipa-server-upgrade 

Remove the EL7 server from the ipa-ca A record (because it can't answer ACME requests):

# kinit admin
# ipa dnsrecord-del example.test. ipa-ca
No option to delete specific record provided.
Delete all? Yes/No (default No): 
Current DNS record contents:

A record: 192.168.122.141, 192.168.122.123

Delete A record '192.168.122.141'? Yes/No (default No): y
Delete A record '192.168.122.123'? Yes/No (default No): 
  Record name: ipa-ca
  A record: 192.168.122.123

On EL8 client install certbot:
# dnf -y install certbot --enablerepo=epel

Get an ACME certificate
# certbot --server https://ipa-ca.example.test/acme/directory register -m nobody --agree-tos --no-eff-email

# certbot --server https://ipa-ca.example.test/acme/directory certonly --domain `hostname` --standalone

Upstream ticket:
https://pagure.io/freeipa/issue/8974

Upstream PR https://github.com/freeipa/freeipa/pull/5995

This supersedes comment #9.

This change is: when adding a profile fails, the LDAP operation will be rolled back as well and a warning displayed. This will allow the profile to be added at a later time.

Running ipa-server-upgrade once the installation is finished will:

1. Re-issue the Apache web certificate so it has the appropriate SAN for ipa-ca
2. Add the acmeIPAServerCert profile to the local pki so that ACME certificates can be issued

To verify this:

1. install 7.9 server
2. install 8.5 replica (with this change)
3. run ipa-server-upgrade on the replica once the replica install is complete
4. ipa-acme-manage enable on replica
5. remove the 7.9 server from the ipa-ca A record
6. install a client and install certbot from epel
7. register certbot on the client: certbot --server https://ipa-ca.example.test/acme/directory register -m nobody --agree-tos --no-eff-email
8. request a cert on the client: certbot --server https://ipa-ca.example.test/acme/directory certonly --domain `hostname` --standalone

Fixed upstream
master:
https://pagure.io/freeipa/c/4414d50d2a5452477eea40bec58266b19f43065e

Fixed upstream
ipa-4-9:
https://pagure.io/freeipa/c/a6e708ab4006d6623c37de1692de5362fcdb5dd6

Verified using RHEL8.5 and RHEL7.9 machine with latest nightly compose 

7.9: ipa-server-4.6.8-5.el7_9.7.x86_64
8.5: ipa-server-4.9.6-6.module+el8.5.0+12660+88e16a2c.x86_64

7.9: ipa-server-install --domain ...--realm ...
(success)

8.5:  ipa-client-install ... --server ...

Client configuration complete.
The ipa-client-install command was successful

8.5 ipa-replica-install ...
...
The ipa-replica-install command was successful

Therefore marking as verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ipa bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4230