Bug 1934991

Summary: ACME fails to generate a cert on migrated RHEL8.4 server
Product: Red Hat Enterprise Linux 8 Reporter: Mohammad Rizwan <myusuf>
Component: ipaAssignee: Thomas Woerner <twoerner>
Status: CLOSED ERRATA QA Contact: ipa-qe <ipa-qe>
Severity: unspecified Docs Contact:
Priority: urgent    
Version: 8.4CC: frenaud, pcech, rcritten, ssidhaye, tscherf
Target Milestone: rcKeywords: Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ipa-4.9.5-1 Doc Type: Bug Fix
Doc Text:
Cause: Adding a new IPA server into an existing cluster does not install the new ACME profile preventing ACME from operating. Consequence: ACME is not available as a service Fix: The profile will be added on replica installation. Additional work is needed for ACME to interoperate in a mixed environment, including obtaining a new Apache server cert with ipa-ca.<domain> as a SAN and limiting the DNS A record to ACME-capable hosts. Result: Certificates can be issued over the ACME protocol.
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-09 18:22:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Mohammad Rizwan 2021-03-04 07:29:53 UTC
Description of problem:
When RHEL8.3 server migrated to RHEL8.4, ACME fails to generate the cert and throws a traceback in acme debug log.


Version-Release number of selected component (if applicable):
ipa-server 4.9.2 1.module+el8.4.0+9973+3d202164
pki-ca 10.10.5 1.module+el8.4.0+10167+ab954dab

How reproducible:
always

Steps to Reproduce:
1. Install RHEL8.3 master

2. Install replica on RHEL8.4 machine and migrate(make it ca renewal master and enable crl generation role on it). Remove master safely.

take inspiration from https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/installing_identity_management/index#migrating

3. install client against rhel8.4 replica (migrated master)

4. enable the acme role on rhel8.4 replica
$ ipa-acme-manage enable

5. install httpd and mod_md on client

6. set selinux boolean on client $ setsebool -P httpd_can_network_connect 1

7. request acme cert using mod_md on client
 
[root@client ~]# cat >/etc/httpd/conf.d/acme.conf <<EOF
LogLevel warn md:notice

MDCertificateAuthority https://ipa-ca.testrelm.test/acme/directory
MDCertificateAgreement accepted

MDomain client.testrelm.test

<VirtualHost *:443>
    ServerName client.testrelm.test

    SSLEngine on
    # no certificates specification
</VirtualHost>
EOF

[root@client ~]# systemctl restart httpd
[root@client ~]# systemctl reload httpd

8. Try accessing client from master

$ curl -v https://<client-hostname>

Actual results:
ACME certificate not issued. traceback in debug log (attached to bug)

"Unable to get enrollment template for acmeIPAServerCert: Profile not found"

Expected results:
No traceback and ACME cert issued

Additional info:

Comment 3 Rob Crittenden 2021-03-04 14:31:46 UTC
The 8.3 installation doesn't provide the ACME profile and when promoting a client to a server the code that would add the default profiles is skipped. The result is a server that is ACME-capable but lacks the required profile.

Comment 4 Rob Crittenden 2021-03-04 14:37:40 UTC
Upstream ticket:
https://pagure.io/freeipa/issue/8738

Comment 5 Mohammad Rizwan 2021-03-04 15:11:43 UTC
Rob,

Initial scenario was with rhel8.3 server, rhel8.4 replica and rhel8.4 client against replica.

The scenario you mentioned in comment#3 (promoting rhel8.4 client to replica) produces same traceback.

Comment 6 Rob Crittenden 2021-03-11 16:16:13 UTC
The impact of this is that when adding an 8.4.0 server to an existing IPA cluster ACME will be enabled but the ACME profile isn't created in the CA so all ACME requests will fail.

It doesn't affect new installs.

Comment 7 Rob Crittenden 2021-03-31 14:18:05 UTC
To load the missing profile manually one can run (replacing dc=example,dc=test as appropriate):

$ kinit admin
$ ldapadd -Y GSSAPI
dn: cn=acmeIPAServerCert,cn=certprofiles,cn=ca,dc=example,dc=test
objectClass: ipacertprofile
objectClass: top
cn: acmeIPAServerCert
description: ACME IPA service certificate profile
ipaCertProfileStoreIssued: FALSE
<blank line>
^D

On those servers with ACME available a new Subject Alternate Name needs to be added to the Apache web cert so it can answer as ipa-ca.<domain>

In this case the server is ipa.example.test in the example.test domain:

# ipa-getcert resubmit -f /var/lib/ipa/certs/httpd.crt -D ipa.example.test -D ipa-ca.example.test

ACME is an all or nothing service in that either all CA's provide or none do, and it requires a known name to connect to. The ipa-ca name in IPA is a shortcut to find an IPA CA primarily for CRL and OCSP. For ACME to work in a mixed environment where some servers are incapable of providing ACME then those servers need to be removed from the ipa-ca DNS A record otherwise any ACME requests directed at a non-ACME CA host will fail.

Comment 8 Florence Blanc-Renaud 2021-05-18 13:20:57 UTC
Fixed upstream
master:
https://pagure.io/freeipa/c/b01547da79c189bfb5b4b32e54b2ef2cb733741b

Comment 9 Florence Blanc-Renaud 2021-05-19 12:17:45 UTC
Fixed upstream
ipa-4-9:
https://pagure.io/freeipa/c/7239864be38f13b5d6968552ea565a8dfedcf0dd

Comment 17 Sumedh Sidhaye 2021-07-02 09:18:05 UTC
Builds used for verification:


ipa-server-4.9.5-1.module+el8.5.0+11410+91a33fe4.x86_64.rpm                                    
ipa-server-common-4.9.5-1.module+el8.5.0+11410+91a33fe4.noarch.rpm                             
ipa-server-dns-4.9.5-1.module+el8.5.0+11410+91a33fe4.noarch.rpm                                
ipa-server-trust-ad-4.9.5-1.module+el8.5.0+11410+91a33fe4.x86_64.rpm

Tests:

2021-07-02T08:51:55 collecting ... collected 4 items

2021-07-02T08:51:55 

2021-07-02T09:06:00 src/migration/acme/test_acme_migration.py::TestACME::test_migration PASSED [ 25%]

2021-07-02T09:07:58 src/migration/acme/test_acme_migration.py::TestACME::test_enable_acme_service PASSED [ 50%]

2021-07-02T09:08:19 src/migration/acme/test_acme_migration.py::TestACME::test_mod_md PASSED  [ 75%]

2021-07-02T09:08:26 src/migration/acme/test_acme_migration.py::TestACME::test_disable_acme_service PASSED [100%]

2021-07-02T09:08:26 

2021-07-02T09:08:26 - generated xml file: /home/jenkins/workspace/trigger-test-suite-tool/test-suite/junit.xml -

2021-07-02T09:08:26 - generated html file: file:///home/jenkins/workspace/trigger-test-suite-tool/test-suite/report.html -

2021-07-02T09:08:26 ========================== 4 passed in 990.92 seconds ==========================



Based on above observation marking Bugzilla verified

Attaching report.html for reference

Comment 20 errata-xmlrpc 2021-11-09 18:22:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ipa bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4230