Bug 2186710

Summary: certmonger fails to renew certificates in the second "getcert resubmit" if it is too close to the first one
Product: Red Hat Enterprise Linux 8 Reporter: Quynh Anh Pham <qpham>
Component: certmongerAssignee: Rob Crittenden <rcritten>
Status: CLOSED INSUFFICIENT_DATA QA Contact: ipa-qe <ipa-qe>
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.6Flags: rcritten: needinfo? (qpham)
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-06-27 20:37:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Quynh Anh Pham 2023-04-14 07:38:03 UTC
Description of problem:
certmonger on RHEL 8.6 was configured to fetch the certificates from Windows AD CS and Windows NDES server. "getcert request" and first time "getcert resubmit" succeeded with "MONITORING" status and "issued" time of the cert updated after first time "getcert resubmit". However, the second time "getcert resubmit" seems to not do renew the cert and the "issued" time of the cert did not change. If wait for an amount of time and "getcert resubmit" again, then it worked and renewed the cert. it seems that certmonger did not send the second renewal "getcert resubmit" to the NDES server if it is too close to the first renewal.

Version-Release number of selected component (if applicable):
certmonger-0.79.13-5.el8.src.rpm

How reproducible:


Steps to Reproduce:

1. Enrol certificate using SCEP / NDES: 
Request Generation and Submission: "getcert request -c 'NDESENROL' -f "/etc/pki/tls/certs/test/test-1.crt" -k "/etc/pki/tls/private/test/test-1.key" -G "RSA" -g "4096" -L "40DXXXXXXXXXXX""

Responses:
a. Certmonger: "New signing request "20230413225051" added." 
b. Certmonger 'getcert list' to examine status of enrolled certificate: 
[root@idmhost ~]# getcert list
Number of certificates and requests being tracked: 1.
Request ID '20230413225051':
        status: MONITORING
        stuck: no
        key pair storage: type=FILE,location='/etc/pki/tls/private/test/test-1.key'
        certificate: type=FILE,location='/etc/pki/tls/certs/test/test-1.crt'
        signing request thumbprint (MD5): B08F6AE0 5946A808 0DC4190D DB8CB378
        signing request thumbprint (SHA1): 4BB03D80 3648952F 11F01CFB ECB69D10 12DB54C4
        CA: NDESENROL
        issuer: CN=IssuingCA,DC=EXAMPLE,DC=COM
        subject: CN=idmhost.example.com
        issued: 2023-04-13 22:40:36 UTC
        expires: 2023-04-14 22:40:36 UTC
        dns: idmhost.example.com
        principal name: host/idmhost.example.com
        key usage: digitalSignature,nonRepudiation,keyEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command:
        track: yes
        auto-renew: yes
c. Examine the backend Issuing CA to see Issued Certificate: "Certificate Serial Number '190000026a5a86156fb193a5ee00130000026a' issued.

2. Now we resubmit for renewal above requestid: "getcert resubmit -i '20230413225051'"
Response: "Resubmitting "20230413225051" to "NDESENROL"."

3. Examine result: "getcert list"
Response: 
[root@idmhost ~]# getcert list
Number of certificates and requests being tracked: 1.
Request ID '20230413225051':
        status: MONITORING
        stuck: no
        key pair storage: type=FILE,location='/etc/pki/tls/private/test/test-1.key'
        certificate: type=FILE,location='/etc/pki/tls/certs/test/test-1.crt'
        signing request thumbprint (MD5): 6EAAF885 EABE319F FE3D6778 A23647BC
        signing request thumbprint (SHA1): 07EFD693 F1FB8FEB 6DF66FBB E04F5879 6B97E989
        CA: NDESENROL
        issuer: CN=IssuingCA,DC=EXAMPLE,DC=COM
        subject: CN=idmhost.example.com
        issued: 2023-04-13 22:45:32 UTC
        expires: 2023-04-14 22:45:32 UTC
        dns: idmhost.example.com
        principal name: host/idmhost.example.com
        key usage: digitalSignature,nonRepudiation,keyEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command:
        track: yes
        auto-renew: yes
Observations:
Same 'requestid' retained.
New certificate issued by backend Issuing CA: Certificate Serial Number '190000026b821b1573111a206800130000026b' issued.

3. Restart certmonger service: "sudo systemctl restart certmonger"

4. Examine status of tracked certificate: "getcert list"
Response:
[root@idmhost ~]# getcert list
Number of certificates and requests being tracked: 1.
Request ID '20230413225051':
        status: MONITORING
        stuck: no
        key pair storage: type=FILE,location='/etc/pki/tls/private/test/test-1.key'
        certificate: type=FILE,location='/etc/pki/tls/certs/test/test-1.crt'
        signing request thumbprint (MD5): 6EAAF885 EABE319F FE3D6778 A23647BC
        signing request thumbprint (SHA1): 07EFD693 F1FB8FEB 6DF66FBB E04F5879 6B97E989
        CA: NDESENROL
        issuer: CN=IssuingCA,DC=EXAMPLE,DC=COM
        subject: CN=idmhost.example.com
        issued: 2023-04-13 22:45:32 UTC
        expires: 2023-04-14 22:45:32 UTC
        dns: idmhost.example.com
        principal name: host/idmhost.example.com
        key usage: digitalSignature,nonRepudiation,keyEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command:
        track: yes
        auto-renew: yes

Obs: All looks OK.

5. Now we attempt second renewal: "getcert resubmit -i '20230413225051'"
Response: Resubmitting "20230413225051" to "NDESENROL".

6. Examine status of tracked certificate: "getcert list"
Response:
Number of certificates and requests being tracked: 1.
Request ID '20230413225051':
        status: MONITORING
        stuck: no
        key pair storage: type=FILE,location='/etc/pki/tls/private/test/test-1.key'
        certificate: type=FILE,location='/etc/pki/tls/certs/test/test-1.crt'
        signing request thumbprint (MD5): 6EAAF885 EABE319F FE3D6778 A23647BC
        signing request thumbprint (SHA1): 07EFD693 F1FB8FEB 6DF66FBB E04F5879 6B97E989
        CA: NDESENROL
        issuer: CN=IssuingCA,DC=EXAMPLE,DC=COM
        subject: CN=idmhost.example.com
        issued: 2023-04-13 22:45:32 UTC
        expires: 2023-04-14 22:45:32 UTC
        dns: idmhost.example.com
        principal name: host/idmhost.example.com
        key usage: digitalSignature,nonRepudiation,keyEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command:
        track: yes
        auto-renew: yes

Obs: The certificate does not appear to have renewed and there was no error generated. 
Examining the backend Issuing CA I note that NO certificate was issued during the last resubmit attempt.

7. Restart certmonger service: "sudo systemctl restart certmonger"

8. Examine tracked certificate: "getcert list"
Response:
Number of certificates and requests being tracked: 1.
Request ID '20230413225051':
        status: MONITORING
        stuck: no
        key pair storage: type=FILE,location='/etc/pki/tls/private/test/test-1.key'
        certificate: type=FILE,location='/etc/pki/tls/certs/test/test-1.crt'
        signing request thumbprint (MD5): 6EAAF885 EABE319F FE3D6778 A23647BC
        signing request thumbprint (SHA1): 07EFD693 F1FB8FEB 6DF66FBB E04F5879 6B97E989
        CA: NDESENROL
        issuer: CN=IssuingCA,DC=EXAMPLE,DC=COM
        subject: CN=idmhost.example.com
        issued: 2023-04-13 22:45:32 UTC
        expires: 2023-04-14 22:45:32 UTC
        dns: idmhost.example.com
        principal name: host/idmhost.example.com
        key usage: digitalSignature,nonRepudiation,keyEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command:
        track: yes
        auto-renew: yes

Obs: Not seeing the lost certificate problem appear as I was encountering previously. The only thing I can think of is that the reboot of the server cleared whatever was causing that problem.
Still, only the initial certificate renewal succeeds. Subsequent renewals fail SILENTLY.

9. Examining Windows Application Event Log on our NDES server I note that there are no reported NDES errors.

10. Examining the local journal on the RHEL host: "journalctl -n 50 | grep certmonger"
Obs: It appears that certmonger thinks that a request has been submitted to the NDES server successfully?
Output of last few lines from journal:
"Apr 13 23:34:31 idmhost.example.com certmonger[6053]: SCEP status is "success".
Apr 13 23:34:31 idmhost.example.com certmonger[6061]: Certificate in file "/etc/pki/tls/certs/test/test-1.crt" issued by CA and saved.
[root@idmhost ~]# date
Thu Apr 13 23:34:49 UTC 2023"

Obs: Clearly, there was no submission to the NDES server and there was no 'new' artefacts to the key and crt dirs. 
I confirmed the request date / time by running the date command immediately after running the failed resubmit and you can see that the times concur.



Actual results:
Second "getcert resubmit" did not renew the cert if it is run too close to the first "getcert resubmit"

Expected results:
Second "getcert resubmit" should renew the cert

Additional info:
If wait an amount of time, the second "getcert resubmit" succeeded in renewing the cert again (as long as it is not too close to the first renewal)

Comment 1 Rob Crittenden 2023-04-14 13:05:52 UTC
Can you create the file /etc/sysconfig/certmonger with contents: OPTS=-d3

Then restart certmonger, reproduce this again, and provide the full system journal?