The changes have been merged via PRs: 

https://github.com/dogtagpki/pki/pull/182
https://github.com/dogtagpki/pki/pull/199

Document is merged via PR:
https://github.com/dogtagpki/pki/pull/197

Verification Steps:
===================

There are 3 different scenarios to be tested. I'll add them as separate comments

Scenario 1 (IPA Specific environment -- Uses LDAPI):
---------------------------------------------------

# Setting up a fake env:

1. Do a basic `ipa-server-install`
2. Verify `pki -U https://<localhost>:8443 ca-cert-find` works
3. timedatectl set-ntp false && timedatectl set-time <set time to beyond cert expirty date>
4. ipactl restart # some error OR should be stuck when trying to restart PKI server

# Procedure:

https://github.com/dogtagpki/pki/blob/master/docs/admin/Offline_System_Certificate_Renewal.md#ipa-environment-uses-ldapi

Scenario 2 (PKI Specific environment) (Use LDAPS):
--------------------------------------------------

# Setting up a fake env:

1. Do a pkispawn CA
2. set date beyond cert expiry
3. Ensure a valid DS certificate is available

# procedure:

https://github.com/dogtagpki/pki/blob/master/docs/admin/Offline_System_Certificate_Renewal.md#standalone-pki-environment-uses-ldaps

# Shortcut procedure for setting up fake env:

Since this step requires configuring TLS on Directory Server, I usually install IPA as it configures everything for us.

1. Do a basic `ipa-server-install`. This would do a TLS configured Directory Server.

2. timedatectl set-ntp false && timedatectl set-time <set time to beyond cert expirty date>

3. Determine the $DS_SERIAL by `certutil -L -d /etc/dirsrv/slapd-REALM/ -n Server-Cert`

4. Determine the $IPA_RA_SERIAL by `keytool -printcert -file /var/lib/ipa/ra-agent.pem`

5. pki-server cert-fix --ldapi-socket /var/run/slapd-REALM.socket --agent-uid admin --extra-cert $DS_SERIAL --extra-cert $IPA_RA_SERIAL

This step just renews a the DS cert and IPA_RA cert as they need to be valid

6. pki-server cert-fix  --ldap-url <LDAP URL> --agent-uid admin # This would fix all the system certs

Scenario 3 (PKI Specific environment) (can use LDAP or LDAPS):
--------------------------------------------------------------

# Setting up a fake env:

1. Do a pkispawn CA
2. set date beyond cert expiry
3. restart PKI server

# procedure:

https://github.com/dogtagpki/pki/blob/master/docs/admin/Offline_System_Certificate_Renewal.md#manual-renewal-process

Since testing this tool requires significant efforts, please consider testing https://bugzilla.redhat.com/show_bug.cgi?id=1679480 together with this bug

Hi Dinesh,

I have tried the first scenario and based on that I have few observations.

Scenario 1 : IPA Specific

1. When renewing a regular certificate why we need to stop/start CA instance.I mean if we renew a non-system based certificate then also I see a pki-tomcat restart?
2. There is a place where we are resetting the password. below are the logs. Could you please have a look there.


INFO: Resetting password for uid=admin,ou=people,o=ipaca
SASL/EXTERNAL authentication started
SASL username

3. In pki-tomcat logs , we are seeing below logs. 

2019-07-10 04:19:12 [http-nio-8080-exec-1] WARNING: Failed to read product version String. /usr/share/pki/CS_SERVER_VERSION (No such file or directory)
java.io.FileNotFoundException: /usr/share/pki/CS_SERVER_VERSION (No such file or directory)

4. If we run : # pki-server cert-fix --ldapi-socket /var/run/slapd-EXAMPLE-COM.socket --agent-uid pkidbuser  --extra-cert 6 --debug 5

here we have put pkidbuser in place of admin so the command fails with :

INFO: Starting the instance
Job for pki-tomcatd failed because a timeout was exceeded.
See "systemctl status pki-tomcatd" and "journalctl -xe" for details.
INFO: Selftests enabled for subsystems: ca
INFO: Restoring previous LDAP configuration
ERROR: Command: systemctl start pki-tomcatd

Later if we run the right command, System goes in a state that it is not recovered. 

# pki-server cert-fix --ldapi-socket /var/run/slapd-EXAMPLE-COM.socket --agent-uid admin  --extra-cert 6


INFO: Starting the instance
Job for pki-tomcatd failed because a timeout was exceeded.
See "systemctl status pki-tomcatd" and "journalctl -xe" for details.
INFO: Selftests enabled for subsystems: ca
INFO: Restoring previous LDAP configuration
ERROR: Command: systemctl start pki-tomcatd

Hi Geetika,

For (1) and (2), as I explained over IRC, the CA restarts once when this tool changes from cert-based auth mechanism to password-based auth mechanism. This is done by updating CS.cfg. Hence a restart.

There are 2 password resets -- for admin and pkidbuser (optional)... And hence the another CA restart. The tool checks whether `internaldb` password is set in password.conf.. If available, uses it. If not, resets the password of pkidbuser.


For (3), I have been seeing that in my other instances too. It was not introduced by this tool.

For (4), I looked at the box you shared. The system was in a very bad state based on the following observations:
* password.conf had `internaldb` while it should not -- Probably the cert-fix tool was terminated/crashed midway
* pkidbuser password was probably reset twice and the password was lost when you ran with `--agent-uid pkidbuser`... 
* CS.cfg is corrupted.

In fact, after removing `internaldb` key from password.conf and setting the CS.cfg manually, i was able to run the cert-fix tool. It actually renewed all certs BUT failed at the last point where it restores CS.cfg values and tries to start the server. I wasn't sure on how to fix the CS.cfg

Geetika,

I took as a challenge to fix your box. Guess what? I was able to fix it!! :)

Note that I reverted the CS.cfg to CS.cfg.bak.20190709083459 from archives. -- this doesn't use a secure connection to LDAP. (Did you install with custom IPA config?)

I also added the internaldb=<the value you shared>  to the password.conf... the box works:

[root@pki1 ~]# pki -U https://localhost:8443 ca-cert-find
WARNING: BAD_CERT_DOMAIN encountered on 'CN=pki1.example.com,O=EXAMPLE.COM' indicates a common-name mismatch
WARNING: UNTRUSTED ISSUER encountered on 'CN=pki1.example.com,O=EXAMPLE.COM' indicates a non-trusted CA cert 'CN=Certificate Authority,O=EXAMPLE.COM'
Import CA certificate (Y/n)? n
----------------
54 entries found
----------------
  Serial Number: 0x1
  Subject DN: CN=Certificate Authority,O=EXAMPLE.COM
  Issuer DN: CN=Certificate Authority,O=EXAMPLE.COM
  Status: VALID
  Type: X.509 version 3
  Key Algorithm: PKCS #1 RSA with 3072-bit key
  Not Valid Before: Tue Jul 09 08:34:45 EDT 2019
  Not Valid After: Sat Jul 09 08:34:45 EDT 2039
  Issued On: Tue Jul 09 08:34:45 EDT 2019
  Issued By: system

<clipped>

[root@pki1 ~]# timedatectl
               Local time: Sun 2022-05-22 00:20:05 EDT
           Universal time: Sun 2022-05-22 04:20:05 UTC
                 RTC time: Sun 2022-05-22 00:20:05
                Time zone: America/New_York (EDT, -0400)
System clock synchronized: no
              NTP service: inactive
          RTC in local TZ: yes


I haven't changed the date back to the current settings. Let me know if your box is stable now :)

--Dinesh

Some further discussion of this.

First, some of the things there were questions about (especially: why do we reset certain account passwords?)
are explained in my blog post: https://frasertweedale.github.io/blog-redhat/posts/2019-03-18-cert-fix-redux.html.

We have agreed that using pkidbuser as the --agent-uid is a corner case that should be prevented
(i.e. have a sanity check that that account is -not- used).  It will be a small patch.  I see two
options:

1. Verify this BZ in spite of pkidbuser corner case, and open a new BZ to address the corner case
   (detect and prevent pkidbuser being used as --admin-uid).  Or;

2. Mark this BZ FailedQA, cut patch, await new build, etc.

I'm don't mind either way and I'll most likely cut the patch tomorrow (unless Dinesh wants to take it).

Dinesh, thanks for your analysis.  Geetika thanks for your solid QE work!

Thanks Fraser , Dinesh for helping me in this Bug testing.

Another query about usage of : --agent-uid <String>        UID of Dogtag agent user

Test Case 2: I have added a localuser which is not part of any group(admin/agent). So in that case if we use it as --agent-uid how it should behave.

dn: uid=localuser,ou=people,o=ipaca
objectClass: cmsuser
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
objectClass: top
cn: localuser
sn: localuser
usertype: happytest
mail: root@localhost
uid: localuser
userstate: 1

==========================================

Step 1 : Description says :       --agent-uid <String>        UID of Dogtag agent user

Run The same command with a non agent dogtag user

# pki-server cert-fix --ldapi-socket /var/run/slapd-EXAMPLE-COM.socket --agent-uid localuser  --extra-cert 6 --debug 5
INFO: Loading instance: pki-tomcat
INFO: Loading global Tomcat config: /etc/tomcat/tomcat.conf
INFO: Loading PKI Tomcat config: /usr/share/pki/etc/tomcat.conf
INFO: Loading instance Tomcat config: /etc/pki/pki-tomcat/tomcat.conf
INFO: Loading password config: /etc/pki/pki-tomcat/password.conf
INFO: Loading instance registry: /etc/sysconfig/pki/tomcat/pki-tomcat/pki-tomcat
INFO: Loading subsystem: ca
INFO: Loading subsystem config: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg
INFO: Fixing the following system certs: []
INFO: Renewing the following additional certs: ['6']
INFO: Stopping the instance to proceed with system cert renewal
DEBUG: Command: systemctl stop pki-tomcatd
INFO: Configuring LDAP password authentication
INFO: Selftests disabled for subsystems: ca
INFO: Password dm_pass_file
INFO: 
INFO: ['ldapsearch', '-H', 'ldapi://%2Fvar%2Frun%2Fslapd-EXAMPLE-COM.socket', '-Y', 'EXTERNAL', '-s', 'base', '-b', 'o=ipaca', '1.1']
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
INFO: Resetting password for uid=localuser,ou=people,o=ipaca
INFO: ['ldappasswd', '-H', 'ldapi://%2Fvar%2Frun%2Fslapd-EXAMPLE-COM.socket', '-Y', 'EXTERNAL', '-T', '/tmp/tmpatuzqa2n', 'uid=localuser,ou=people,o=ipaca']
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
INFO: Starting the instance
DEBUG: Command: systemctl start pki-tomcatd
INFO: Sleeping for 10 seconds to allow server time to start...
INFO: Requesting new cert for 6; writing to /etc/pki/pki-tomcat/certs/6-renewed.crt
INFO: Trying to setup a secure connection to CA subsystem.
DEBUG: Starting new HTTPS connection (1): pki1.example.com:8443
DEBUG: https://pki1.example.com:8443 "GET /ca/rest/account/login HTTP/1.1" 200 133
INFO: Secure connection with CA is established.
INFO: Placing cert creation request for serial: 6
DEBUG: https://pki1.example.com:8443 "GET /ca/rest/certrequests/profiles/caManualRenewal HTTP/1.1" 200 470
DEBUG: https://pki1.example.com:8443 "POST /ca/rest/certrequests HTTP/1.1" 200 318
DEBUG: https://pki1.example.com:8443 "GET /ca/rest/certrequests/58 HTTP/1.1" 200 284
DEBUG: https://pki1.example.com:8443 "GET /ca/rest/certs/0x3a HTTP/1.1" 200 None
INFO: Request ID: 58
INFO: Request Status: complete
DEBUG: request_data: {'CertRequestInfo': {'request_id': '58', 'request_type': 'renewal', 'request_status': 'complete', 'request_url': 'https://pki1.example.com:8443/ca/rest/certrequests/58'}}
DEBUG: cert_data: {'CertData': {'serial_number': '0x3a', 'subject_dn': 'CN=ipa-ca-agent,O=EXAMPLE.COM', 'status': 'VALID'}}
INFO: Serial Number: 0x3a
INFO: Issuer: CN=Certificate Authority,O=EXAMPLE.COM
INFO: Subject: CN=ipa-ca-agent,O=EXAMPLE.COM
DEBUG: Pretty Print:
DEBUG:     Certificate: 
        Data: 
            Version:  v3
            Serial Number: 0x3A
            Signature Algorithm: SHA256withRSA - 1.2.840.113549.1.1.11
            Issuer: CN=Certificate Authority,O=EXAMPLE.COM
            Validity: 
                Not Before: Sunday, May 22, 2022 11:42:29 AM EDT America/New_York
                Not  After: Monday, May 22, 2023 11:42:29 AM EDT America/New_York
            Subject: CN=ipa-ca-agent,O=EXAMPLE.COM
            Subject Public Key Info: 
                Algorithm: RSA - 1.2.840.113549.1.1.1
                Public Key: 
                    Exponent: 65537
                    Public Key Modulus: (2048 bits) :
                        D6:37:F9:2C:2F:A6:01:F0:EB:4E:5B:A0:04:6C:8A:F6:
                        CA:A8:E1:68:7C:A4:BC:28:48:8F:BD:56:CE:6D:8F:97:
                        85:4B:15:82:9B:F8:17:90:D2:6D:3B:B0:05:98:BC:EC:
                        EE:37:26:1A:F0:9F:D4:C6:69:A4:F7:61:5A:52:B4:C8:
                        01:78:A8:5F:01:EA:6D:4A:12:EC:89:56:4B:F0:01:9F:
                        00:AC:E3:55:DA:A2:2E:88:77:63:42:DE:1B:8E:23:62:
                        26:A5:75:A9:3C:95:72:AC:B4:4C:2D:48:33:61:8E:32:
                        A2:E7:A8:47:63:36:19:B1:CA:D5:44:84:41:96:23:64:
                        1C:65:83:64:38:28:5C:2E:9D:09:B7:80:41:25:F7:BD:
                        CB:9D:6A:82:87:47:A6:19:6E:BC:88:40:C7:5F:29:19:
                        4A:FF:26:0B:40:B9:14:EE:12:BB:F1:90:37:0D:10:D3:
                        3E:86:91:E5:0A:B4:89:D5:32:CA:82:61:D3:65:11:A4:
                        A6:98:3A:18:ED:FA:B6:C4:F5:F8:E1:71:66:2D:EE:20:
                        AC:23:AD:61:C1:E7:F7:58:71:81:CA:F6:14:0D:93:9B:
                        2C:A0:F2:66:5C:2E:5B:5B:4D:92:A6:21:65:95:D7:D4:
                        0A:4F:7C:B9:EE:12:B9:06:B1:40:AA:91:86:40:D5:4D
            Extensions: 
                Identifier: Authority Key Identifier - 2.5.29.35
                    Critical: no 
                    Key Identifier: 
                        53:64:5F:74:F8:0F:0D:07:7D:B3:09:AF:A4:FD:47:F2:
                        10:CA:1A:DA
                Identifier: Authority Info Access: - 1.3.6.1.5.5.7.1.1
                    Critical: no 
                    Access Description: 
                        Method #0: ocsp
                        Location #0: URIName: http://ipa-ca.example.com/ca/ocsp
                Identifier: Key Usage: - 2.5.29.15
                    Critical: yes 
                    Key Usage: 
                        Digital Signature 
                        Non Repudiation 
                        Key Encipherment 
                Identifier: Extended Key Usage: - 2.5.29.37
                    Critical: no 
                    Extended Key Usage: 
                        1.3.6.1.5.5.7.3.2
                        1.3.6.1.5.5.7.3.4
        Signature: 
            Algorithm: SHA256withRSA - 1.2.840.113549.1.1.11
            Signature: 
                9D:FA:06:49:32:B2:AC:E6:A3:96:A1:E5:DF:E0:C1:E9:
                D4:23:19:05:F4:17:1E:98:79:5A:9D:57:12:CB:3E:A9:
                A6:C8:CC:05:E3:2C:F1:7B:6D:56:29:3A:5F:B3:63:A4:
                34:37:75:BC:28:A6:1D:93:63:02:CA:A1:9D:80:E5:BD:
                5B:6F:C6:65:9E:B5:DD:A3:7A:0A:BA:5E:27:69:6A:82:
                5D:DE:AF:D4:42:14:1E:18:99:D8:E5:FC:59:A7:E4:04:
                85:7F:ED:9E:FE:12:65:CD:E3:27:56:C1:D3:AC:8B:AD:
                3A:B1:68:5B:6B:7D:C1:3D:BB:1A:D7:67:40:56:AD:85:
                9F:93:73:3D:5E:22:43:8C:A8:16:61:02:42:3F:72:CA:
                02:5A:D2:76:21:5C:2A:7D:F9:6F:9C:DE:E5:29:61:C7:
                6C:26:84:7A:21:E2:3E:37:45:EB:C2:B3:13:B7:0D:E4:
                89:58:24:E1:3F:19:B5:1A:A4:79:F2:41:95:BF:63:72:
                D8:26:11:89:3B:63:5E:93:88:B2:80:B8:EB:3D:F6:2D:
                9D:BC:13:C4:05:66:F9:7F:8E:38:08:06:18:86:61:60:
                82:80:48:CB:ED:E0:F7:7F:94:B4:BD:C5:0B:78:EE:47:
                05:5E:16:8A:B8:11:E5:03:F6:9A:91:37:B4:02:58:BE:
                D2:03:1C:7B:AF:E4:35:CB:27:67:6E:33:7C:82:54:20:
                95:F1:0F:65:4D:9A:3A:EE:D3:39:02:4B:17:10:EB:9F:
                25:56:29:6C:88:A2:F6:19:97:E0:3C:76:B7:AA:FE:95:
                2F:EB:9A:A0:65:97:4B:DE:90:9F:52:37:94:DF:08:04:
                2E:F4:BA:D9:46:EA:D5:EC:2C:62:DD:A4:6F:80:EF:81:
                72:94:1B:0A:FF:1F:9F:63:35:A8:AD:70:85:D0:A2:59:
                9F:51:DE:80:AA:38:E1:85:65:28:BA:08:EF:C9:95:35:
                59:34:42:8C:3C:71:36:2B:6E:CB:BE:06:BA:19:3D:F7
        FingerPrint
            MD2:
                15:4E:BA:2F:93:34:8C:25:4F:EE:BC:EE:37:15:94:66
            MD5:
                9E:A9:F8:2B:4F:10:11:51:B6:97:F6:1D:D2:C9:2C:69
            SHA-1:
                65:DD:29:69:86:F4:D1:C7:0D:78:14:1A:2F:8A:BF:ED:
                09:5A:C7:95
            SHA-256:
                99:84:5A:51:DA:4F:01:D4:2F:E0:3A:53:3B:0E:87:C3:
                E1:E4:EA:DF:E1:C7:60:CD:87:AB:9A:BB:16:02:12:9D
            SHA-512:
                A8:46:16:F3:1C:EE:99:3B:F4:BE:63:C5:30:D2:7B:9E:
                05:63:E0:43:A1:46:1D:25:46:06:C2:7C:79:25:DA:4A:
                C3:EC:02:AE:64:18:C6:33:B7:1E:69:1C:FE:6E:75:B5:
                BD:FF:B2:DD:21:80:73:D1:B4:33:F2:81:22:83:7C:A9

DEBUG: https://pki1.example.com:8443 "GET /ca/rest/certs/0x3a HTTP/1.1" 200 None
INFO: New cert is available at: /etc/pki/pki-tomcat/certs/6-renewed.crt
INFO: Stopping the instance
DEBUG: Command: systemctl stop pki-tomcatd
INFO: Selftests enabled for subsystems: ca
INFO: Restoring previous LDAP configuration
INFO: Starting the instance with renewed certs
DEBUG: Command: systemctl start pki-tomcatd
[root@pki1 ~]#

Raised : https://bugzilla.redhat.com/show_bug.cgi?id=1729215 based on https://bugzilla.redhat.com/show_bug.cgi?id=1696849#c12

@Geetika and @Fraser,

As per discussion with cfu on IRC, the system is in an intermediate state and we can make an exception:

Snip of the chat for reference:

<cfu> gkapoor, dmoluguw , my point is, if the system is in "preparation mode" (or whatever you call that, it's not yet up and running fully), then why are we worrying about these things?  Can other people reach the CA at this time?
<cfu> gkapoor, the thing is I do not see any sign of cert being issued from the debug log.. in fact, it looks like it just got started and nothing has happened yet
<cfu> gkapoor, is there auditing yet?
<gkapoor__> cfu checking 
<gkapoor__> cfu nopes no traces in logs 
<cfu> gkapoor, yeah, so I'll say it's an exception.  As far as CC goes, this should be treated like "installation time" when things are still being set up
<cfu> gkapoor, dmoluguw, so I'm gonna say make a note and leave it
<gkapoor__> cfu hmm...but in that case it is not working as expected
<cfu> gkapoor, I'm saying maybe you need to make an exception and lower your expectation ;-)
<cfu> gkapoor, dmoluguw , when a system is being set up, such as changing out its system certs, it's considered the same as "installation time" which CC allows for exception.  During installation, no auditing is expected, admins are allowed to change out certs, etc.
<cfu> gkapoor, dmoluguw , although one might want to enable firewall and only allow local connections during such time



Also, I have fixed added a small patch for #1729215 and I have added both of you as reviewers. Thank you Geetika for the stress testing and Fraser for the details! :)

based on https://bugzilla.redhat.com/show_bug.cgi?id=1696849#c13 it means that --agent-uid has no relevance because there is no check for group id and it works for any user which is in LDAP.

It might depend on whether it's a "renewal request" or a fresh enrollment.  I can't recall of top of head which we use in cert-fix.  And it is Friday evening so I will not look until next week :)

just to track and we don't miss on checking comment#16 and #17, I am putting this bug on Needinfo from Fraser.

Dinesh, while doing disable selftest command , i have below observations.

Test Case 2: Manual renewal Process 
====================================

1. Disable selftest.


# pki-server -v --debug selftest 
Command: selftest
Module: selftest
Commands:
 selftest-enable               Enable selftests.             
 selftest-disable              Disable selftests. 

# pki-server -v --debug selftest-disable -i pki-RootCA-CMC3 
Command: selftest-disable -i pki-RootCA-CMC3
Module: selftest
[root@pki1 ~]# pki-server -v --debug selftest-enable -i pki-RootCA-CMC3 
Command: selftest-enable -i pki-RootCA-CMC3
Module: selftest


Question 1: I don't see anything in debug logs for selftest enable/disable. Should it get reported/audited somewhere that selftest is disabled?
Generally doing any operation with selftest needs "CertUserDBAuthentication". which we are not doing while running # pki-server selftest-disable

Geetika,

We neither log anything to debug log nor to audit logs.

Reason 1: This tool is supposed to be used when the system is down/offline. When the PKI server is down, it would be logging the Cert expired error messages to debug log. At this point, it is clearly logged that something is wrong in the system and that only trusted PKI admin would be running this tool. 

Reason 2: Let's say for some reason someone decides to turn off Selftest in a running PKI server. It would require `root` access to the PKI server and this tool can be run only on the server. 

How selftest module works: It removes/adds the "critical" part from the CS.cfg -- https://github.com/dogtagpki/pki/blob/master/base/ca/shared/conf/CS.cfg#L1190

w.r.t. why cert-fix works with --agent-cert pkidbuser (which lacks Certificate Manager group membership), I have confirmed
that cert-fix does use renewal requests (rather than "Fresh enrollment").  I am still investigating why the request succeeds
with a non-agent certificate (it may be by design, or I may be missing something).

Hi Fraser,

Could you please try to review it.

Thanks
Geetika

I am marking this bug verified for now based on feedback from Dinesh and Fraser.
I have raised another bug : https://bugzilla.redhat.com/show_bug.cgi?id=1739124 which deals with certificate errors due to self signed certificate in LDAPS.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3416