Bug 1696849 - Support LDAPI and LDAPS for cert-fix tool [NEEDINFO]
Summary: Support LDAPI and LDAPS for cert-fix tool
Status: ON_QA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: pki-core
Version: 8.1
Hardware: Unspecified
OS: Unspecified
Target Milestone: rc
: 8.1
Assignee: Fraser Tweedale
QA Contact: PKI QE
Depends On: 1468348
Blocks: 1644708 1472344 1550132 1647919 1669257 1690191
TreeView+ depends on / blocked
Reported: 2019-04-05 18:09 UTC by Dinesh Prasanth
Modified: 2019-08-16 10:42 UTC (History)
18 users (show)

Fixed In Version: pki-core-10.6-8010020190613214740.8ba0ffbe
Doc Type: Enhancement
Doc Text:
The Offline Cert Renewal tool now supports both LDAPI (IPA specific environment) and LDAPS (PKI standalone environment) Ref: https://github.com/dogtagpki/pki/blob/master/docs/admin/Offline_System_Certificate_Renewal.md NOTE: admins are advised to disable remote access to the system while cert-fix tool is being executed.
Clone Of: 1468348
Last Closed:
Type: Bug
gkapoor: needinfo? (ftweedal)

Attachments (Terms of Use)

Comment 3 Dinesh Prasanth 2019-04-30 22:34:20 UTC
The changes have been merged via PRs: 


Document is merged via PR:

Comment 4 Dinesh Prasanth 2019-04-30 22:41:31 UTC
Verification Steps:

There are 3 different scenarios to be tested. I'll add them as separate comments

Scenario 1 (IPA Specific environment -- Uses LDAPI):

# Setting up a fake env:

1. Do a basic `ipa-server-install`
2. Verify `pki -U https://<localhost>:8443 ca-cert-find` works
3. timedatectl set-ntp false && timedatectl set-time <set time to beyond cert expirty date>
4. ipactl restart # some error OR should be stuck when trying to restart PKI server

# Procedure:


Comment 5 Dinesh Prasanth 2019-04-30 22:47:57 UTC
Scenario 2 (PKI Specific environment) (Use LDAPS):

# Setting up a fake env:

1. Do a pkispawn CA
2. set date beyond cert expiry
3. Ensure a valid DS certificate is available

# procedure:


# Shortcut procedure for setting up fake env:

Since this step requires configuring TLS on Directory Server, I usually install IPA as it configures everything for us.

1. Do a basic `ipa-server-install`. This would do a TLS configured Directory Server.

2. timedatectl set-ntp false && timedatectl set-time <set time to beyond cert expirty date>

3. Determine the $DS_SERIAL by `certutil -L -d /etc/dirsrv/slapd-REALM/ -n Server-Cert`

4. Determine the $IPA_RA_SERIAL by `keytool -printcert -file /var/lib/ipa/ra-agent.pem`

5. pki-server cert-fix --ldapi-socket /var/run/slapd-REALM.socket --agent-uid admin --extra-cert $DS_SERIAL --extra-cert $IPA_RA_SERIAL

This step just renews a the DS cert and IPA_RA cert as they need to be valid

6. pki-server cert-fix  --ldap-url <LDAP URL> --agent-uid admin # This would fix all the system certs

Comment 6 Dinesh Prasanth 2019-04-30 22:50:31 UTC
Scenario 3 (PKI Specific environment) (can use LDAP or LDAPS):

# Setting up a fake env:

1. Do a pkispawn CA
2. set date beyond cert expiry
3. restart PKI server

# procedure:


Comment 7 Dinesh Prasanth 2019-05-01 17:36:03 UTC
Since testing this tool requires significant efforts, please consider testing https://bugzilla.redhat.com/show_bug.cgi?id=1679480 together with this bug

Comment 9 Geetika Kapoor 2019-07-10 10:58:53 UTC
Hi Dinesh,

I have tried the first scenario and based on that I have few observations.

Scenario 1 : IPA Specific

1. When renewing a regular certificate why we need to stop/start CA instance.I mean if we renew a non-system based certificate then also I see a pki-tomcat restart?
2. There is a place where we are resetting the password. below are the logs. Could you please have a look there.

INFO: Resetting password for uid=admin,ou=people,o=ipaca
SASL/EXTERNAL authentication started
SASL username

3. In pki-tomcat logs , we are seeing below logs. 

2019-07-10 04:19:12 [http-nio-8080-exec-1] WARNING: Failed to read product version String. /usr/share/pki/CS_SERVER_VERSION (No such file or directory)
java.io.FileNotFoundException: /usr/share/pki/CS_SERVER_VERSION (No such file or directory)

4. If we run : # pki-server cert-fix --ldapi-socket /var/run/slapd-EXAMPLE-COM.socket --agent-uid pkidbuser  --extra-cert 6 --debug 5

here we have put pkidbuser in place of admin so the command fails with :

INFO: Starting the instance
Job for pki-tomcatd@pki-tomcat.service failed because a timeout was exceeded.
See "systemctl status pki-tomcatd@pki-tomcat.service" and "journalctl -xe" for details.
INFO: Selftests enabled for subsystems: ca
INFO: Restoring previous LDAP configuration
ERROR: Command: systemctl start pki-tomcatd@pki-tomcat.service

Later if we run the right command, System goes in a state that it is not recovered. 

# pki-server cert-fix --ldapi-socket /var/run/slapd-EXAMPLE-COM.socket --agent-uid admin  --extra-cert 6

INFO: Starting the instance
Job for pki-tomcatd@pki-tomcat.service failed because a timeout was exceeded.
See "systemctl status pki-tomcatd@pki-tomcat.service" and "journalctl -xe" for details.
INFO: Selftests enabled for subsystems: ca
INFO: Restoring previous LDAP configuration
ERROR: Command: systemctl start pki-tomcatd@pki-tomcat.service

Comment 10 Dinesh Prasanth 2019-07-10 20:54:52 UTC
Hi Geetika,

For (1) and (2), as I explained over IRC, the CA restarts once when this tool changes from cert-based auth mechanism to password-based auth mechanism. This is done by updating CS.cfg. Hence a restart.

There are 2 password resets -- for admin and pkidbuser (optional)... And hence the another CA restart. The tool checks whether `internaldb` password is set in password.conf.. If available, uses it. If not, resets the password of pkidbuser.

For (3), I have been seeing that in my other instances too. It was not introduced by this tool.

For (4), I looked at the box you shared. The system was in a very bad state based on the following observations:
* password.conf had `internaldb` while it should not -- Probably the cert-fix tool was terminated/crashed midway
* pkidbuser password was probably reset twice and the password was lost when you ran with `--agent-uid pkidbuser`... 
* CS.cfg is corrupted.

In fact, after removing `internaldb` key from password.conf and setting the CS.cfg manually, i was able to run the cert-fix tool. It actually renewed all certs BUT failed at the last point where it restores CS.cfg values and tries to start the server. I wasn't sure on how to fix the CS.cfg

Comment 11 Dinesh Prasanth 2019-07-11 00:19:00 UTC

I took as a challenge to fix your box. Guess what? I was able to fix it!! :)

Note that I reverted the CS.cfg to CS.cfg.bak.20190709083459 from archives. -- this doesn't use a secure connection to LDAP. (Did you install with custom IPA config?)

I also added the internaldb=<the value you shared>  to the password.conf... the box works:

[root@pki1 ~]# pki -U https://localhost:8443 ca-cert-find
WARNING: BAD_CERT_DOMAIN encountered on 'CN=pki1.example.com,O=EXAMPLE.COM' indicates a common-name mismatch
WARNING: UNTRUSTED ISSUER encountered on 'CN=pki1.example.com,O=EXAMPLE.COM' indicates a non-trusted CA cert 'CN=Certificate Authority,O=EXAMPLE.COM'
Import CA certificate (Y/n)? n
54 entries found
  Serial Number: 0x1
  Subject DN: CN=Certificate Authority,O=EXAMPLE.COM
  Issuer DN: CN=Certificate Authority,O=EXAMPLE.COM
  Status: VALID
  Type: X.509 version 3
  Key Algorithm: PKCS #1 RSA with 3072-bit key
  Not Valid Before: Tue Jul 09 08:34:45 EDT 2019
  Not Valid After: Sat Jul 09 08:34:45 EDT 2039
  Issued On: Tue Jul 09 08:34:45 EDT 2019
  Issued By: system


[root@pki1 ~]# timedatectl
               Local time: Sun 2022-05-22 00:20:05 EDT
           Universal time: Sun 2022-05-22 04:20:05 UTC
                 RTC time: Sun 2022-05-22 00:20:05
                Time zone: America/New_York (EDT, -0400)
System clock synchronized: no
              NTP service: inactive
          RTC in local TZ: yes

I haven't changed the date back to the current settings. Let me know if your box is stable now :)


Comment 12 Fraser Tweedale 2019-07-11 09:59:33 UTC
Some further discussion of this.

First, some of the things there were questions about (especially: why do we reset certain account passwords?)
are explained in my blog post: https://frasertweedale.github.io/blog-redhat/posts/2019-03-18-cert-fix-redux.html.

We have agreed that using pkidbuser as the --agent-uid is a corner case that should be prevented
(i.e. have a sanity check that that account is -not- used).  It will be a small patch.  I see two

1. Verify this BZ in spite of pkidbuser corner case, and open a new BZ to address the corner case
   (detect and prevent pkidbuser being used as --admin-uid).  Or;

2. Mark this BZ FailedQA, cut patch, await new build, etc.

I'm don't mind either way and I'll most likely cut the patch tomorrow (unless Dinesh wants to take it).

Dinesh, thanks for your analysis.  Geetika thanks for your solid QE work!

Comment 13 Geetika Kapoor 2019-07-11 11:45:17 UTC
Thanks Fraser , Dinesh for helping me in this Bug testing.

Another query about usage of : --agent-uid <String>        UID of Dogtag agent user

Test Case 2: I have added a localuser which is not part of any group(admin/agent). So in that case if we use it as --agent-uid how it should behave.

dn: uid=localuser,ou=people,o=ipaca
objectClass: cmsuser
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
objectClass: top
cn: localuser
sn: localuser
usertype: happytest
mail: root@localhost
uid: localuser
userstate: 1


Step 1 : Description says :       --agent-uid <String>        UID of Dogtag agent user

Run The same command with a non agent dogtag user

# pki-server cert-fix --ldapi-socket /var/run/slapd-EXAMPLE-COM.socket --agent-uid localuser  --extra-cert 6 --debug 5
INFO: Loading instance: pki-tomcat
INFO: Loading global Tomcat config: /etc/tomcat/tomcat.conf
INFO: Loading PKI Tomcat config: /usr/share/pki/etc/tomcat.conf
INFO: Loading instance Tomcat config: /etc/pki/pki-tomcat/tomcat.conf
INFO: Loading password config: /etc/pki/pki-tomcat/password.conf
INFO: Loading instance registry: /etc/sysconfig/pki/tomcat/pki-tomcat/pki-tomcat
INFO: Loading subsystem: ca
INFO: Loading subsystem config: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg
INFO: Fixing the following system certs: []
INFO: Renewing the following additional certs: ['6']
INFO: Stopping the instance to proceed with system cert renewal
DEBUG: Command: systemctl stop pki-tomcatd@pki-tomcat.service
INFO: Configuring LDAP password authentication
INFO: Selftests disabled for subsystems: ca
INFO: Password dm_pass_file
INFO: ['ldapsearch', '-H', 'ldapi://%2Fvar%2Frun%2Fslapd-EXAMPLE-COM.socket', '-Y', 'EXTERNAL', '-s', 'base', '-b', 'o=ipaca', '1.1']
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
INFO: Resetting password for uid=localuser,ou=people,o=ipaca
INFO: ['ldappasswd', '-H', 'ldapi://%2Fvar%2Frun%2Fslapd-EXAMPLE-COM.socket', '-Y', 'EXTERNAL', '-T', '/tmp/tmpatuzqa2n', 'uid=localuser,ou=people,o=ipaca']
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
INFO: Starting the instance
DEBUG: Command: systemctl start pki-tomcatd@pki-tomcat.service
INFO: Sleeping for 10 seconds to allow server time to start...
INFO: Requesting new cert for 6; writing to /etc/pki/pki-tomcat/certs/6-renewed.crt
INFO: Trying to setup a secure connection to CA subsystem.
DEBUG: Starting new HTTPS connection (1): pki1.example.com:8443
DEBUG: https://pki1.example.com:8443 "GET /ca/rest/account/login HTTP/1.1" 200 133
INFO: Secure connection with CA is established.
INFO: Placing cert creation request for serial: 6
DEBUG: https://pki1.example.com:8443 "GET /ca/rest/certrequests/profiles/caManualRenewal HTTP/1.1" 200 470
DEBUG: https://pki1.example.com:8443 "POST /ca/rest/certrequests HTTP/1.1" 200 318
DEBUG: https://pki1.example.com:8443 "GET /ca/rest/certrequests/58 HTTP/1.1" 200 284
DEBUG: https://pki1.example.com:8443 "GET /ca/rest/certs/0x3a HTTP/1.1" 200 None
INFO: Request ID: 58
INFO: Request Status: complete
DEBUG: request_data: {'CertRequestInfo': {'request_id': '58', 'request_type': 'renewal', 'request_status': 'complete', 'request_url': 'https://pki1.example.com:8443/ca/rest/certrequests/58'}}
DEBUG: cert_data: {'CertData': {'serial_number': '0x3a', 'subject_dn': 'CN=ipa-ca-agent,O=EXAMPLE.COM', 'status': 'VALID'}}
INFO: Serial Number: 0x3a
INFO: Issuer: CN=Certificate Authority,O=EXAMPLE.COM
INFO: Subject: CN=ipa-ca-agent,O=EXAMPLE.COM
DEBUG: Pretty Print:
DEBUG:     Certificate: 
            Version:  v3
            Serial Number: 0x3A
            Signature Algorithm: SHA256withRSA - 1.2.840.113549.1.1.11
            Issuer: CN=Certificate Authority,O=EXAMPLE.COM
                Not Before: Sunday, May 22, 2022 11:42:29 AM EDT America/New_York
                Not  After: Monday, May 22, 2023 11:42:29 AM EDT America/New_York
            Subject: CN=ipa-ca-agent,O=EXAMPLE.COM
            Subject Public Key Info: 
                Algorithm: RSA - 1.2.840.113549.1.1.1
                Public Key: 
                    Exponent: 65537
                    Public Key Modulus: (2048 bits) :
                Identifier: Authority Key Identifier -
                    Critical: no 
                    Key Identifier: 
                Identifier: Authority Info Access: -
                    Critical: no 
                    Access Description: 
                        Method #0: ocsp
                        Location #0: URIName: http://ipa-ca.example.com/ca/ocsp
                Identifier: Key Usage: -
                    Critical: yes 
                    Key Usage: 
                        Digital Signature 
                        Non Repudiation 
                        Key Encipherment 
                Identifier: Extended Key Usage: -
                    Critical: no 
                    Extended Key Usage: 
            Algorithm: SHA256withRSA - 1.2.840.113549.1.1.11

DEBUG: https://pki1.example.com:8443 "GET /ca/rest/certs/0x3a HTTP/1.1" 200 None
INFO: New cert is available at: /etc/pki/pki-tomcat/certs/6-renewed.crt
INFO: Stopping the instance
DEBUG: Command: systemctl stop pki-tomcatd@pki-tomcat.service
INFO: Selftests enabled for subsystems: ca
INFO: Restoring previous LDAP configuration
INFO: Starting the instance with renewed certs
DEBUG: Command: systemctl start pki-tomcatd@pki-tomcat.service
[root@pki1 ~]#

Comment 15 Dinesh Prasanth 2019-07-11 20:18:23 UTC
@Geetika and @Fraser,

As per discussion with cfu on IRC, the system is in an intermediate state and we can make an exception:

Snip of the chat for reference:

<cfu> gkapoor, dmoluguw , my point is, if the system is in "preparation mode" (or whatever you call that, it's not yet up and running fully), then why are we worrying about these things?  Can other people reach the CA at this time?
<cfu> gkapoor, the thing is I do not see any sign of cert being issued from the debug log.. in fact, it looks like it just got started and nothing has happened yet
<cfu> gkapoor, is there auditing yet?
<gkapoor__> cfu checking 
<gkapoor__> cfu nopes no traces in logs 
<cfu> gkapoor, yeah, so I'll say it's an exception.  As far as CC goes, this should be treated like "installation time" when things are still being set up
<cfu> gkapoor, dmoluguw, so I'm gonna say make a note and leave it
<gkapoor__> cfu hmm...but in that case it is not working as expected
<cfu> gkapoor, I'm saying maybe you need to make an exception and lower your expectation ;-)
<cfu> gkapoor, dmoluguw , when a system is being set up, such as changing out its system certs, it's considered the same as "installation time" which CC allows for exception.  During installation, no auditing is expected, admins are allowed to change out certs, etc.
<cfu> gkapoor, dmoluguw , although one might want to enable firewall and only allow local connections during such time

Also, I have fixed added a small patch for #1729215 and I have added both of you as reviewers. Thank you Geetika for the stress testing and Fraser for the details! :)

Comment 16 Geetika Kapoor 2019-07-12 07:57:43 UTC
based on https://bugzilla.redhat.com/show_bug.cgi?id=1696849#c13 it means that --agent-uid has no relevance because there is no check for group id and it works for any user which is in LDAP.

Comment 17 Fraser Tweedale 2019-07-12 08:07:51 UTC
It might depend on whether it's a "renewal request" or a fresh enrollment.  I can't recall of top of head which we use in cert-fix.  And it is Friday evening so I will not look until next week :)

Comment 18 Geetika Kapoor 2019-07-16 14:31:55 UTC
just to track and we don't miss on checking comment#16 and #17, I am putting this bug on Needinfo from Fraser.

Comment 19 Geetika Kapoor 2019-07-17 10:34:27 UTC
Dinesh, while doing disable selftest command , i have below observations.

Test Case 2: Manual renewal Process 

1. Disable selftest.

# pki-server -v --debug selftest 
Command: selftest
Module: selftest
 selftest-enable               Enable selftests.             
 selftest-disable              Disable selftests. 

# pki-server -v --debug selftest-disable -i pki-RootCA-CMC3 
Command: selftest-disable -i pki-RootCA-CMC3
Module: selftest
[root@pki1 ~]# pki-server -v --debug selftest-enable -i pki-RootCA-CMC3 
Command: selftest-enable -i pki-RootCA-CMC3
Module: selftest

Question 1: I don't see anything in debug logs for selftest enable/disable. Should it get reported/audited somewhere that selftest is disabled?
Generally doing any operation with selftest needs "CertUserDBAuthentication". which we are not doing while running # pki-server selftest-disable

Comment 20 Dinesh Prasanth 2019-07-17 13:35:40 UTC

We neither log anything to debug log nor to audit logs.

Reason 1: This tool is supposed to be used when the system is down/offline. When the PKI server is down, it would be logging the Cert expired error messages to debug log. At this point, it is clearly logged that something is wrong in the system and that only trusted PKI admin would be running this tool. 

Reason 2: Let's say for some reason someone decides to turn off Selftest in a running PKI server. It would require `root` access to the PKI server and this tool can be run only on the server. 

How selftest module works: It removes/adds the "critical" part from the CS.cfg -- https://github.com/dogtagpki/pki/blob/master/base/ca/shared/conf/CS.cfg#L1190

Comment 21 Fraser Tweedale 2019-07-18 06:58:02 UTC
w.r.t. why cert-fix works with --agent-cert pkidbuser (which lacks Certificate Manager group membership), I have confirmed
that cert-fix does use renewal requests (rather than "Fresh enrollment").  I am still investigating why the request succeeds
with a non-agent certificate (it may be by design, or I may be missing something).

Comment 22 Geetika Kapoor 2019-08-12 15:15:27 UTC
Hi Fraser,

Could you please try to review it.


Note You need to log in before you can comment on or make changes to this bug.