Can you provide more information re. the error when trying to start pki-tomcat? Please upload Dogtag logs, as well as 389-ds logs.

I am not able to transfer logs to this site easily.  Here is what we are seeing:

# ipactl -d restart
dirsrv active
krb5kdc active
kadmin active
named active
ipa_memchached active
httpd active
pki-tomcat active
wget ... https://server.domain.name:8443/ca/admin/ca/getStatus
return code=8
connecting to server.domain.name;8443 .... connected.
HTTP/1.1 404 Not Found
ERROR 404 Not Found.

It retries for 30 seconds then shuts down all the services.

I don't know what/where the dogtab log is?

There is no current logs in the /var/log/pki/pki-tomcat/ca directory.

The only log in /var/log/pki/pki-tomcat is catalina-`date`.log  It has no errors. The messages match what we see on the functioning replicas.

The error log in /var/log/dirsrv/slapd-DOMAIN-NAME/errors:
Skipping CoS Definition cn=Password Policy,cn=accounts...
set_krb5_creds Could not get initial credentials for principal ldap/server.domain.name in keytab /etc/dirsrv/ds.keytab (Cannot contact any KDC for requested realm)
schema-compat-plugin - tree scan will start in 5 sec
Checking for DES passowrd to convert to AES...
sasl_ldap__sasl_interactive_bind - Error: could not perform interactive bing for id mech Unspecidied GSS failure. No Kerberos credentials available errno 0 (Success)
slapd_ldap_bind - Error: could not perform interative bind for id authentication mechanism error -2

We ran ipa-getkeytab -s ca_server.domain.name -p server.domain.name -k /etc/dirsrv/ds.keytab after the certs were renewed.

# klist -kt /etc/dirsrv/ds.keytab
shows updated timestamp with correct principal
4 11/15/2018 ldap/server.domain.name

Hi,

you can try to start the services with
# ipactl restart --ignore-service-failures

This will let the other services running even if pki tomcat fails to start.
Then I would check that the replication is working (ds.keytab is used to authenticate the replication): verify that the DS keytab is correct and can be used to query other master:

# kinit -kt /etc/dirsrv/ds.keytab ldap/`hostname`
# klist
# ldapsearch -Y GSSAPI -h `hostname` -b "" -s base
# ldapsearch -Y GSSAPI -h the.other.master.fqdn -b "" -s base

Also you mentioned that the master has a CA, but is were the replicas setup with --setup-ca?

#ipactl restart --ignore-service-failures
  Starting Directory Service
  Failed to read data from Directory Service; unknown error when retrieving     list of services from LDAP (errno 13) Permission Denied.

#kinit admin
  Kinit: Generic error (see e-text) while getting initial credentials

#kinit -kt /etc/dirsrv/ds.keytab ldap/'hostname'
  kinit: Keytab contacts no suitable keys for ldap/'hostname' while getting initial credentials

#klist
klist: Credentials cache keyring 'persistent:1005187:1005187' not found

# ldapsearch -Y GSSAPI -h `hostname` -b "" -s base
ldap_sasl_interactive_bind_s:can't contact LDAP server (-1)


# ldapsearch -Y GSSAPI -h the.other.master.fqdn -b "" -s base
SASL/GSSAPI authentication started
ldap_sasl_interactive_bind_s:can't contact LDAP server (-2)
additional info: SASL (-1): generic failure:GSSAPI Error: Unspecified GSS failure. Minor Code may provide more info (no Kerberos credentials available)

Hi,
the first error needs to be investigated:
Failed to read data from Directory Service; unknown error when retrieving     list of services from LDAP (errno 13) Permission Denied.

Can you provide the logs from dirsrv corresponding to when 'ipactl restart --ignore-service-failures' is run? On a running system, ipactl first starts the directory service, then connects and performs a search looking for a list of services to start:

(in /var/log/dirsrv/slapd-xxx/access):
[date] conn=8 fd=78 slot=78 connection from local to /var/run/slapd-DOMAIN-COM.socket
[date] conn=8 AUTOBIND dn="cn=Directory Manager"
[date] conn=8 op=0 BIND dn="cn=Directory Manager" method=sasl version=3 mech=EXTERNAL
[date] conn=8 op=0 RESULT err=0 tag=97 nentries=0 etime=0.0001239907 dn="cn=Directory Manager"
[date] conn=8 op=1 SRCH base="replica.domain.com,cn=masters,cn=ipa,cn=etc,dc=domain,dc=com" scope=2 filter="(ipaConfigString=enabledService)" attrs="ipaConfigString cn"
[date] conn=8 op=1 RESULT err=0 tag=101 nentries=9 etime=0.0003771201 notes=U

We need to find why you get permission denied.
- Did you define a user called root inside IPA or a user with uid=0?
- What is the ldap_uri defined in /etc/ipa/default.conf on this host? Does it start with ldapi://... ?
- What are the permissions for /var/run/slapd-DOMAIN-COM.socket ?

The commands were not run correctly last time.  
We re-ran the commands from above.  Here are the results.

# ipactl restart --ignore-service-failures
 Failed to start pki-tomcat
 Force start, ignoring pki-tomcat service, continue normal operation


# kinit admin
#
  success!


# kinit -kt /etc/dirsrv/ds.keytab ldap/'hostname'
# 
  No output is returned just a command prompt


# klist
ticket cache: KEYRING:persistent:0:krbccache
default principal: ldap/`hostname`@REALM
valid               expires             service principal
current date time   current day time    krbtgt REALM@REALM


# ldapsearch -Y GSSAPI -h `hostname` -b "" -s base
  returned the top level of the LDAP schema

# ldapsearch -Y GSSAPI -h the.other.master.fqdn -b "" -s base
  returned the same as the previous command other than the netscapemdsuffix

We also ran these commands-

replica# kvno ldap/`hostanme`
ldap/`hostanme` kvno = 4


server# kvno ldap/`hostname`
ldap/`hostname` kvno = 2


server# kinit -kt /etc/dirsrv/ds.keytab ldap/'hostname' 
No output is returned on the server or the replica from this command

We changed kinit to klist -

replica# klist -kt /etc/dirsrv/ds.keytab ldap/'hostname'
2 date    time    ldap/`hostname`
4 newdate newtime ldap/`hostname`


server# klist -kt /etc/dirsrv/ds.keytab ldap/'hostname'
2 date    time    ldap/`hostname`

You shouldn't have needed to manually update the certificates on the masters. certmonger should do that for you.

What steps did you do to manually import the certs?

The RootRA (ipaCert) certificate did not renew automatically.
We used the steps for manually renewing replica certs from:
https://access.redhat.com/solutions/3357331

We exported the following certificates using certutil on the server:
/etc/httpd/alias ipaCert
/etc/pki/pki-tomcat/alias  ocspSigningCert, subsystemCert, auditSigningCert

We imported them using certutil on the replica.

We restarted certmonger and the certificates then showed MONITORING with valid expiration dates.

We have noticed the serial number for the ipaCert and the serial number in the uid=ipara,ou=People object keeps getting off.  

2 weeks ago we updated the ldap object to the older serial number (69), which matches all the servers ipaCert.  

It seems the ldap object got updated with a new serial number (268107781) and the usercertificate was added as well but the ipaCert is unchanged.  

Is it better to take the latest certificate from the ldap object and import it on each server using certutil, or just update the serial number in ldap to the older number?

You weren't supposed to update all of the certificates. You probably didn't need to touch the agent cert either. You basically circumvented the proper renewal of them.

You are now going to have to manually update some things. See https://www.freeipa.org/page/IPA_2x_Certificate_Renewal and find the part starting "Let's continue getting the CA up and running."

Follow those steps up to "Now you can try to restart the CA to see what happens. It should come up fine" and do the restart. See if that helps.

Hello, could you please provide information if this issue is still valid?