Bug 2217101

Hi,

the check is failing in this portion of code: https://github.com/dogtagpki/pki/blob/29124d8a4e138c96416da88535f853f7a8ff0903/base/server/healthcheck/pki/server/healthcheck/clones/connectivity_and_data.py#L23-L45
----- 8< -----
    def check_ca_clones(self):
        for host in self.clone_cas:
            cur_clone_msg = ' Host: ' + host.Hostname + ' Port: ' + host.SecurePort
            # Reach out and get some certs, to serve as a data and connectivity check
            try:
                connection = PKIConnection(protocol='https',
                                           hostname=host.Hostname,
                                           port=host.SecurePort,
                                           verify=False)

                cert_client = CertClient(connection)
                # get the first 3 in case we cant to make a sanity check of replicated data
                certs = cert_client.list_certs(size=3)

                if certs is not None and len(certs.cert_data_info_list) == 3:
                    logger.info('Cert data successfully obtained from clone.')
                else:
                    raise BaseException('CA clone problem reading data.' + cur_clone_msg)           <<<<<<<< HERE
            except BaseException as e:
                logger.error("Internal server error %s", e)
                raise BaseException('Internal error testing CA clone.' + cur_clone_msg)

        return
----- 8< -----


Can you check how many certificates are present in the certificate repository? The code asks for 3 certs at most and errors if the number of returned certs is different.

# ldapsearch-D cn=directory\ manager -W -b ou=certificateRepository,ou=ca,o=ipaca

From dirsrv access logs I suspect that there is only 1 cert:
[08/Jun/2023:00:20:18.469325243 -0400] conn=22 op=8863 SRCH base="ou=certificateRepository,ou=ca,o=ipaca" scope=0 filter="(|(objectClass=*)(objectClass=ldapsubentry))" attrs="description"
[08/Jun/2023:00:20:18.469561451 -0400] conn=22 op=8863 RESULT err=0 tag=101 nentries=1 wtime=0.000135678 optime=0.000239513 etime=0.000372296

Maybe the customer manually cleaned its certificate repository?

You can ignore my previous comment about the number of certs
> # ldapsearch-D cn=directory\ manager -W -b ou=certificateRepository,ou=ca,o=ipaca
>
> From dirsrv access logs I suspect that there is only 1 cert:
> [08/Jun/2023:00:20:18.469325243 -0400] conn=22 op=8863 SRCH base="ou=certificateRepository,ou=ca,o=ipaca" scope=0 filter="(|(objectClass=*)(objectClass=ldapsubentry))" attrs="description"
> [08/Jun/2023:00:20:18.469561451 -0400] conn=22 op=8863 RESULT err=0 tag=101 nentries=1 wtime=0.000135678 optime=0.000239513 etime=0.000372296
This is a base search (scope=0) so it's normal that only 1 entry is returned.

The next thing that comes to mind is to check the VLV indices on the replica. If they are corrupt they need to be rebuilt (follow https://access.redhat.com/solutions/7017480).
Can you grab a full sos report of the replica after a run of ipa-healthcheck from another server? The interesting logs would be /var/log/pki/pki-tomcat/* and /var/log/dirsrv/*, so that we can check the LDAP search performed on the replica when the certs are queried.

Moving this issue to pki-core. The check is provided in pki source code.