1152084 – [RFE] Create a tool to simplify troubleshooting

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1152084 - [RFE] Create a tool to simplify troubleshooting

Summary: [RFE] Create a tool to simplify troubleshooting

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 8
Classification:	Red Hat
Component:	ipa
Sub Component:
Version:	8.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	8.0
Assignee:	IPA Maintainers
QA Contact:	ipa-qe
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	1623549 (view as bug list)
Depends On:	1683261
Blocks:	1547051 1701002
TreeView+	depends on / blocked

Reported:	2014-10-13 12:45 UTC by Martin Kosek
Modified:	2020-11-14 12:06 UTC (History)
CC List:	14 users (show)
Fixed In Version:	ipa-healthcheck-0.2-3.module+el8.1.0+3235+375327f5
Doc Type:	Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-11-05 20:52:26 UTC
Type:	---
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2019:3348	0	None	None	None	2019-11-05 20:52:52 UTC

Description Martin Kosek 2014-10-13 12:45:42 UTC

This bug is created as a clone of upstream ticket:
https://fedorahosted.org/freeipa/ticket/4008

Create a tool that would collect the following information about the deployment:

 a. Version of the packages
 b. Is CA installed?
 c. Is it chained, self signed etc.
 d. How many replicas?
 e. Do replicas run DNS, CAs?
 f. What is the topology?
 g. Is NIS/Compat enabled?
 h. Is migration mode is on?
 i. What is the status of the internal certificates? How soon they expire? Is certmonger configured to renew them?
 j. Are trusts enabled?
 k. Home many trusts are three?
 l. Is sync is enabled?
 m. Collect install logs
 ...

This tool will allow fast information collection about the deployment and enable more efficient troubleshooting of the potential issues.

Comment 4 Martin Kosek 2014-10-31 11:41:02 UTC

== Updated description ==

Please note that the scope and description of original upstream ticket
https://fedorahosted.org/freeipa/ticket/4008
was updated:

Create a tool that would help with diagnosis and collecting information about server/client deployment.

== Diagnosis ==
The tool should have at least following features:
* Pluggable interface so that checks can be easily provided by developers/community/support (though only limited to Python language for start)
* Clear division between root/non-root checks (skip the checks where current permissions are not enough)
* Server and client checks
* Optionally, admin/privileged person should be able to run it remotely (e.g. via OpenLMI) from FreeIPA Web UI.

'''Initial ideas'''
* Validate that keytabs are ok (known only, /etc/krb5.keytab, /etc/httpd/conf/ipa.keytab and /etc/dirsrv/ds.keytab)
* Get a host TGT
* Verify that the certificates are ok (start with HTTP and DS, maybe machine cert)
* Connectivity, perhaps using the conncheck tool.
* Replication status

== Log collection ==
Optionally, the tool could make log collection easier for further debugs on freeipa-users list for example.

'''Initial ideas'''
* Version of the packages
* Is CA installed?
* Is it chained, self signed etc.
* How many replicas?
* Do replicas run DNS, CAs?
* What is the topology?
* Is NIS/Compat enabled?
* Is migration mode is on?
* What is the status of the internal certificates? How soon they expire? Is certmonger configured to renew them?
* Are trusts enabled?
* Home many trusts are three?
* Is sync is enabled?
* Collect install logs

Comment 7 Petr Vobornik 2016-01-07 15:09:22 UTC

Upstream ticket:
https://fedorahosted.org/freeipa/ticket/5558

Comment 8 Martin Kosek 2016-07-08 06:40:12 UTC

Current upstream design page:
http://www.freeipa.org/page/V4/Diagnostics_Tool

Comment 9 Martin Kosek 2016-07-08 07:34:32 UTC

Current POC implementation by Tomas Babej, as part of his Thesis:
https://github.com/tbabej/freeipa/tree/ipadiag-rebased

Comment 10 Thorsten Scherf 2016-09-11 10:33:31 UTC

It would be nice to add a plugin to the troubleshooting tool which collects logs from multiple (IPA) systems. Something like engine-log-collector [1] in RHEV. It could be used to collect most important log files from all configured IPA server systems and optionally also from client systems.

[1] https://access.redhat.com/solutions/61546

Comment 16 Florence Blanc-Renaud 2018-12-05 12:22:23 UTC

*** Bug 1623549 has been marked as a duplicate of this bug. ***

Comment 21 Rob Crittenden 2019-06-11 12:30:53 UTC

Imported ipa-healthcheck package to provide this functionality.

Comment 23 Rob Crittenden 2019-06-12 17:31:52 UTC

The ipa-healthcheck tool will check for common problems which are often pain points like certificate expiration (i) and some common trust problems along with a number of pain points like file permissions, space issues and replication issues (topology is covered loosely).

There is the capability to provide state on certain topics but today only the version is directly reported.

ipa-healthcheck may never be appropriate as the basis for a fact-based reporter like puppet or Ansible might look for. It will also never collect logs.

Comment 24 Nikhil Dehadrai 2019-07-11 16:41:13 UTC

IPA version: ipa-server-4.7.90.pre1-3.module+el8.1.0+3389+a3c612fa.x86_64
HealthCheck version: ipa-healthcheck-0.2-3.module+el8.1.0+3389+a3c612fa.noarch

Tested the bug with following observations:

1) Command line options:

HELP:
----
[root@vm-idm-018 ipa]# ipa-healthcheck --help
usage: ipa-healthcheck [-h] [--debug] [--list-sources] [--source SOURCE]
                       [--check CHECK] [--output-type {json,human}]
                       [--failures-only] [--output-file FILENAME]
                       [--indent INDENT] [--input-file INFILE]

optional arguments:
  -h, --help            show this help message and exit
  --debug               Include debug output
  --list-sources        List all available sources
  --source SOURCE       Source of checks, e.g. ipahealthcheck.foo.bar
  --check CHECK         Check to execute, e.g. BazCheck
  --output-type {json,human}
                        Output method
  --failures-only       Exclude SUCCESS severity on output

json:
  Output information in JSON format

  --output-file FILENAME
                        File to store output
  --indent INDENT       Indention level of JSON output

human:
  Display output in a more human-friendly way

  --input-file INFILE   File to translate


Commandline validations:
--------------------------------
# ipa-healthcheck --check
usage: ipa-healthcheck [-h] [--debug] [--list-sources] [--source SOURCE]
[--check CHECK] [--output-type

{json,human}]
[--failures-only] [--output-file FILENAME]
[--indent INDENT] [--input-file INFILE]
ipa-healthcheck: error: argument --check: expected one argument

[root@kvm-02-guest15 ~]# ipa-healthcheck --source 
usage: ipa-healthcheck [-h] [--debug] [--list-sources] [--source SOURCE]
[--check CHECK] [--output-type {json,human}
]
[--failures-only] [--output-file FILENAME]
[--indent INDENT] [--input-file INFILE]
ipa-healthcheck: error: argument --source: expected one argument

[root@kvm-02-guest15 ~]# ipa-healthcheck --check ipahealthcheck.meta.services
--source is required when --check is used

[
[root@kvm-02-guest15 ~]# ipa-healthcheck --source ipahealthcheck.meta.services --check certmonger
[{"source": "ipahealthcheck.meta.services", "check": "certmonger", "severity": 0, "uuid": "2b7cc118-50a1-4b7e-9923-6495ba97f605", "when": "20190708101911Z", "duration": "0.010843", "kw": {"status": true}}][root@kvm-02-guest15 ~]# 
[root@kvm-02-guest15 ~]# 
[root@kvm-02-guest15 ~]# ipa-healthcheck --check certmonger
--source is required when --check is used

[root@kvm-02-guest15 ~]# ipa-healthcheck --source ipahealthcheck.meta.services --check certmonger --output-type human
SUCCESS: ipahealthcheck.meta.services.certmonger


root@kvm-02-guest15 ~]# ipa-healthcheck --source ipahealthcheck.meta.services --check certmonger --output-type json
[{"source": "ipahealthcheck.meta.services", "check": "certmonger", "severity": 0, "uuid": "f01b8a8f-b702-4a6f-aec2-f21cb072f470", "when": "20190708102036Z", "duration": "0.010515", "kw": {"status": true}}]

[root@kvm-02-guest15 ~]# 
[root@kvm-02-guest15 ~]# ipa-healthcheck --source ipahealthcheck.meta.services --check certmonger --output-type json --output-file /tmp/test.json --indent 4
[root@kvm-02-guest15 ~]# cat /tmp/test.json 
[
{
"source": "ipahealthcheck.meta.services",
"check": "certmonger",
"severity": 0,
"uuid": "4dae6e18-7253-4033-b9a8-9e2a1fb0669f",
"when": "20190708102139Z",
"duration": "0.010347",
"kw":

{ "status": true }
}


2) Check File permissions for ipa
-----------------------------------
[root@kvm-02-guest15 ipahealthcheck]# ipa-healthcheck --failures-only --output-type human
WARNING: ipahealthcheck.ipa.files.IPAFileCheck._etc_ipa_ca.crt_mode: Permissions of /etc/ipa/ca.crt are 0222 and should be 0644
[root@kvm-02-guest15 ipahealthcheck]# ipactl restart
Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Restarting httpd Service
Restarting ipa-custodia Service
Restarting pki-tomcatd Service
Restarting ipa-otpd Service
Restarting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful
[root@kvm-02-guest15 ipahealthcheck]# ipa-healthcheck --failures-only --output-type human
WARNING: ipahealthcheck.ipa.files.IPAFileCheck._etc_ipa_ca.crt_mode: Permissions of /etc/ipa/ca.crt are 0222 and should be 0644
[root@kvm-02-guest15 ipahealthcheck]# chmod 644 /etc/ipa/ca.crt 
[root@kvm-02-guest15 ipahealthcheck]# ipa-healthcheck --failures-only --output-type human
[root@kvm-02-guest15 ipahealthcheck]#


3) If Services stopped on IPA
----------------------------------
root@kvm-04-guest19 ~]# ipactl stop
Stopping ipa-dnskeysyncd Service
Stopping ipa-otpd Service
Stopping pki-tomcatd Service
Stopping ipa-custodia Service
Stopping httpd Service
Stopping named Service
Stopping kadmin Service
Stopping krb5kdc Service
Stopping Directory Service
ipa: INFO: The ipactl command was successful
[root@kvm-04-guest19 ~]# ipa-healthcheck --failures-only --output-type human
ERROR: ipahealthcheck.meta.services.dirsrv: dirsrv: not running
ERROR: ipahealthcheck.meta.services.httpd: httpd: not running
ERROR: ipahealthcheck.meta.services.krb5kdc: krb5kdc: not running
ERROR: ipahealthcheck.meta.services.named: named: not running
ERROR: ipahealthcheck.meta.services.pki_tomcatd: pki_tomcatd: not running
ERROR: ipahealthcheck.dogtag.ca.DogtagCertsConnectivityCheck: Request for certificate failed, ldap2 is not connected (ldap2_139893703243368 in MainThread)
CRITICAL: ipahealthcheck.ds.replication.ReplicationConflictCheck: cannot connect to 'ldapi://%2Fvar%2Frun%2Fslapd-TESTRELM-TEST.socket': Connection refused
CRITICAL: ipahealthcheck.ipa.certs.IPACertTracking: ldap2 is not connected (ldap2_139893703243368 in MainThread)
CRITICAL: ipahealthcheck.ipa.certs.IPARAAgent: Skipping because no LDAP connection
CRITICAL: ipahealthcheck.ipa.certs.IPACertRevocation: ldap2 is not connected (ldap2_139893703243368 in MainThread)
CRITICAL: ipahealthcheck.ipa.files.IPAFileCheck: ldap2 is not connected (ldap2_139893703243368 in MainThread)
ERROR: ipahealthcheck.ipa.host.IPAHostKeytab: Failed to obtain host TGT: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529639068): Cannot contact any KDC for realm 'TESTRELM.TEST'
ERROR: ipahealthcheck.ipa.topology.IPATopologyDomainCheck: topologysuffix-verify domain failed, ldap2 is not connected (ldap2_139893703243368 in MainThread)
CRITICAL: ipahealthcheck.ipa.topology.IPATopologyDomainCheck: ldap2 is not connected (ldap2_139893703243368 in MainThread)
[root@kvm-04-guest19 ~]# 
[root@kvm-04-guest19 ~]# ipactl start
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting httpd Service
Starting ipa-custodia Service
Starting pki-tomcatd Service
Starting ipa-otpd Service
Starting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful
[root@kvm-04-guest19 ~]# ipa-healthcheck --failures-only --output-type human
[root@kvm-04-guest19 ~]#


4) Upgrade Scenarios (Success)
------------------------------------
run ipa-upgrade from rhel 80 > 8.1, run ipa-health check
[root@vm-idm-006 ~]# tail -1 /var/log/ipaupgrade.log 
2019-07-08T11:24:16Z INFO The ipa-server-upgrade command was successful
[root@vm-idm-006 ~]# rpm -q ipa-server
ipa-server-4.8.0-1.module+el8.1.0+3577+202f0a51.x86_64
[root@vm-idm-006 ~]# ipa-healthcheck --failures-only --output-type human
[root@vm-idm-006 ~]#


5) Upgrade Scenario (Failure)
----------------------------------
[root@vm-idm-016 ~]# ipa-healthcheck --failures-only --output-type human
Unable to find server cert nickname in /etc/dirsrv/slapd-TESTRELM-TEST/dse.ldif
ERROR: ipahealthcheck.meta.services.dirsrv: dirsrv: not running
ERROR: ipahealthcheck.meta.services.httpd: httpd: not running
ERROR: ipahealthcheck.meta.services.krb5kdc: krb5kdc: not running
ERROR: ipahealthcheck.meta.services.named: named: not running
ERROR: ipahealthcheck.meta.services.pki_tomcatd: pki_tomcatd: not running
ERROR: ipahealthcheck.dogtag.ca.DogtagCertsConnectivityCheck: Request for certificate failed, ldap2 is not connected (ldap2_140083977303824 in MainThread)
CRITICAL: ipahealthcheck.ds.replication.ReplicationConflictCheck: cannot connect to 'ldapi://%2Fvar%2Frun%2Fslapd-TESTRELM-TEST.socket': No such file or directory
CRITICAL: ipahealthcheck.ipa.certs.IPACertTracking: ldap2 is not connected (ldap2_140083977303824 in MainThread)
CRITICAL: ipahealthcheck.ipa.certs.IPARAAgent: Skipping because no LDAP connection
CRITICAL: ipahealthcheck.ipa.certs.IPACertRevocation: ldap2 is not connected (ldap2_140083977303824 in MainThread)
CRITICAL: ipahealthcheck.ipa.files.IPAFileCheck: ldap2 is not connected (ldap2_140083977303824 in MainThread)
ERROR: ipahealthcheck.ipa.host.IPAHostKeytab: Failed to obtain host TGT: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529639068): Cannot contact any KDC for realm 'TESTRELM.TEST'
ERROR: ipahealthcheck.ipa.topology.IPATopologyDomainCheck: topologysuffix-verify domain failed, ldap2 is not connected (ldap2_140083977303824 in MainThread)
CRITICAL: ipahealthcheck.ipa.topology.IPATopologyDomainCheck: ldap2 is not connected (ldap2_140083977303824 in MainThread)


6) Running Healthcheck on Client:
----------------------------------
#running ipa-healthcheck on client
[root@kvm-06-guest05 ~]# ipa-healthcheck 
Unable to initialize ipahealthcheck.dogtag: IPA is not configured on this system.

7) Promoting the client to be replica:
------------------------------------------
[root@kvm-06-guest05 ~]# ipa-healthcheck --failures-only --output-type human
[root@kvm-06-guest05 ~]#

8) Certificate Expired:
-------------------------
[root@vm-idm-018 ipa]# ipa-healthcheck --failures-only --output-type human
ERROR: ipahealthcheck.meta.services.pki_tomcatd: pki_tomcatd: not running
ERROR: ipahealthcheck.dogtag.ca.DogtagCertsConnectivityCheck: Request for certificate failed, cannot connect to 'https://vm-idm-018.testrelm.test:443/ca/agent/ca/displayBySerial': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)
ERROR: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711130443: Request id 20190711130443 expired on 20210711130444Z
ERROR: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711135122: Request id 20190711135122 expired on 20210630130048Z
ERROR: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711135126: Request id 20190711135126 expired on 20210630130046Z
ERROR: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711135129: Request id 20190711135129 expired on 20210630130047Z
ERROR: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711135135: Request id 20190711135135 expired on 20210630130156Z
ERROR: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711135136: Request id 20190711135136 expired on 20210630130046Z
ERROR: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711135142: Request id 20190711135142 expired on 20210711130344Z
ERROR: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711135152: Request id 20190711135152 expired on 20210711130424Z
ERROR: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711130443: Request id 20190711130443 expired on 20210711130444Z
ERROR: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711135122: Request id 20190711135122 expired on 20210630130048Z
ERROR: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711135126: Request id 20190711135126 expired on 20210630130046Z
ERROR: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711135129: Request id 20190711135129 expired on 20210630130047Z
ERROR: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711135135: Request id 20190711135135 expired on 20210630130156Z
ERROR: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711135136: Request id 20190711135136 expired on 20210630130046Z
ERROR: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711135142: Request id 20190711135142 expired on 20210711130344Z
ERROR: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711135152: Request id 20190711135152 expired on 20210711130424Z
ERROR: ipahealthcheck.ipa.certs.IPANSSChainValidation./etc/pki/pki-tomcat/alias:Server-Cert cert-pki-ca: Validation of Server-Cert cert-pki-ca in /etc/pki/pki-tomcat/alias failed: certutil: certificate is invalid: Peer's Certificate has expired.

ERROR: ipahealthcheck.ipa.certs.IPANSSChainValidation./etc/dirsrv/slapd-TESTRELM-TEST:Server-Cert: Validation of Server-Cert in /etc/dirsrv/slapd-TESTRELM-TEST/ failed: certutil: certificate is invalid: Peer's Certificate has expired.

ERROR: ipahealthcheck.ipa.certs.IPAOpenSSLChainValidation./var/lib/ipa/certs/httpd.crt: Certificate validation for /var/lib/ipa/certs/httpd.crt failed: O = TESTRELM.TEST, CN = vm-idm-018.testrelm.test
error 10 at 0 depth lookup: certificate has expired

ERROR: ipahealthcheck.ipa.certs.IPAOpenSSLChainValidation./var/lib/ipa/ra-agent.pem: Certificate validation for /var/lib/ipa/ra-agent.pem failed: O = TESTRELM.TEST, CN = IPA RA
error 10 at 0 depth lookup: certificate has expired

ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135135: Request for certificate failed, cannot connect to 'https://vm-idm-018.testrelm.test:443/ca/agent/ca/displayBySerial': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135122: Request for certificate failed, cannot connect to 'https://vm-idm-018.testrelm.test:443/ca/agent/ca/displayBySerial': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135126: Request for certificate failed, cannot connect to 'https://vm-idm-018.testrelm.test:443/ca/agent/ca/displayBySerial': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135129: Request for certificate failed, cannot connect to 'https://vm-idm-018.testrelm.test:443/ca/agent/ca/displayBySerial': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135132: Request for certificate failed, cannot connect to 'https://vm-idm-018.testrelm.test:443/ca/agent/ca/displayBySerial': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135136: Request for certificate failed, cannot connect to 'https://vm-idm-018.testrelm.test:443/ca/agent/ca/displayBySerial': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135152: Request for certificate failed, cannot connect to 'https://vm-idm-018.testrelm.test:443/ca/agent/ca/displayBySerial': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135142: Request for certificate failed, cannot connect to 'https://vm-idm-018.testrelm.test:443/ca/agent/ca/displayBySerial': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190711130443: Request for certificate failed, cannot connect to 'https://vm-idm-018.testrelm.test:443/ca/agent/ca/displayBySerial': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)
[root@vm-idm-018 ipa]#

[root@vm-idm-018 ipa]# ipa-healthcheck --source ipahealthcheck.ipa.certs --output-type human
ERROR: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711130443: Request id 20190711130443 expired on 20210711130444Z
ERROR: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711135122: Request id 20190711135122 expired on 20210630130048Z
ERROR: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711135126: Request id 20190711135126 expired on 20210630130046Z
ERROR: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711135129: Request id 20190711135129 expired on 20210630130047Z
SUCCESS: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711135132
ERROR: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711135135: Request id 20190711135135 expired on 20210630130156Z
ERROR: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711135136: Request id 20190711135136 expired on 20210630130046Z
ERROR: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711135142: Request id 20190711135142 expired on 20210711130344Z
ERROR: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711135152: Request id 20190711135152 expired on 20210711130424Z
ERROR: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711130443: Request id 20190711130443 expired on 20210711130444Z
ERROR: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711135122: Request id 20190711135122 expired on 20210630130048Z
ERROR: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711135126: Request id 20190711135126 expired on 20210630130046Z
ERROR: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711135129: Request id 20190711135129 expired on 20210630130047Z
SUCCESS: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711135132
ERROR: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711135135: Request id 20190711135135 expired on 20210630130156Z
ERROR: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711135136: Request id 20190711135136 expired on 20210630130046Z
ERROR: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711135142: Request id 20190711135142 expired on 20210711130344Z
ERROR: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711135152: Request id 20190711135152 expired on 20210711130424Z
SUCCESS: ipahealthcheck.ipa.certs.IPACertTracking.20190711135135
SUCCESS: ipahealthcheck.ipa.certs.IPACertTracking.20190711135122
SUCCESS: ipahealthcheck.ipa.certs.IPACertTracking.20190711135126
SUCCESS: ipahealthcheck.ipa.certs.IPACertTracking.20190711135129
SUCCESS: ipahealthcheck.ipa.certs.IPACertTracking.20190711135132
SUCCESS: ipahealthcheck.ipa.certs.IPACertTracking.20190711135136
SUCCESS: ipahealthcheck.ipa.certs.IPACertTracking.20190711135152
SUCCESS: ipahealthcheck.ipa.certs.IPACertTracking.20190711135142
SUCCESS: ipahealthcheck.ipa.certs.IPACertTracking.20190711130443
SUCCESS: ipahealthcheck.ipa.certs.IPACertNSSTrust.caSigningCert cert-pki-ca
SUCCESS: ipahealthcheck.ipa.certs.IPACertNSSTrust.ocspSigningCert cert-pki-ca
SUCCESS: ipahealthcheck.ipa.certs.IPACertNSSTrust.subsystemCert cert-pki-ca
SUCCESS: ipahealthcheck.ipa.certs.IPACertNSSTrust.auditSigningCert cert-pki-ca
SUCCESS: ipahealthcheck.ipa.certs.IPACertNSSTrust.Server-Cert cert-pki-ca
ERROR: ipahealthcheck.ipa.certs.IPANSSChainValidation./etc/pki/pki-tomcat/alias:Server-Cert cert-pki-ca: Validation of Server-Cert cert-pki-ca in /etc/pki/pki-tomcat/alias failed: certutil: certificate is invalid: Peer's Certificate has expired.

ERROR: ipahealthcheck.ipa.certs.IPANSSChainValidation./etc/dirsrv/slapd-TESTRELM-TEST:Server-Cert: Validation of Server-Cert in /etc/dirsrv/slapd-TESTRELM-TEST/ failed: certutil: certificate is invalid: Peer's Certificate has expired.

ERROR: ipahealthcheck.ipa.certs.IPAOpenSSLChainValidation./var/lib/ipa/certs/httpd.crt: Certificate validation for /var/lib/ipa/certs/httpd.crt failed: O = TESTRELM.TEST, CN = vm-idm-018.testrelm.test
error 10 at 0 depth lookup: certificate has expired

ERROR: ipahealthcheck.ipa.certs.IPAOpenSSLChainValidation./var/lib/ipa/ra-agent.pem: Certificate validation for /var/lib/ipa/ra-agent.pem failed: O = TESTRELM.TEST, CN = IPA RA
error 10 at 0 depth lookup: certificate has expired

SUCCESS: ipahealthcheck.ipa.certs.IPARAAgent
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135135: Request for certificate failed, cannot connect to 'https://vm-idm-018.testrelm.test:443/ca/agent/ca/displayBySerial': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135122: Request for certificate failed, cannot connect to 'https://vm-idm-018.testrelm.test:443/ca/agent/ca/displayBySerial': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135126: Request for certificate failed, cannot connect to 'https://vm-idm-018.testrelm.test:443/ca/agent/ca/displayBySerial': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135129: Request for certificate failed, cannot connect to 'https://vm-idm-018.testrelm.test:443/ca/agent/ca/displayBySerial': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135132: Request for certificate failed, cannot connect to 'https://vm-idm-018.testrelm.test:443/ca/agent/ca/displayBySerial': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135136: Request for certificate failed, cannot connect to 'https://vm-idm-018.testrelm.test:443/ca/agent/ca/displayBySerial': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135152: Request for certificate failed, cannot connect to 'https://vm-idm-018.testrelm.test:443/ca/agent/ca/displayBySerial': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135142: Request for certificate failed, cannot connect to 'https://vm-idm-018.testrelm.test:443/ca/agent/ca/displayBySerial': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190711130443: Request for certificate failed, cannot connect to 'https://vm-idm-018.testrelm.test:443/ca/agent/ca/displayBySerial': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)
SUCCESS: ipahealthcheck.ipa.certs.IPACertmongerCA.IPA
SUCCESS: ipahealthcheck.ipa.certs.IPACertmongerCA.dogtag-ipa-ca-renew-agent
SUCCESS: ipahealthcheck.ipa.certs.IPACertmongerCA.dogtag-ipa-ca-renew-agent-reuse
[root@vm-idm-018 ipa]#

After certificate expiration issue fixed:
----------------------------------------
[root@vm-idm-018 ipa]# ipa-healthcheck --source ipahealthcheck.ipa.certs --output-type human
SUCCESS: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711130443
SUCCESS: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711135122
SUCCESS: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711135126
SUCCESS: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711135129
SUCCESS: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711135132
SUCCESS: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711135135
SUCCESS: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711135136
SUCCESS: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711135142
SUCCESS: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190711135152
SUCCESS: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711130443
SUCCESS: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711135122
SUCCESS: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711135126
SUCCESS: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711135129
SUCCESS: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711135132
SUCCESS: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711135135
SUCCESS: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711135136
SUCCESS: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711135142
SUCCESS: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190711135152
SUCCESS: ipahealthcheck.ipa.certs.IPACertTracking.20190711135135
SUCCESS: ipahealthcheck.ipa.certs.IPACertTracking.20190711135122
SUCCESS: ipahealthcheck.ipa.certs.IPACertTracking.20190711135126
SUCCESS: ipahealthcheck.ipa.certs.IPACertTracking.20190711135129
SUCCESS: ipahealthcheck.ipa.certs.IPACertTracking.20190711135132
SUCCESS: ipahealthcheck.ipa.certs.IPACertTracking.20190711135136
SUCCESS: ipahealthcheck.ipa.certs.IPACertTracking.20190711135152
SUCCESS: ipahealthcheck.ipa.certs.IPACertTracking.20190711135142
SUCCESS: ipahealthcheck.ipa.certs.IPACertTracking.20190711130443
SUCCESS: ipahealthcheck.ipa.certs.IPACertNSSTrust.caSigningCert cert-pki-ca
SUCCESS: ipahealthcheck.ipa.certs.IPACertNSSTrust.ocspSigningCert cert-pki-ca
SUCCESS: ipahealthcheck.ipa.certs.IPACertNSSTrust.subsystemCert cert-pki-ca
SUCCESS: ipahealthcheck.ipa.certs.IPACertNSSTrust.auditSigningCert cert-pki-ca
SUCCESS: ipahealthcheck.ipa.certs.IPACertNSSTrust.Server-Cert cert-pki-ca
SUCCESS: ipahealthcheck.ipa.certs.IPANSSChainValidation./etc/pki/pki-tomcat/alias:Server-Cert cert-pki-ca
SUCCESS: ipahealthcheck.ipa.certs.IPANSSChainValidation./etc/dirsrv/slapd-TESTRELM-TEST:Server-Cert
SUCCESS: ipahealthcheck.ipa.certs.IPAOpenSSLChainValidation./var/lib/ipa/certs/httpd.crt
SUCCESS: ipahealthcheck.ipa.certs.IPAOpenSSLChainValidation./var/lib/ipa/ra-agent.pem
SUCCESS: ipahealthcheck.ipa.certs.IPARAAgent
SUCCESS: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135135
SUCCESS: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135122
SUCCESS: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135126
SUCCESS: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135129
SUCCESS: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135132
SUCCESS: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135136
SUCCESS: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135152
SUCCESS: ipahealthcheck.ipa.certs.IPACertRevocation.20190711135142
SUCCESS: ipahealthcheck.ipa.certs.IPACertRevocation.20190711130443
SUCCESS: ipahealthcheck.ipa.certs.IPACertmongerCA.IPA
SUCCESS: ipahealthcheck.ipa.certs.IPACertmongerCA.dogtag-ipa-ca-renew-agent
SUCCESS: ipahealthcheck.ipa.certs.IPACertmongerCA.dogtag-ipa-ca-renew-agent-reuse
[root@vm-idm-018 ipa]#

9) System with Trust:
-----------------------
[root@kvm-02-guest15 ~]# ipa-healthcheck --source ipahealthcheck.ipa.trust --output-type human
SUCCESS: ipahealthcheck.ipa.trust.IPATrustAgentCheck
SUCCESS: ipahealthcheck.ipa.trust.IPATrustDomainsCheck
SUCCESS: ipahealthcheck.ipa.trust.IPATrustCatalogCheck
SUCCESS: ipahealthcheck.ipa.trust.IPAsidgenpluginCheck
SUCCESS: ipahealthcheck.ipa.trust.IPATrustAgentMemberCheck
SUCCESS: ipahealthcheck.ipa.trust.IPATrustControllerPrincipalCheck
SUCCESS: ipahealthcheck.ipa.trust.IPATrustControllerServiceCheck
SUCCESS: ipahealthcheck.ipa.trust.IPATrustControllerConfCheck
SUCCESS: ipahealthcheck.ipa.trust.IPATrustControllerGroupSIDCheck

Thus on basis of above observations, the basic tier1 tests PASSED, marking status of bug to "VERIFIED"

Comment 26 errata-xmlrpc 2019-11-05 20:52:26 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3348

Note You need to log in before you can comment on or make changes to this bug.