Bug 1810683 - unable to add new IdM servers under RHEL 7.6 and now also under 7.7 after upgrading for possible fixes
Summary: unable to add new IdM servers under RHEL 7.6 and now also under 7.7 after upg...
Status: CLOSED DUPLICATE of bug 1754845
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ipa
Version: 7.7
Hardware: Unspecified
OS: Unspecified
Target Milestone: rc
: ---
Assignee: Florence Blanc-Renaud
QA Contact: ipa-qe
Depends On:
TreeView+ depends on / blocked
Reported: 2020-03-05 17:49 UTC by Dave
Modified: 2020-04-28 22:17 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2020-04-28 22:17:51 UTC
Target Upstream Version:

Attachments (Terms of Use)
kra & ipa logs (356.38 KB, application/x-xz)
2020-03-05 18:42 UTC, Dave
no flags Details

Description Dave 2020-03-05 17:49:32 UTC
Description of problem:
unable to add new IdM servers to existing environment
(same issue under 7.6, upgraded to 7.7 (current), with identical issue) 

Version-Release number of selected component (if applicable):

How reproducible:
100% - we have added some, currently 7 (4 data centers), but cannot add any further (need to add a 2nd to new Data Center, and then another pair in another new Data Center)

Steps to Reproduce:
1. existing IdM env with full options installed/enabled
# ipa-server-install --mkhomedir --setup-kra --setup-adtrust --setup-ca --ssh-trust-dns --setup-dns --auto-reverse --no-dnssec-validation 
(and AD trust was added, one forest with 9 domains)

2. add replicas (same options as existing servers)
2a. start with required client piece
# ipa-client-install --mkhomedir --enable-dns-updates --no-krb5-offline-passwords --ssh-trust-dns --force-ntpd

2b. promote to server/replica
# ipa-replica-install --mkhomedir --setup-kra --setup-adtrust --setup-ca --ssh-trust-dns --setup-dns --auto-reverse --no-dnssec-validation 
OR (split out the KRA piece with is causing the problem, after we have a working replica)
# ipa-replica-install --mkhomedir --setup-adtrust --setup-ca --ssh-trust-dns --setup-dns --auto-reverse --no-dnssec-validation 
# ipa-kra-install

Actual results:
The ipa-kra-install command failed, exception: RuntimeError: KRA configuration failed.
KRA configuration failed.
The ipa-kra-install command failed. See /var/log/ipaserver-kra-install.log for more information

Expected results:
ipa-replica-install with KRA completes and is successful

Additional info:

[28/Feb/2020:13:56:02][localhost-startStop-1]: ============================================
[28/Feb/2020:13:56:02][localhost-startStop-1]: =====  DEBUG SUBSYSTEM INITIALIZED   =======
[28/Feb/2020:13:56:02][localhost-startStop-1]: ============================================
[28/Feb/2020:13:56:02][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false
[28/Feb/2020:13:56:02][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
[28/Feb/2020:13:56:02][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-tomcat
[28/Feb/2020:13:56:02][localhost-startStop-1]: CMSEngine: cert not found:auditSigningCert cert-pki-tomcat
[28/Feb/2020:13:56:02][localhost-startStop-1]: CMSEngine: Exception:org.mozilla.jss.crypto.ObjectNotFoundException: Certificate not found: auditSigningCert cert-pki-tomcat
[28/Feb/2020:13:56:02][localhost-startStop-1]: CMSEngine: done init id=debug
[28/Feb/2020:13:56:02][localhost-startStop-1]: CMSEngine: initialized debug
[28/Feb/2020:13:56:02][localhost-startStop-1]: CMSEngine: initSubsystem id=log
[28/Feb/2020:13:56:02][localhost-startStop-1]: CMSEngine: ready to init id=log
[28/Feb/2020:13:56:02][localhost-startStop-1]: Event filters:
[28/Feb/2020:13:56:02][localhost-startStop-1]:  - ASYMKEY_GENERATION_REQUEST: (Outcome=Failure)
[28/Feb/2020:13:56:02][localhost-startStop-1]:  - ASYMKEY_GENERATION_REQUEST_PROCESSED: (Outcome=Failure)
[28/Feb/2020:13:56:02][localhost-startStop-1]:  - KEY_GEN_ASYMMETRIC: (Outcome=Failure)
[28/Feb/2020:13:56:02][localhost-startStop-1]:  - KEY_RECOVERY_AGENT_LOGIN: (Outcome=Failure)
[28/Feb/2020:13:56:02][localhost-startStop-1]:  - RANDOM_GENERATION: (Outcome=Failure)
[28/Feb/2020:13:56:02][localhost-startStop-1]:  - SECURITY_DATA_ARCHIVAL_REQUEST: (Outcome=Failure)
[28/Feb/2020:13:56:02][localhost-startStop-1]:  - SECURITY_DATA_ARCHIVAL_REQUEST_PROCESSED: (Outcome=Failure)
[28/Feb/2020:13:56:02][localhost-startStop-1]:  - SECURITY_DATA_RECOVERY_REQUEST: (Outcome=Failure)
[28/Feb/2020:13:56:02][localhost-startStop-1]:  - SECURITY_DATA_RECOVERY_REQUEST_PROCESSED: (Outcome=Failure)
[28/Feb/2020:13:56:02][localhost-startStop-1]:  - SECURITY_DATA_RECOVERY_REQUEST_STATE_CHANGE: (Outcome=Failure)
[28/Feb/2020:13:56:02][localhost-startStop-1]:  - SERVER_SIDE_KEYGEN_REQUEST: (Outcome=Failure)
[28/Feb/2020:13:56:02][localhost-startStop-1]:  - SERVER_SIDE_KEYGEN_REQUEST_PROCESSED: (Outcome=Failure)
[28/Feb/2020:13:56:02][localhost-startStop-1]:  - SYMKEY_GENERATION_REQUEST: (Outcome=Failure)
[28/Feb/2020:13:56:02][localhost-startStop-1]:  - SYMKEY_GENERATION_REQUEST_PROCESSED: (Outcome=Failure)

Can provide additional logs/additional logging levels/etc, as recommended

Comment 2 Alexander Bokovoy 2020-03-05 18:34:04 UTC
Please provide additional logs (Dogtag logs and IPA installation logs from /var/log/).

Fraser, could you please look into them?

Comment 3 Dave 2020-03-05 18:42:10 UTC
Created attachment 1667875 [details]
kra & ipa logs

Comment 14 Fraser Tweedale 2020-03-12 05:38:08 UTC
It seems likely to be a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1754845.
Customer should update to pki-core-10.5.16-6.el7_7 on all servers.

Comment 15 Dave 2020-03-26 15:21:36 UTC
(In reply to Fraser Tweedale from comment #14)
> It seems likely to be a duplicate of
> https://bugzilla.redhat.com/show_bug.cgi?id=1754845.
> Customer should update to pki-core-10.5.16-6.el7_7 on all servers.

updated to latest 7.7 channel.. we still cannot bring up a replica, it's failing in a different location during the replica install, and this time it had not yet created a /var/log/kra directory.

this time we were joining awsw-p-aci-prdipa13 and it was attaching to the cl-rhm-0253 IPA server

[root@awsw-p-aci-prdipa13 ~]# rpm -q pki-server ipa-server

[root@cl-rhm-0253 ~]# rpm -q pki-server ipa-server

attaching ipa install logs from 13 and kra logs from 253

Comment 21 Dave 2020-03-27 11:30:23 UTC
so, this system was successfully a server (replica) in the past. We uninstalled it to test the new rpm's. The problem is doing any testing in production. Partly because we've brought production down at least twice trying to add a replica, where it added itself to the SRV records but failed to fully become a server, and kerberos was broken, so we couldn't get to the UI, nor could we kinit as admin command-line to attempt to fix any thing.

Currently, w/the virus situation, they are critical of doing any changes to live systems for fear the pertinent people may become unavailable.

Anyhow, the cert has an old date from when it was previously a replica:

Valid from:	Fri Feb 28 12:03:10 2020 UTC
Valid to:	Mon Feb 28 12:03:10 2022 UTC

Even so, it would seem it should be able to manage a cert of the same name, it was given admin rights to configure everything it needed to become a full server.

We 'could' uninstall it, make sure there are no remnants for this system (except in the CA), and try again, or should we hold off so we could troubleshoot further?

Comment 26 Fraser Tweedale 2020-04-23 08:26:28 UTC
The case moved on from the initial PKI-related issue (updating to 7.7.z resolved it).
The current issue is quite different.

The case has also been inactive (waiting on customer) for a couple of weeks.  What is
the next step?  Can we close this BZ?

Comment 27 Dave 2020-04-28 16:39:52 UTC
the KRA issue was resolved with the new rpm's, we have gotten past the problem system, and have added 4 new servers since applying the new rpm's

uninstall/mangedby issue cloned to a new BZ

this one can be closed as resolved

Comment 28 Fraser Tweedale 2020-04-28 22:17:51 UTC
Thanks; closing as duplicate.

*** This bug has been marked as a duplicate of bug 1754845 ***

Note You need to log in before you can comment on or make changes to this bug.