Bug 1518376

It appears this was the problem... 

WARNING: Your system is running out of entropy, you may experience long delays

The rngd service failed:

Starting Hardware RNG Entropy Gatherer Daemon...
read error
hwrng: no available rng
Unable to open file: /dev/tpm0
can't open any entropy source
Maybe RNG device modules are not loaded
rngd.service: main process exited, code=exited, status=1/FAILURE
Unit rngd.service entered failed state.
rngd.service failed.

*** This bug has been marked as a duplicate of bug 1464200 ***

I'm reopening this up because I've had the same problem in RHEL7.4 and
now RHEL 7.5 nightly and with these last two times there was
no ngd.service failure... 

Plus the system is been eaten alive by these processes now 
and when I reboot (if I don't uninstall the ipaserver).

14764 ?        R      0:33 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
14765 ?        R      0:34 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
14766 ?        R      0:33 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
14767 ?        R      0:33 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
14768 ?        R      0:33 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
14769 ?        R      0:33 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
14770 ?        R      0:32 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
14771 ?        R      0:32 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit

Here is the screen shot:
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/29]: configuring certificate server instance
  [2/29]: exporting Dogtag certificate store pin
  [3/29]: stopping certificate server instance to update CS.cfg
  [4/29]: backing up CS.cfg
  [5/29]: disabling nonces
  [6/29]: set up CRL publishing
  [7/29]: enable PKIX certificate path discovery and validation
  [8/29]: starting certificate server instance
  [9/29]: configure certmonger for renewals
  [10/29]: requesting RA certificate from CA
  [error] RuntimeError: request timed out
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    request timed out
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

The updated log are under
    http://people.redhat.com/steved/.bz1518376/

Hi,

can you check on your system if there is a file /var/run/ipa/renewal.lock and what is its content?

On a clean system, this file is either not present or contains the following:
# cat /var/run/ipa/renewal.lock
[lock]
locked = 0

The file may be a left-over from the failing installation, and if it contains locked=1, then the request for a RA certificate will wait for the lock to be released and exit on timeout.

The workaround in this case is either to wait for long enough, or to delete the file and re-launch ipa-server-install.

Note that you may need to remove the cert requests manually after a failing installation:
ipa-server-install --uninstall -U
getcert list
^^ If this command returns requests, you must run getcert stop-tracking <ID> for each remaining request.

(In reply to Florence Blanc-Renaud from comment #5)
> Hi,
> 
> can you check on your system if there is a file /var/run/ipa/renewal.lock
> and what is its content?
> 
> On a clean system, this file is either not present or contains the following:
> # cat /var/run/ipa/renewal.lock
> [lock]
> locked = 0
The file exists... 
ipa# cat /var/run/ipa/renewal.lock
[lock]
locked = 0

> 
> The file may be a left-over from the failing installation, and if it
> contains locked=1, then the request for a RA certificate will wait for the
> lock to be released and exit on timeout.
> 
> The workaround in this case is either to wait for long enough, or to delete
> the file and re-launch ipa-server-install.
I'll try removing the file and reinstalling.

> 
> Note that you may need to remove the cert requests manually after a failing
> installation:
> ipa-server-install --uninstall -U
> getcert list
> ^^ If this command returns requests, you must run getcert stop-tracking <ID>
> for each remaining request.
Its appears this is good....

ipa# getcert list
Number of certificates and requests being tracked: 0.

Thanks!

I tried again... with the same timeout

(In reply to Steve Dickson from comment #6)
> (In reply to Florence Blanc-Renaud from comment #5)
> > Hi,
> > 
> > can you check on your system if there is a file /var/run/ipa/renewal.lock
> > and what is its content?
> > 
> > On a clean system, this file is either not present or contains the following:
> > # cat /var/run/ipa/renewal.lock
> > [lock]
> > locked = 0
> The file exists... 
> ipa# cat /var/run/ipa/renewal.lock
> [lock]
> locked = 0
This file exists in the above state.

> 
> > 
> > Note that you may need to remove the cert requests manually after a failing
> > installation:
> > ipa-server-install --uninstall -U
> > getcert list
> > ^^ If this command returns requests, you must run getcert stop-tracking <ID>
> > for each remaining request.
> Its appears this is good....
> 
> ipa# getcert list
> Number of certificates and requests being tracked: 0.
This now has something in it... 
ipa# getcert list
Number of certificates and requests being tracked: 1.
Request ID '20171206164006':
	status: MONITORING
	stuck: no
	key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
	certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
	CA: dogtag-ipa-ca-renew-agent
	issuer: CN=Certificate Authority,O=BOSTON.DEVEL.REDHAT.COM
	subject: CN=IPA RA,O=BOSTON.DEVEL.REDHAT.COM
	expires: 2019-11-26 16:40:30 UTC
	key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
	eku: id-kp-serverAuth,id-kp-clientAuth
	pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
	post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
	track: yes
	auto-renew: yes

Are you able to install a new RHEL7.5 ipa-server?

Hi,

from the logs provided, it looks like your system is slow delivering the certificate:
2017-12-05T18:45:51Z DEBUG certmonger request is in state dbus.String(u'GENERATING_KEY_PAIR', variant_level=1)
...
2017-12-05T18:51:29Z DEBUG certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1)
2017-12-05T18:51:37Z DEBUG Traceback (most recent call last):

This means that it needs more that 5min to deliver the cert, and is above the default timeout (300s).

You can configure the timeout by creating a file /etc/ipa/installer.conf with the following content:

# cat /etc/ipa/installer.conf
[global]
startup_timeout=800

(the timeout is in seconds and can be tuned for your system)
and retry ipa-server-install with this setting.

Please let me know if it solves your problem.

Here is some strangeness that might interesting... 

In perpetration doing the install with the bumped up
timeout value I did a uninstall and got the following 
error:

ipa# ipa-server-install --uninstall -U
Shutting down all IPA services
Unconfiguring ntpd
Configuring certmonger to stop tracking system certificates for KRA
Configuring certmonger to stop tracking system certificates for CA
Unconfiguring CA
Unconfiguring krb5kdc
Unconfiguring kadmin
Unconfiguring directory server
ipa         : ERROR    Unable to find server cert nickname in /etc/dirsrv/slapd-BOSTON-DEVEL-REDHAT-COM/dse.ldif
Removing IPA client configuration
Removing Kerberos service principals from /etc/krb5.keytab
Disabling client Kerberos and LDAP configurations
Redundant SSSD configuration file /etc/sssd/sssd.conf was moved to /etc/sssd/sssd.conf.deleted
nscd daemon is not installed, skip configuration
nslcd daemon is not installed, skip configuration
Client uninstall complete.
The ipa-client-install command was successful

Then when I rebooted I got the following error:

[FAILED] Failed to start 389 Directory Server BOSTON-DEVEL-REDHAT-COM..
See 'systemctl status dirsrv' for details.

So maybe something is not being cleaned up correctly??

(In reply to Steve Dickson from comment #9)
>
> Then when I rebooted I got the following error:
> 
> [FAILED] Failed to start 389 Directory Server BOSTON-DEVEL-REDHAT-COM..
> See 'systemctl status dirsrv' for details.
> 
> So maybe something is not being cleaned up correctly??

Note, even when I yum remove ipa-server, I still got this error.

You may try
systemctl disable dirsrv

(the service unit was probably not removed by the uninstall).
When I see errors in the uninstall, I usually relaunch the same command a second time to make sure that everything was removed.

Well it did get a little further by upping the timeout

onfiguring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/29]: configuring certificate server instance
  [2/29]: exporting Dogtag certificate store pin
  [3/29]: stopping certificate server instance to update CS.cfg
  [4/29]: backing up CS.cfg
  [5/29]: disabling nonces
  [6/29]: set up CRL publishing
  [7/29]: enable PKIX certificate path discovery and validation
  [8/29]: starting certificate server instance
  [9/29]: configure certmonger for renewals
  [10/29]: requesting RA certificate from CA
  [11/29]: setting up signing cert profile
  [12/29]: setting audit signing renewal to 2 years
  [13/29]: restarting certificate server
  [error] CalledProcessError: Command '/bin/systemctl restart pki-tomcatd' returned non-zero exit status 1
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    Command '/bin/systemctl restart pki-tomcatd' returned non-zero exit status 1
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

The install log
   http://people.redhat.com/steved/.bz1518376/ipaserver-install.log

(In reply to Florence Blanc-Renaud from comment #11)
> You may try
> systemctl disable dirsrv
> 
> (the service unit was probably not removed by the uninstall).
> When I see errors in the uninstall, I usually relaunch the same command a
> second time to make sure that everything was removed.

When I removed the 389-ds-base package, the ipa-server
also got removed. So when I installed the ipa-server 
the 389-ds-base was install. 

I'll go ahead and uninstall the server (twice if 
necessary), disable  dirsrv@BOSTON-DEVEL-REDHAT-COM,
reboot and then try another install with the
bumped up timeout

Thanks for the help!

(In reply to Florence Blanc-Renaud from comment #11)
> 
> (the service unit was probably not removed by the uninstall).
> When I see errors in the uninstall, I usually relaunch the same command a
> second time to make sure that everything was removed.
Here is what it looks when I do the uninstall twice
(I'm thinking this is all expected)

ipa# ipa-server-install --uninstall -U
WARNING:
IPA server is not configured on this system. If you want to install the
IPA server, please install it using 'ipa-server-install'.

WARNING: Failed to connect to Directory Server to find information about
replication agreements. Uninstallation will continue despite the possible
existing replication agreements.

If this server is the last instance of CA,
KRA, or DNSSEC master, uninstallation may result in data loss.
Shutting down all IPA services
Configuring certmonger to stop tracking system certificates for KRA
Configuring certmonger to stop tracking system certificates for CA
Removing IPA client configuration
IPA client is not configured on this system.
The ipa-client-install command failed. See /var/log/ipaclient-uninstall.log for more information

http://people.redhat.com/steved/.bz1518376/ipaclient-uninstall.log

(In reply to Steve Dickson from comment #13)
> (In reply to Florence Blanc-Renaud from comment #11)
> > You may try
> > systemctl disable dirsrv
> > 
> > (the service unit was probably not removed by the uninstall).
> > When I see errors in the uninstall, I usually relaunch the same command a
> > second time to make sure that everything was removed.
> 
> When I removed the 389-ds-base package, the ipa-server
> also got removed. So when I installed the ipa-server 
> the 389-ds-base was install. 
> 
> I'll go ahead and uninstall the server (twice if 
> necessary), disable  dirsrv@BOSTON-DEVEL-REDHAT-COM,
> reboot and then try another install with the
> bumped up timeout
From an IRC conversion it was suggested I beef up my VM,
which I did... 8 cpus 13k memory (most I can do) but
unfortunately I got the same error as in Comment 7

Again, I'll ask has anybody had any success installing
an ipa server with the latest RHEL 7.5 bits? 

I have been trying for the last week with no success...

Hi,

I just tried to install with ipa-server 4.5.4-6.el7 on RHEL 7.5 and succeeded on our lab.
The default lab machines for RHEL 7.5 have 4 CPUs and 3GB RAM (the official doc recommends at least 2 GB of RAM and 1 GB swap space).

Hi,

were you able to retry with enough memory? Please share your logs if the installation is still failing (if the issue is still about pki-tomcatd failing to restart, we will also need the logs from /var/log/pki).

(In reply to Florence Blanc-Renaud from comment #17)
> Hi,
> 
> were you able to retry with enough memory? Please share your logs if the
> installation is still failing (if the issue is still about pki-tomcatd
> failing to restart, we will also need the logs from /var/log/pki).

After I do the install the machine is eaten alive by the
dogtag-ipa-ca-renew-agent-submit process 

ipa$ ps ax | grep dogtag
 1389 ?        R      0:54 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
 1390 ?        R      0:54 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
 1391 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
 1392 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
 1395 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
 1399 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
 1400 ?        R      0:54 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
 1401 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
 1402 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing
 1403 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing
 1405 ?        R      0:54 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing
 1406 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing
 1407 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing
 1409 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing
 1410 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing
 1411 ?        R      0:53 /usr/bin/python2 -E /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing

ipa$ uptime
 09:04:31 up 7 min,  1 user,  load average: 19.57, 15.56, 7.53

I'm not sure how to proceed with this. It isn't something we've been able to reproduce independently. Given this is a VM is there a chance you can try provisioning a fresh one and trying there?

(In reply to Rob Crittenden from comment #21)
> I'm not sure how to proceed with this. It isn't something we've been able to
> reproduce independently. Given this is a VM is there a chance you can try
> provisioning a fresh one and trying there?

I was able to do an install on a different machine and
different vm... I think we can close this bz