Bug 862437 - Cert install errors
Cert install errors
Status: CLOSED DUPLICATE of bug 817080
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: ipa (Show other bugs)
Unspecified Unspecified
medium Severity unspecified
: rc
: ---
Assigned To: Rob Crittenden
Depends On:
  Show dependency treegraph
Reported: 2012-10-02 17:56 EDT by Jesse Triplett
Modified: 2012-12-10 17:13 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2012-12-10 17:13:55 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
The DEbug Logs from the Replica Installation (147.81 KB, text/plain)
2012-10-22 18:30 EDT, Jesse Triplett
no flags Details
The INstallation Logs from the Replica Installation (72.51 KB, text/plain)
2012-10-22 18:30 EDT, Jesse Triplett
no flags Details

  None (edit)
Description Jesse Triplett 2012-10-02 17:56:12 EDT
Description of problem:

The customer is running a RHEL 6.2 IPA server and during a replica promotion there is a failure.  The IPA is versions 389-ds-base- and the error is:

  root        : CRITICAL failed to restart ds instance Command '/usr/sbin/setup-ds.pl --silent --logfile - -f /tmp/tmp3prmP5' returned non-zero exit status 1
  [3/3]: restarting directory server
done configuring pkids.
root        : ERROR    Didn't get new certmonger request, got Request "20120616171834" modified.

That is from the install logs for IPA.  I have attached an strace with 2048 strings and have requested a 4096 string strace.  

How reproducible:

Not yet known
Actual results:

Additional info:
Comment 2 Dmitri Pal 2012-10-04 18:23:53 EDT
What do you mean "promotion"?

This seems like the re-install issue.
See: https://bugzilla.redhat.com/show_bug.cgi?id=817080

Cleaning the certmonger directories would solve the issue.
I will link this to existing ticket and kick it back into needs triage.
Comment 3 Dmitri Pal 2012-10-04 18:24:56 EDT
Upstream ticket:
Comment 4 Rob Crittenden 2012-10-05 10:55:29 EDT
I'm going to guess that they previously tried to install IPA on this machine and it failed, or it was a client and something went wrong with the uninstall (in either case).

What has happened is certmonger is already tracking a certificate that we're trying to track.

To fix this, run:

# ipa-server-install --uninstall -U
# service certmonger start
# ipa-getcert list

There may be more tracked requests than just 20120616171834. If they have not specifically tracked these certificates themselves it should be safe to untrack them.

To do so, for each one run:

# ipa-getcert stop-tracking -i <request_id>
Comment 5 Jesse Triplett 2012-10-05 11:06:22 EDT
Will that uninstall the IPA server?  Do they need to reinstall the ipa server afterwards?
Comment 6 Rob Crittenden 2012-10-05 11:09:17 EDT
To be safe, can you describe what you mean by promotion and what the exactly is being done here?

I based my comment on the fact that this looked like a standard replica installation. If this is true then you run the uninstall on the replica machine, then do the clean up as I suggested.
Comment 7 Jesse Triplett 2012-10-05 11:17:19 EDT
So they need to install a replica yes.  They got to installing the CA cert but "the CA fails to start up properly on the replica server."  IPA is one of my huge knowledge gaps over here in IDM. The promotion of the replica to master status is the end goal here but they aren't to the point of having the replica installed and running yet.

If your recomendation is to perform these steps I will pass them along.
Comment 8 Rob Crittenden 2012-10-05 12:52:59 EDT
There is no promotion of replica to master. Installing another IPA server using ipa-replica-prepare/ipa-replica-install installs another IPA master server.

Yes, have them run these commands on the replica. If the installation of the CA fails they will need to look at /var/log/ipareplica-install.log for additional information. We don't get much out of the CA installer except for a pass/fail so are unable to provide a lot of guidance about what happened.
Comment 9 Jesse Triplett 2012-10-08 12:10:14 EDT
Just spoke with the customer.  They are going to run these commands Friday night. 

  # ipa-server-install --uninstall -U
  # service certmonger start
  # ipa-getcert list

Thanks for your help.  I'll let you know the outcome next week.
Comment 10 Jesse Triplett 2012-10-16 16:54:34 EDT
Just to clarify, the customer was talking about the CA authority promotion.  That is where the process is failing.  They tried the testpatch that was given and the commands: 

  # ipa-server-install --uninstall -U
  # service certmonger start
  # ipa-getcert list

Which have not resolved the issue.  What is the next course of action here?

Thank You
Comment 12 Dmitri Pal 2012-10-16 20:06:27 EDT
(In reply to comment #10)
> Just to clarify, the customer was talking about the CA authority promotion. 
> That is where the process is failing.  They tried the testpatch that was
> given and the commands: 
>   # ipa-server-install --uninstall -U
>   # service certmonger start
>   # ipa-getcert list
> Which have not resolved the issue.  What is the next course of action here?
> Thank You

# ipa-getcert list
return the list of the certs currently being tracked and was the command 
# ipa-getcert stop-tracking -i <request_id> 
run for every cert listed?
Comment 13 Rob Crittenden 2012-10-17 09:51:31 EDT
I think you need to clarify what the customer is trying to do. They are trying to install a new replica to replace an existing master? If they are using a dogtag CA then there is no promotion, all CAs are masters. The documentation refers only to configuring which one generates a CRL and in 2.2, all masters generate a CRL.
Comment 14 Jesse Triplett 2012-10-17 10:38:12 EDT
So the customer has a single existing IPA server that they would like to decommision.  In its place they would like to have 2 redundant ipa servers.  The issue comes when, during the installation of one of the new servers, they try to promote it to be the Master CA. Does that answer your question?
Comment 15 Rob Crittenden 2012-10-17 10:52:35 EDT
Yes, but it conflates two issues. This problem has nothing to do with promoting a master, it has to do with replica installation. From what I can tell a previous installation failed for some reason and uninstall did not completely remove everything. This is preventing re-install. If they stop tracking the certificates that should allow reinstallation.
Comment 16 Jesse Triplett 2012-10-17 10:59:05 EDT
Agreed. However, the steps to stop tracking the certs (eg: # ipa-server-install --uninstall -U ; # service certmonger start ; # ipa-getcert list) did not resolve the issue. Would it behoove the customer at this point to completely reinstall or is there maybe another way you know of to try to stop the system from tracking the certs?

-----Begin Copied Message-----
# ipa-getcert list
return the list of the certs currently being tracked and was the command 
# ipa-getcert stop-tracking -i <request_id> 
run for every cert listed?
-----End Copied Message-----

I will ask the customer these questions and get back to you on that.
Comment 17 Jesse Triplett 2012-10-18 12:58:15 EDT
From the customer:
-----Begin Copied Message-----
The ipa-getcert list returned 0 entries.  Here's the output we saw before trying the reinstall.

# service certmonger start
Starting certmonger:                                       [  OK  ]
# ipa-getcert list
Number of certificates and requests being tracked: 0.

-----End Copied Message-----

Is there anything else we can have them try?
Comment 18 Rob Crittenden 2012-10-18 13:26:48 EDT
They should try the replica installation again then. If it fails we'll need to see the full /var/log/ipareplica-install.log.
Comment 19 Jesse Triplett 2012-10-18 17:30:34 EDT
The customer has been asked to perform the replica installation again and upload the /var/log/ipareplica-install.log when complete.  Thanks for your help so far
Comment 22 Jesse Triplett 2012-10-19 18:33:12 EDT
The customer is taring them up.  I don't expect to have them until Monday but they may be in this evening.
Comment 23 Jesse Triplett 2012-10-22 18:30:01 EDT
Created attachment 631761 [details]
The DEbug Logs from the Replica Installation
Comment 24 Jesse Triplett 2012-10-22 18:30:34 EDT
Created attachment 631762 [details]
The INstallation Logs from the Replica Installation
Comment 25 Jesse Triplett 2012-10-22 18:32:43 EDT
Hello, I've uploaded the ipa replica installation and debug logs to the BZ.  I'm going to look over them myself.

Comment 28 Jesse Triplett 2012-10-23 14:29:39 EDT
In talking to the customer, I found out that this server used to serve another purpose and it is possible that the ca got corrupted somehow. Here is a couple of questions that could possibly lead to a fix for them:

How is the cert generated from dogtag?  Does dogtags just export the private key?
Is there a way to have the master ca generate a new ca that could be re-prepared for a replica install?


Comment 29 Rob Crittenden 2012-10-23 14:35:49 EDT
The prepared file contains a number of different things, including SSL certificates for the web and directory servers, the RA agent certificate and private key and the CA keys (the file is encrypted).

You can always prepare a new file for this replica using ipa-replica-prepare.

There seems to be some evidence of a lunasa HSM on the machine, can you confirm?
Comment 30 Jesse Triplett 2012-10-23 17:39:46 EDT
From the customer:

#75 Created By: Fiacchino, jim  (10/23/2012 4:14 PM) Last Modified By: Fiacchino, jim  (10/23/2012 4:14 PM) 
No, lunasa isn't installed on the server.

The customer's question regarding regenerating the ca cert from the ca authority on the IPA master was from them feeling that maybe there was corruption in the master's certificate system.  How likely do you feel it is that corruption of this nature could be the issue?
Comment 31 Rob Crittenden 2012-10-24 17:40:01 EDT
I suppose that it is possible that the wrong CA was provided to ipa-replica-prepare. It uses the file /root/ca.p12. It is expected that this is the CA as generated during the initial CA installation.
Comment 32 Jesse Triplett 2012-10-24 17:43:43 EDT
Is there a way to have the master CA generate a new CA?  If so we could re-prepare the file and retry in the installation.
Comment 33 Rob Crittenden 2012-10-24 17:49:40 EDT
There should be a way to regenerate that PKCS#12 file. It is provided to us at the end of the CA installation. I'm hoping that one of the CS engineers cc'd here will chime in with instructions on how to regenerate this file. We recommend at the end of the IPA server installation that this file be safely stored some place.
Comment 36 Rob Crittenden 2012-10-25 17:13:56 EDT
In an IPA context this is what you need to do:

On a working master with a CA installed:

- Get the internal password from /var/lib/pki-ca/conf/password.conf and put it into the file /tmp/internal.txt
- Create the file /tmp/password.txt and put your Directory Manager password into it
- run: PKCS12Export -debug -d /var/lib/pki-ca/alias -w /tmp/password.txt -p /tmp/internal.txt -o cacert.p12
- rm -f /tmp/password.txt /tmp/internal.txt

Copy/move cacert.p12 to /root (you may want to save off a copy of whatever is there, I'm conservative).

On that master re-run: ipa-replica-prepare

Use the new prepared file to try to install a replica on the non-working server.

We'd like to see the list of certificates in the CA NSS certificate database. Can you provide the output of this on the working master:

# certutil -L -d /var/lib/pki-ca/alias
Comment 38 Jesse Triplett 2012-10-26 14:14:38 EDT
Here are the requested certificates:

Certificate Nickname                                         Trust Attributes

caSigningCert cert-pki-ca                                    CTu,Cu,Cu
Server-Cert cert-pki-ca                                      u,u,u
auditSigningCert cert-pki-ca                                 u,u,Pu
ocspSigningCert cert-pki-ca                                  u,u,u
subsystemCert cert-pki-ca                                    u,u,u
Comment 41 Rob Crittenden 2012-12-10 17:13:55 EST
Closing as a duplicate of ​817080.

The uninstaller was enhanced to look for remaining certificates tracked by certmonger when the uninstall finishes. The user will be notified if any remain.

*** This bug has been marked as a duplicate of bug 817080 ***

Note You need to log in before you can comment on or make changes to this bug.