Bug 986985 - Upgrade from 3.2.0-11.30 to 3.2.0-11.37 fails during 'Preparing CA' stage.
Upgrade from 3.2.0-11.30 to 3.2.0-11.37 fails during 'Preparing CA' stage.
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine-setup (Show other bugs)
3.2.0
All Linux
urgent Severity high
: ---
: 3.2.2
Assigned To: Alon Bar-Lev
Pavel Stehlik
integration
: ZStream
Depends On: 980926
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-22 10:36 EDT by Idith Tal-Kohen
Modified: 2013-11-27 19:10 EST (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously the PKI CA creation script used the keytool utitlity directly, which could cause compatibility issues and failure to upgrade the Manager with the message "Error: Can't create trust store". The keytool from JAVA_HOME is now used for generating CA certificates. This error no longer presents.
Story Points: ---
Clone Of: 980926
Environment:
Last Closed: 2013-08-13 09:08:31 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 419503 None None None Never
Red Hat Knowledge Base (Solution) 442613 None None None Never
oVirt gerrit 16524 None None None Never

  None (edit)
Description Idith Tal-Kohen 2013-07-22 10:36:13 EDT
+++ This bug was initially created as a clone of Bug #980926 +++

Description of problem:

When upgrading from from RHEV-M 3.2.0-11.30 to 3.2.0-11.37, 'rhevm-upgrade' fails during 'Preparing CA' stage with the below error:

Preparing CA...                                      [ ERROR ]

 **Error: Upgrade failed, rolling back**
 **Reason: Error: Can't create trust store**


Version-Release number of selected component (if applicable):
rhevm-setup-3.2.0-11.37.el6ev.noarch

How reproducible:


Steps to Reproduce:
1. Update the rhevm-setup package to the version mentioned above. 
2. execute rhevm-upgrade 
3.

Actual results:

Installation fails with the error listed above. 


Expected results:

Upgrade to complete without error.


Additional info:

From ovirt-engine-upgrade*.log:

2013-07-03 09:27:17::DEBUG::rhevm-upgrade::542::root:: Converting truststore
2013-07-03 09:27:17::DEBUG::common_utils::453::root:: Executing command --> '/usr/bin/keytool -import -noprompt -keystore /etc/pki/ovirt-engine/.truststore.tm
p -storepass ******** -keypass ******** -alias cacert -trustcacerts -file /etc/pki/ovirt-engine/ca.pem' in working directory '/'
2013-07-03 09:27:18::DEBUG::common_utils::491::root:: output = keytool error: java.lang.Exception: Input not an X.509 certificate

2013-07-03 09:27:18::DEBUG::common_utils::492::root:: stderr = 
2013-07-03 09:27:18::DEBUG::common_utils::493::root:: retcode = 1
2013-07-03 09:27:18::ERROR::rhevm-upgrade::1337::root:: Traceback (most recent call last):
  File "/usr/bin/rhevm-upgrade", line 1331, in main
    runFunc([ca.prepare], MSG_INFO_PKI_PREPARE)
  File "/usr/bin/rhevm-upgrade", line 649, in runFunc
    func()
  File "/usr/bin/rhevm-upgrade", line 556, in prepare
    utils.execCmd(cmdList=cmd, maskList=mask, failOnError=True, msg=MSG_ERROR_FAILED_CREATE_TRUSTSTORE)
  File "/usr/share/ovirt-engine/scripts/common_utils.py", line 496, in execCmd
    raise Exception(msg)
Exception: Error: Can't create trust store


It apppears that the 'keytool' command is failing due to the fact that the x.509 certificate in /etc/pki/ovirt-engine/ca.pem is listed in both text form, and Base64 form. As a workaround, removing the text form and leaving what is between the ---BEGIN CERTIFICATE--- & ---END CERTIFICATE--- stanza (and then running rhevm-upgrade again), allows the upgrade to proceed.

--- Additional comment from Alon Bar-Lev on 2013-07-05 14:55:42 EDT ---

Hello,

Can you please attach: /etc/pki/ovirt-engine/ca.pem

Thanks,
Alon

--- Additional comment from Rich Jerrido on 2013-07-05 16:25:02 EDT ---



--- Additional comment from Alon Bar-Lev on 2013-07-05 16:51:16 EDT ---

(In reply to Rich Jerrido from comment #2)
> Created attachment 769382 [details]
> non-working ca.pem from RHEV-M

Thanks!

Certificate is valid.

Can you please see where keytool is pointing?

$ readlink -f /usr/bin/keytool

This is the final stroke before I enforce specific jre.

--- Additional comment from Rich Jerrido on 2013-07-05 17:24:21 EDT ---

(In reply to Alon Bar-Lev from comment #3)
> (In reply to Rich Jerrido from comment #2)
> > Created attachment 769382 [details]
> > non-working ca.pem from RHEV-M
> 
> Thanks!
> 
> Certificate is valid.
> 
> Can you please see where keytool is pointing?
> 
> $ readlink -f /usr/bin/keytool
> 
> This is the final stroke before I enforce specific jre.

$ readlink -f /usr/bin/keytool
/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25.x86_64/jre/bin/keytool

For what it's worth, (and as additional background), this system was upgraded from 3.1 to 3.2. During the upgrade process to 3.2, I did run into bz961081, due to having the IBM JRE installed. I have since removed that JRE (prior to this update & subsequent bz). So I don't know if this is residual cruft from that upgrade.

--- Additional comment from Alon Bar-Lev on 2013-07-05 17:28:32 EDT ---

(In reply to Rich Jerrido from comment #4)
> For what it's worth, (and as additional background), this system was
> upgraded from 3.1 to 3.2. During the upgrade process to 3.2, I did run into
> bz961081, due to having the IBM JRE installed.

Please vote for this bug solution to be included in 3.2.z :)

--- Additional comment from Alon Bar-Lev on 2013-07-05 17:33:20 EDT ---

(In reply to Rich Jerrido from comment #4)
> $ readlink -f /usr/bin/keytool
> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25.x86_64/jre/bin/keytool

Strange. I cannot reproduce this with same version.

What is the output of:

$ rm -f /tmp/ks1
$ keytool -import -keystore /tmp/ks1 -alias cacert -trustcacerts -file /etc/pki/ovirt-engine/ca.pem -storepass password -keypass password -noprompt

--- Additional comment from Rich Jerrido on 2013-07-08 11:12:13 EDT ---

Output of:

$ rm -f /tmp/ks1
$ keytool -import -keystore /tmp/ks1 -alias cacert -trustcacerts -file /etc/pki/ovirt-engine/ca.pem -storepass password -keypass password -noprompt
Certificate was added to keystore

--- Additional comment from Alon Bar-Lev on 2013-07-08 11:14:42 EDT ---

(In reply to Rich Jerrido from comment #7)
> Output of:
> 
> $ rm -f /tmp/ks1
> $ keytool -import -keystore /tmp/ks1 -alias cacert -trustcacerts -file
> /etc/pki/ovirt-engine/ca.pem -storepass password -keypass password -noprompt
> Certificate was added to keystore

Hmmm... so it is not reproduced at your environment either... strange. So in what state you are in? before upgrade or after upgrade? if before and you are running upgrade again, do you face the same issue?

--- Additional comment from Rich Jerrido on 2013-07-08 12:18:33 EDT ---

Current stage of this environment is post-upgrade, running the 3.2.0-11.37 bits.

When upgrading, I updated rhevm-setup first, then ran rhevm-upgrade which produced the errors in ovirt-engine-upgrade*.log as noted in the Description. 

Successive runs of rhevm-upgrade produced the same error. On a hunch, I removed everything in ca.pem except for what was with the ---BEGIN CERTIFICATE--- & ---END CERTIFICATE--- stanzas. After doing that, rhevm-upgrade proceeded normally & without further error. 

I don't know if I've hit some weird corner case (due to previously having the IBM JRE installed).

--- Additional comment from Alon Bar-Lev on 2013-07-08 12:30:33 EDT ---

(In reply to Rich Jerrido from comment #9)
> Current stage of this environment is post-upgrade, running the 3.2.0-11.37
> bits.
> 
> When upgrading, I updated rhevm-setup first, then ran rhevm-upgrade which
> produced the errors in ovirt-engine-upgrade*.log as noted in the
> Description. 
> 
> Successive runs of rhevm-upgrade produced the same error. On a hunch, I
> removed everything in ca.pem except for what was with the ---BEGIN
> CERTIFICATE--- & ---END CERTIFICATE--- stanzas. After doing that,
> rhevm-upgrade proceeded normally & without further error. 
> 
> I don't know if I've hit some weird corner case (due to previously having
> the IBM JRE installed).

understood. so you re-ran the setup and have a valid .truststore.

--- Additional comment from Alon Bar-Lev on 2013-07-08 12:32:05 EDT ---

Just in case I prepared the following, last bit of legacy in pki (I hope).

---

packaging: setup: enforce java home for pki

pki ca creation script use keytool utility directly, this may use
keytool utility of jdk other than openjdk. as some compatibility issues
were found, use the keytool from the JAVA_HOME we use for our
application.

pki migration to PKCS#12 format also use keytool, apply the same method.

Change-Id: I23ca5bc86cca6e9115a425ff885ab973a4e4135b
Signed-off-by: Alon Bar-Lev <alonbl@redhat.com>

--- Additional comment from Alon Bar-Lev on 2013-07-17 16:51:19 EDT ---
Comment 3 errata-xmlrpc 2013-08-13 09:08:31 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1149.html

Note You need to log in before you can comment on or make changes to this bug.