Bug 986985

+++ This bug was initially created as a clone of Bug #980926 +++

Description of problem:

When upgrading from from RHEV-M 3.2.0-11.30 to 3.2.0-11.37, 'rhevm-upgrade' fails during 'Preparing CA' stage with the below error:

Preparing CA...                                      [ ERROR ]

 **Error: Upgrade failed, rolling back**
 **Reason: Error: Can't create trust store**


Version-Release number of selected component (if applicable):
rhevm-setup-3.2.0-11.37.el6ev.noarch

How reproducible:


Steps to Reproduce:
1. Update the rhevm-setup package to the version mentioned above. 
2. execute rhevm-upgrade 
3.

Actual results:

Installation fails with the error listed above. 


Expected results:

Upgrade to complete without error.


Additional info:

From ovirt-engine-upgrade*.log:

2013-07-03 09:27:17::DEBUG::rhevm-upgrade::542::root:: Converting truststore
2013-07-03 09:27:17::DEBUG::common_utils::453::root:: Executing command --> '/usr/bin/keytool -import -noprompt -keystore /etc/pki/ovirt-engine/.truststore.tm
p -storepass ******** -keypass ******** -alias cacert -trustcacerts -file /etc/pki/ovirt-engine/ca.pem' in working directory '/'
2013-07-03 09:27:18::DEBUG::common_utils::491::root:: output = keytool error: java.lang.Exception: Input not an X.509 certificate

2013-07-03 09:27:18::DEBUG::common_utils::492::root:: stderr = 
2013-07-03 09:27:18::DEBUG::common_utils::493::root:: retcode = 1
2013-07-03 09:27:18::ERROR::rhevm-upgrade::1337::root:: Traceback (most recent call last):
  File "/usr/bin/rhevm-upgrade", line 1331, in main
    runFunc([ca.prepare], MSG_INFO_PKI_PREPARE)
  File "/usr/bin/rhevm-upgrade", line 649, in runFunc
    func()
  File "/usr/bin/rhevm-upgrade", line 556, in prepare
    utils.execCmd(cmdList=cmd, maskList=mask, failOnError=True, msg=MSG_ERROR_FAILED_CREATE_TRUSTSTORE)
  File "/usr/share/ovirt-engine/scripts/common_utils.py", line 496, in execCmd
    raise Exception(msg)
Exception: Error: Can't create trust store


It apppears that the 'keytool' command is failing due to the fact that the x.509 certificate in /etc/pki/ovirt-engine/ca.pem is listed in both text form, and Base64 form. As a workaround, removing the text form and leaving what is between the ---BEGIN CERTIFICATE--- & ---END CERTIFICATE--- stanza (and then running rhevm-upgrade again), allows the upgrade to proceed.

--- Additional comment from Alon Bar-Lev on 2013-07-05 14:55:42 EDT ---

Hello,

Can you please attach: /etc/pki/ovirt-engine/ca.pem

Thanks,
Alon

--- Additional comment from Rich Jerrido on 2013-07-05 16:25:02 EDT ---



--- Additional comment from Alon Bar-Lev on 2013-07-05 16:51:16 EDT ---

(In reply to Rich Jerrido from comment #2)
> Created attachment 769382 [details]
> non-working ca.pem from RHEV-M

Thanks!

Certificate is valid.

Can you please see where keytool is pointing?

$ readlink -f /usr/bin/keytool

This is the final stroke before I enforce specific jre.

--- Additional comment from Rich Jerrido on 2013-07-05 17:24:21 EDT ---

(In reply to Alon Bar-Lev from comment #3)
> (In reply to Rich Jerrido from comment #2)
> > Created attachment 769382 [details]
> > non-working ca.pem from RHEV-M
> 
> Thanks!
> 
> Certificate is valid.
> 
> Can you please see where keytool is pointing?
> 
> $ readlink -f /usr/bin/keytool
> 
> This is the final stroke before I enforce specific jre.

$ readlink -f /usr/bin/keytool
/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25.x86_64/jre/bin/keytool

For what it's worth, (and as additional background), this system was upgraded from 3.1 to 3.2. During the upgrade process to 3.2, I did run into bz961081, due to having the IBM JRE installed. I have since removed that JRE (prior to this update & subsequent bz). So I don't know if this is residual cruft from that upgrade.

--- Additional comment from Alon Bar-Lev on 2013-07-05 17:28:32 EDT ---

(In reply to Rich Jerrido from comment #4)
> For what it's worth, (and as additional background), this system was
> upgraded from 3.1 to 3.2. During the upgrade process to 3.2, I did run into
> bz961081, due to having the IBM JRE installed.

Please vote for this bug solution to be included in 3.2.z :)

--- Additional comment from Alon Bar-Lev on 2013-07-05 17:33:20 EDT ---

(In reply to Rich Jerrido from comment #4)
> $ readlink -f /usr/bin/keytool
> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25.x86_64/jre/bin/keytool

Strange. I cannot reproduce this with same version.

What is the output of:

$ rm -f /tmp/ks1
$ keytool -import -keystore /tmp/ks1 -alias cacert -trustcacerts -file /etc/pki/ovirt-engine/ca.pem -storepass password -keypass password -noprompt

--- Additional comment from Rich Jerrido on 2013-07-08 11:12:13 EDT ---

Output of:

$ rm -f /tmp/ks1
$ keytool -import -keystore /tmp/ks1 -alias cacert -trustcacerts -file /etc/pki/ovirt-engine/ca.pem -storepass password -keypass password -noprompt
Certificate was added to keystore

--- Additional comment from Alon Bar-Lev on 2013-07-08 11:14:42 EDT ---

(In reply to Rich Jerrido from comment #7)
> Output of:
> 
> $ rm -f /tmp/ks1
> $ keytool -import -keystore /tmp/ks1 -alias cacert -trustcacerts -file
> /etc/pki/ovirt-engine/ca.pem -storepass password -keypass password -noprompt
> Certificate was added to keystore

Hmmm... so it is not reproduced at your environment either... strange. So in what state you are in? before upgrade or after upgrade? if before and you are running upgrade again, do you face the same issue?

--- Additional comment from Rich Jerrido on 2013-07-08 12:18:33 EDT ---

Current stage of this environment is post-upgrade, running the 3.2.0-11.37 bits.

When upgrading, I updated rhevm-setup first, then ran rhevm-upgrade which produced the errors in ovirt-engine-upgrade*.log as noted in the Description. 

Successive runs of rhevm-upgrade produced the same error. On a hunch, I removed everything in ca.pem except for what was with the ---BEGIN CERTIFICATE--- & ---END CERTIFICATE--- stanzas. After doing that, rhevm-upgrade proceeded normally & without further error. 

I don't know if I've hit some weird corner case (due to previously having the IBM JRE installed).

--- Additional comment from Alon Bar-Lev on 2013-07-08 12:30:33 EDT ---

(In reply to Rich Jerrido from comment #9)
> Current stage of this environment is post-upgrade, running the 3.2.0-11.37
> bits.
> 
> When upgrading, I updated rhevm-setup first, then ran rhevm-upgrade which
> produced the errors in ovirt-engine-upgrade*.log as noted in the
> Description. 
> 
> Successive runs of rhevm-upgrade produced the same error. On a hunch, I
> removed everything in ca.pem except for what was with the ---BEGIN
> CERTIFICATE--- & ---END CERTIFICATE--- stanzas. After doing that,
> rhevm-upgrade proceeded normally & without further error. 
> 
> I don't know if I've hit some weird corner case (due to previously having
> the IBM JRE installed).

understood. so you re-ran the setup and have a valid .truststore.

--- Additional comment from Alon Bar-Lev on 2013-07-08 12:32:05 EDT ---

Just in case I prepared the following, last bit of legacy in pki (I hope).

---

packaging: setup: enforce java home for pki

pki ca creation script use keytool utility directly, this may use
keytool utility of jdk other than openjdk. as some compatibility issues
were found, use the keytool from the JAVA_HOME we use for our
application.

pki migration to PKCS#12 format also use keytool, apply the same method.

Change-Id: I23ca5bc86cca6e9115a425ff885ab973a4e4135b
Signed-off-by: Alon Bar-Lev <alonbl>

--- Additional comment from Alon Bar-Lev on 2013-07-17 16:51:19 EDT ---