This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 980926 - Upgrade from 3.2.0-11.30 to 3.2.0-11.37 fails during 'Preparing CA' stage.
Upgrade from 3.2.0-11.30 to 3.2.0-11.37 fails during 'Preparing CA' stage.
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine-setup (Show other bugs)
3.2.0
All Linux
urgent Severity high
: ---
: 3.3.0
Assigned To: Alon Bar-Lev
Pavel Stehlik
integration
: ZStream
: 982475 (view as bug list)
Depends On: 961069
Blocks: 986985
  Show dependency treegraph
 
Reported: 2013-07-03 10:37 EDT by Rich Jerrido
Modified: 2015-09-22 09 EDT (History)
11 users (show)

See Also:
Fixed In Version: is6
Doc Type: Bug Fix
Doc Text:
Previously, using a Java runtime environment other than OpenJDK as the default Java runtime environment would sometimes result in an unusable engine keystore when upgrading to Red Hat Enterprise Virtualization Manager 3.2. This was caused by PKCS#12 being output in a format unusable by OpenJDK. With this update, the OpenJDK keytool utility is now explicitly used, resulting in a usable keystore.
Story Points: ---
Clone Of:
: 986985 (view as bug list)
Environment:
Last Closed: 2014-01-21 12:33:16 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
non-working ca.pem from RHEV-M (4.70 KB, application/x-x509-ca-cert)
2013-07-05 16:25 EDT, Rich Jerrido
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 419503 None None None Never
oVirt gerrit 16524 None None None Never

  None (edit)
Description Rich Jerrido 2013-07-03 10:37:36 EDT
Description of problem:

When upgrading from from RHEV-M 3.2.0-11.30 to 3.2.0-11.37, 'rhevm-upgrade' fails during 'Preparing CA' stage with the below error:

Preparing CA...                                      [ ERROR ]

 **Error: Upgrade failed, rolling back**
 **Reason: Error: Can't create trust store**


Version-Release number of selected component (if applicable):
rhevm-setup-3.2.0-11.37.el6ev.noarch

How reproducible:


Steps to Reproduce:
1. Update the rhevm-setup package to the version mentioned above. 
2. execute rhevm-upgrade 
3.

Actual results:

Installation fails with the error listed above. 


Expected results:

Upgrade to complete without error.


Additional info:

From ovirt-engine-upgrade*.log:

2013-07-03 09:27:17::DEBUG::rhevm-upgrade::542::root:: Converting truststore
2013-07-03 09:27:17::DEBUG::common_utils::453::root:: Executing command --> '/usr/bin/keytool -import -noprompt -keystore /etc/pki/ovirt-engine/.truststore.tm
p -storepass ******** -keypass ******** -alias cacert -trustcacerts -file /etc/pki/ovirt-engine/ca.pem' in working directory '/'
2013-07-03 09:27:18::DEBUG::common_utils::491::root:: output = keytool error: java.lang.Exception: Input not an X.509 certificate

2013-07-03 09:27:18::DEBUG::common_utils::492::root:: stderr = 
2013-07-03 09:27:18::DEBUG::common_utils::493::root:: retcode = 1
2013-07-03 09:27:18::ERROR::rhevm-upgrade::1337::root:: Traceback (most recent call last):
  File "/usr/bin/rhevm-upgrade", line 1331, in main
    runFunc([ca.prepare], MSG_INFO_PKI_PREPARE)
  File "/usr/bin/rhevm-upgrade", line 649, in runFunc
    func()
  File "/usr/bin/rhevm-upgrade", line 556, in prepare
    utils.execCmd(cmdList=cmd, maskList=mask, failOnError=True, msg=MSG_ERROR_FAILED_CREATE_TRUSTSTORE)
  File "/usr/share/ovirt-engine/scripts/common_utils.py", line 496, in execCmd
    raise Exception(msg)
Exception: Error: Can't create trust store


It apppears that the 'keytool' command is failing due to the fact that the x.509 certificate in /etc/pki/ovirt-engine/ca.pem is listed in both text form, and Base64 form. As a workaround, removing the text form and leaving what is between the ---BEGIN CERTIFICATE--- & ---END CERTIFICATE--- stanza (and then running rhevm-upgrade again), allows the upgrade to proceed.
Comment 1 Alon Bar-Lev 2013-07-05 14:55:42 EDT
Hello,

Can you please attach: /etc/pki/ovirt-engine/ca.pem

Thanks,
Alon
Comment 2 Rich Jerrido 2013-07-05 16:25:02 EDT
Created attachment 769382 [details]
non-working ca.pem from RHEV-M
Comment 3 Alon Bar-Lev 2013-07-05 16:51:16 EDT
(In reply to Rich Jerrido from comment #2)
> Created attachment 769382 [details]
> non-working ca.pem from RHEV-M

Thanks!

Certificate is valid.

Can you please see where keytool is pointing?

$ readlink -f /usr/bin/keytool

This is the final stroke before I enforce specific jre.
Comment 4 Rich Jerrido 2013-07-05 17:24:21 EDT
(In reply to Alon Bar-Lev from comment #3)
> (In reply to Rich Jerrido from comment #2)
> > Created attachment 769382 [details]
> > non-working ca.pem from RHEV-M
> 
> Thanks!
> 
> Certificate is valid.
> 
> Can you please see where keytool is pointing?
> 
> $ readlink -f /usr/bin/keytool
> 
> This is the final stroke before I enforce specific jre.

$ readlink -f /usr/bin/keytool
/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25.x86_64/jre/bin/keytool

For what it's worth, (and as additional background), this system was upgraded from 3.1 to 3.2. During the upgrade process to 3.2, I did run into bz961081, due to having the IBM JRE installed. I have since removed that JRE (prior to this update & subsequent bz). So I don't know if this is residual cruft from that upgrade.
Comment 5 Alon Bar-Lev 2013-07-05 17:28:32 EDT
(In reply to Rich Jerrido from comment #4)
> For what it's worth, (and as additional background), this system was
> upgraded from 3.1 to 3.2. During the upgrade process to 3.2, I did run into
> bz961081, due to having the IBM JRE installed.

Please vote for this bug solution to be included in 3.2.z :)
Comment 6 Alon Bar-Lev 2013-07-05 17:33:20 EDT
(In reply to Rich Jerrido from comment #4)
> $ readlink -f /usr/bin/keytool
> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25.x86_64/jre/bin/keytool

Strange. I cannot reproduce this with same version.

What is the output of:

$ rm -f /tmp/ks1
$ keytool -import -keystore /tmp/ks1 -alias cacert -trustcacerts -file /etc/pki/ovirt-engine/ca.pem -storepass password -keypass password -noprompt
Comment 7 Rich Jerrido 2013-07-08 11:12:13 EDT
Output of:

$ rm -f /tmp/ks1
$ keytool -import -keystore /tmp/ks1 -alias cacert -trustcacerts -file /etc/pki/ovirt-engine/ca.pem -storepass password -keypass password -noprompt
Certificate was added to keystore
Comment 8 Alon Bar-Lev 2013-07-08 11:14:42 EDT
(In reply to Rich Jerrido from comment #7)
> Output of:
> 
> $ rm -f /tmp/ks1
> $ keytool -import -keystore /tmp/ks1 -alias cacert -trustcacerts -file
> /etc/pki/ovirt-engine/ca.pem -storepass password -keypass password -noprompt
> Certificate was added to keystore

Hmmm... so it is not reproduced at your environment either... strange. So in what state you are in? before upgrade or after upgrade? if before and you are running upgrade again, do you face the same issue?
Comment 9 Rich Jerrido 2013-07-08 12:18:33 EDT
Current stage of this environment is post-upgrade, running the 3.2.0-11.37 bits.

When upgrading, I updated rhevm-setup first, then ran rhevm-upgrade which produced the errors in ovirt-engine-upgrade*.log as noted in the Description. 

Successive runs of rhevm-upgrade produced the same error. On a hunch, I removed everything in ca.pem except for what was with the ---BEGIN CERTIFICATE--- & ---END CERTIFICATE--- stanzas. After doing that, rhevm-upgrade proceeded normally & without further error. 

I don't know if I've hit some weird corner case (due to previously having the IBM JRE installed).
Comment 10 Alon Bar-Lev 2013-07-08 12:30:33 EDT
(In reply to Rich Jerrido from comment #9)
> Current stage of this environment is post-upgrade, running the 3.2.0-11.37
> bits.
> 
> When upgrading, I updated rhevm-setup first, then ran rhevm-upgrade which
> produced the errors in ovirt-engine-upgrade*.log as noted in the
> Description. 
> 
> Successive runs of rhevm-upgrade produced the same error. On a hunch, I
> removed everything in ca.pem except for what was with the ---BEGIN
> CERTIFICATE--- & ---END CERTIFICATE--- stanzas. After doing that,
> rhevm-upgrade proceeded normally & without further error. 
> 
> I don't know if I've hit some weird corner case (due to previously having
> the IBM JRE installed).

understood. so you re-ran the setup and have a valid .truststore.
Comment 11 Alon Bar-Lev 2013-07-08 12:32:05 EDT
Just in case I prepared the following, last bit of legacy in pki (I hope).

---

packaging: setup: enforce java home for pki

pki ca creation script use keytool utility directly, this may use
keytool utility of jdk other than openjdk. as some compatibility issues
were found, use the keytool from the JAVA_HOME we use for our
application.

pki migration to PKCS#12 format also use keytool, apply the same method.

Change-Id: I23ca5bc86cca6e9115a425ff885ab973a4e4135b
Signed-off-by: Alon Bar-Lev <alonbl@redhat.com>
Comment 12 Alon Bar-Lev 2013-07-17 16:51:19 EDT
*** Bug 982475 has been marked as a duplicate of this bug. ***
Comment 15 Charlie 2013-11-27 19:12:41 EST
This bug is currently attached to errata RHEA-2013:15231. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag.

Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information:

* Cause: What actions or circumstances cause this bug to present.
* Consequence: What happens when the bug presents.
* Fix: What was done to fix the bug.
* Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore')

Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug.

For further details on the Cause, Consequence, Fix, Result format please refer to:

https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes 

Thanks in advance.
Comment 16 errata-xmlrpc 2014-01-21 12:33:16 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2014-0038.html

Note You need to log in before you can comment on or make changes to this bug.