Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 980926

Summary: Upgrade from 3.2.0-11.30 to 3.2.0-11.37 fails during 'Preparing CA' stage.
Product: Red Hat Enterprise Virtualization Manager Reporter: Rich Jerrido <rjerrido>
Component: ovirt-engine-setupAssignee: Alon Bar-Lev <alonbl>
Status: CLOSED ERRATA QA Contact: Pavel Stehlik <pstehlik>
Severity: high Docs Contact:
Priority: urgent    
Version: 3.2.0CC: acathrow, adahms, alonbl, bazulay, cpelland, iheim, jkt, mgoldboi, mwest, rjerrido, sbonazzo
Target Milestone: ---Keywords: ZStream
Target Release: 3.3.0   
Hardware: All   
OS: Linux   
Whiteboard: integration
Fixed In Version: is6 Doc Type: Bug Fix
Doc Text:
Previously, using a Java runtime environment other than OpenJDK as the default Java runtime environment would sometimes result in an unusable engine keystore when upgrading to Red Hat Enterprise Virtualization Manager 3.2. This was caused by PKCS#12 being output in a format unusable by OpenJDK. With this update, the OpenJDK keytool utility is now explicitly used, resulting in a usable keystore.
Story Points: ---
Clone Of:
: 986985 (view as bug list) Environment:
Last Closed: 2014-01-21 17:33:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 961069    
Bug Blocks: 986985    
Attachments:
Description Flags
non-working ca.pem from RHEV-M none

Description Rich Jerrido 2013-07-03 14:37:36 UTC
Description of problem:

When upgrading from from RHEV-M 3.2.0-11.30 to 3.2.0-11.37, 'rhevm-upgrade' fails during 'Preparing CA' stage with the below error:

Preparing CA...                                      [ ERROR ]

 **Error: Upgrade failed, rolling back**
 **Reason: Error: Can't create trust store**


Version-Release number of selected component (if applicable):
rhevm-setup-3.2.0-11.37.el6ev.noarch

How reproducible:


Steps to Reproduce:
1. Update the rhevm-setup package to the version mentioned above. 
2. execute rhevm-upgrade 
3.

Actual results:

Installation fails with the error listed above. 


Expected results:

Upgrade to complete without error.


Additional info:

From ovirt-engine-upgrade*.log:

2013-07-03 09:27:17::DEBUG::rhevm-upgrade::542::root:: Converting truststore
2013-07-03 09:27:17::DEBUG::common_utils::453::root:: Executing command --> '/usr/bin/keytool -import -noprompt -keystore /etc/pki/ovirt-engine/.truststore.tm
p -storepass ******** -keypass ******** -alias cacert -trustcacerts -file /etc/pki/ovirt-engine/ca.pem' in working directory '/'
2013-07-03 09:27:18::DEBUG::common_utils::491::root:: output = keytool error: java.lang.Exception: Input not an X.509 certificate

2013-07-03 09:27:18::DEBUG::common_utils::492::root:: stderr = 
2013-07-03 09:27:18::DEBUG::common_utils::493::root:: retcode = 1
2013-07-03 09:27:18::ERROR::rhevm-upgrade::1337::root:: Traceback (most recent call last):
  File "/usr/bin/rhevm-upgrade", line 1331, in main
    runFunc([ca.prepare], MSG_INFO_PKI_PREPARE)
  File "/usr/bin/rhevm-upgrade", line 649, in runFunc
    func()
  File "/usr/bin/rhevm-upgrade", line 556, in prepare
    utils.execCmd(cmdList=cmd, maskList=mask, failOnError=True, msg=MSG_ERROR_FAILED_CREATE_TRUSTSTORE)
  File "/usr/share/ovirt-engine/scripts/common_utils.py", line 496, in execCmd
    raise Exception(msg)
Exception: Error: Can't create trust store


It apppears that the 'keytool' command is failing due to the fact that the x.509 certificate in /etc/pki/ovirt-engine/ca.pem is listed in both text form, and Base64 form. As a workaround, removing the text form and leaving what is between the ---BEGIN CERTIFICATE--- & ---END CERTIFICATE--- stanza (and then running rhevm-upgrade again), allows the upgrade to proceed.

Comment 1 Alon Bar-Lev 2013-07-05 18:55:42 UTC
Hello,

Can you please attach: /etc/pki/ovirt-engine/ca.pem

Thanks,
Alon

Comment 2 Rich Jerrido 2013-07-05 20:25:02 UTC
Created attachment 769382 [details]
non-working ca.pem from RHEV-M

Comment 3 Alon Bar-Lev 2013-07-05 20:51:16 UTC
(In reply to Rich Jerrido from comment #2)
> Created attachment 769382 [details]
> non-working ca.pem from RHEV-M

Thanks!

Certificate is valid.

Can you please see where keytool is pointing?

$ readlink -f /usr/bin/keytool

This is the final stroke before I enforce specific jre.

Comment 4 Rich Jerrido 2013-07-05 21:24:21 UTC
(In reply to Alon Bar-Lev from comment #3)
> (In reply to Rich Jerrido from comment #2)
> > Created attachment 769382 [details]
> > non-working ca.pem from RHEV-M
> 
> Thanks!
> 
> Certificate is valid.
> 
> Can you please see where keytool is pointing?
> 
> $ readlink -f /usr/bin/keytool
> 
> This is the final stroke before I enforce specific jre.

$ readlink -f /usr/bin/keytool
/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25.x86_64/jre/bin/keytool

For what it's worth, (and as additional background), this system was upgraded from 3.1 to 3.2. During the upgrade process to 3.2, I did run into bz961081, due to having the IBM JRE installed. I have since removed that JRE (prior to this update & subsequent bz). So I don't know if this is residual cruft from that upgrade.

Comment 5 Alon Bar-Lev 2013-07-05 21:28:32 UTC
(In reply to Rich Jerrido from comment #4)
> For what it's worth, (and as additional background), this system was
> upgraded from 3.1 to 3.2. During the upgrade process to 3.2, I did run into
> bz961081, due to having the IBM JRE installed.

Please vote for this bug solution to be included in 3.2.z :)

Comment 6 Alon Bar-Lev 2013-07-05 21:33:20 UTC
(In reply to Rich Jerrido from comment #4)
> $ readlink -f /usr/bin/keytool
> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25.x86_64/jre/bin/keytool

Strange. I cannot reproduce this with same version.

What is the output of:

$ rm -f /tmp/ks1
$ keytool -import -keystore /tmp/ks1 -alias cacert -trustcacerts -file /etc/pki/ovirt-engine/ca.pem -storepass password -keypass password -noprompt

Comment 7 Rich Jerrido 2013-07-08 15:12:13 UTC
Output of:

$ rm -f /tmp/ks1
$ keytool -import -keystore /tmp/ks1 -alias cacert -trustcacerts -file /etc/pki/ovirt-engine/ca.pem -storepass password -keypass password -noprompt
Certificate was added to keystore

Comment 8 Alon Bar-Lev 2013-07-08 15:14:42 UTC
(In reply to Rich Jerrido from comment #7)
> Output of:
> 
> $ rm -f /tmp/ks1
> $ keytool -import -keystore /tmp/ks1 -alias cacert -trustcacerts -file
> /etc/pki/ovirt-engine/ca.pem -storepass password -keypass password -noprompt
> Certificate was added to keystore

Hmmm... so it is not reproduced at your environment either... strange. So in what state you are in? before upgrade or after upgrade? if before and you are running upgrade again, do you face the same issue?

Comment 9 Rich Jerrido 2013-07-08 16:18:33 UTC
Current stage of this environment is post-upgrade, running the 3.2.0-11.37 bits.

When upgrading, I updated rhevm-setup first, then ran rhevm-upgrade which produced the errors in ovirt-engine-upgrade*.log as noted in the Description. 

Successive runs of rhevm-upgrade produced the same error. On a hunch, I removed everything in ca.pem except for what was with the ---BEGIN CERTIFICATE--- & ---END CERTIFICATE--- stanzas. After doing that, rhevm-upgrade proceeded normally & without further error. 

I don't know if I've hit some weird corner case (due to previously having the IBM JRE installed).

Comment 10 Alon Bar-Lev 2013-07-08 16:30:33 UTC
(In reply to Rich Jerrido from comment #9)
> Current stage of this environment is post-upgrade, running the 3.2.0-11.37
> bits.
> 
> When upgrading, I updated rhevm-setup first, then ran rhevm-upgrade which
> produced the errors in ovirt-engine-upgrade*.log as noted in the
> Description. 
> 
> Successive runs of rhevm-upgrade produced the same error. On a hunch, I
> removed everything in ca.pem except for what was with the ---BEGIN
> CERTIFICATE--- & ---END CERTIFICATE--- stanzas. After doing that,
> rhevm-upgrade proceeded normally & without further error. 
> 
> I don't know if I've hit some weird corner case (due to previously having
> the IBM JRE installed).

understood. so you re-ran the setup and have a valid .truststore.

Comment 11 Alon Bar-Lev 2013-07-08 16:32:05 UTC
Just in case I prepared the following, last bit of legacy in pki (I hope).

---

packaging: setup: enforce java home for pki

pki ca creation script use keytool utility directly, this may use
keytool utility of jdk other than openjdk. as some compatibility issues
were found, use the keytool from the JAVA_HOME we use for our
application.

pki migration to PKCS#12 format also use keytool, apply the same method.

Change-Id: I23ca5bc86cca6e9115a425ff885ab973a4e4135b
Signed-off-by: Alon Bar-Lev <alonbl>

Comment 12 Alon Bar-Lev 2013-07-17 20:51:19 UTC
*** Bug 982475 has been marked as a duplicate of this bug. ***

Comment 15 Charlie 2013-11-28 00:12:41 UTC
This bug is currently attached to errata RHEA-2013:15231. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag.

Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information:

* Cause: What actions or circumstances cause this bug to present.
* Consequence: What happens when the bug presents.
* Fix: What was done to fix the bug.
* Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore')

Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug.

For further details on the Cause, Consequence, Fix, Result format please refer to:

https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes 

Thanks in advance.

Comment 16 errata-xmlrpc 2014-01-21 17:33:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2014-0038.html