Bug 1260752 - RHEV-M upgrade to 3.5.4 fails with error "Command '/usr/bin/openssl' failed to execute" for custom apache.p12
Summary: RHEV-M upgrade to 3.5.4 fails with error "Command '/usr/bin/openssl' failed ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.5.4
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ovirt-3.6.2
: 3.6.2
Assignee: Yedidyah Bar David
QA Contact: Gonza
URL:
Whiteboard:
Depends On:
Blocks: 1286607
TreeView+ depends on / blocked
 
Reported: 2015-09-07 15:32 UTC by nijin ashok
Modified: 2019-08-15 05:24 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1286607 (view as bug list)
Environment:
Last Closed: 2016-03-09 21:13:09 UTC
oVirt Team: Integration


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:0376 normal SHIPPED_LIVE Red Hat Enterprise Virtualization Manager 3.6.0 2016-03-10 01:20:52 UTC
oVirt gerrit 49172 master MERGED packaging: setup: pki: Do not fail if pkcs12 unreadable Never
oVirt gerrit 49408 ovirt-engine-3.6 MERGED packaging: setup: pki: Do not fail if pkcs12 unreadable Never
Red Hat Knowledge Base (Solution) 1605173 None None None Never

Description nijin ashok 2015-09-07 15:32:30 UTC
Description of problem:

Upgrade to RHEV-M 3.5.4  is failing with below error in the engine log if we are having a /etc/pki/ovirt-engine/keys/apache.p12 certificate with a password other than "mypass". The logs shows

=========
2015-09-07 10:09:48 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca plugin.execute:937 execute-output: ('/usr/bin/openssl', 'pkcs12', '-in', '/etc/pki/ovirt-engine/keys/apache.p12', '-passin', 'pass:**FILTERED**', '-nokeys') stderr:
Mac verify error: invalid password?

2015-09-07 10:09:48 DEBUG otopi.context context._executeMethod:152 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/otopi/context.py", line 142, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/pki/ca.py", line 436, in _customization_upgrade
    for entry in self._PKI_ENTRIES
  File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/pki/ca.py", line 240, in _ok_to_renew_cert
    x509 = self._extractPKCS12Certificate(pkcs12)
  File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/pki/ca.py", line 108, in _extractPKCS12Certificate
    self._extractPKCS12CertificateString(pkcs12)
  File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/pki/ca.py", line 100, in _extractPKCS12CertificateString
    '-nokeys',
  File "/usr/lib/python2.6/site-packages/otopi/plugin.py", line 942, in execute
    command=args[0],
RuntimeError: Command '/usr/bin/openssl' failed to execute
2015-09-07 10:09:48 ERROR otopi.context context._executeMethod:161 Failed to execute stage 'Environment customization': Command '/usr/bin/openssl' failed to execute
============

From the log it seems that the /etc/pki/ovirt-engine/keys/apache.p12 is extracted with the default "mypass" password. As the customer is having custom /etc/pki/ovirt-engine/keys/apache.p12 without any password, engine-setup fails with the provided error.  

Version-Release number of selected component (if applicable):
rhevm-setup-3.5.4.2-1.3.el6ev.noarch

How reproducible:
100%

Steps to Reproduce:
1. Create a custom apache.p12 file with any random password other than "mypass"

openssl pkcs12 -export -out /etc/pki/ovirt-engine/keys/apache.p12 -inkey /etc/pki/ovirt-engine/keys/apache.key.nopass -in /etc/pki/ovirt-engine/certs/apache.cer

2. Run engine-setup and this will fails with the provided error.

Actual results:

engine-setup fails with error for custom apache.p12 

Expected results:

engine-setup should work with apache.p12 with any password as apache.p12 is allowed to change in the RHEV environment.

Additional info:

One possible workaround to complete engine-setup will be creating apache.p12 with the password "mypass" from the custom apache certificates.

Comment 1 Yaniv Lavi 2015-09-08 10:50:06 UTC
Why is this bug severity is high?

Comment 2 nijin ashok 2015-09-08 11:14:46 UTC
(In reply to Yaniv Dary from comment #1)
> Why is this bug severity is high?

I have reduced the severity and the customer have confirmed that the workaround resolves the issue.

Comment 3 Yedidyah Bar David 2015-09-08 12:01:46 UTC
I wasn't aware of the fact that our documentation [1] tells to replace the files.

A perhaps better approach (and workaround for current bug) would have been to instruct instead to put the custom key/cert outside of /etc/pki/ovirt-engine (in whatever place the user feels is appropriate) and edit /etc/httpd/conf.d/ssl.conf accordingly.

Upstream says [2] that either are allowed, but later says [3] to not override (probably meant "overwrite"?) these files.

The setup log attached to the case says that openssl outputs to stderr:
Mac verify error: invalid password

We can probably check for this and if so skip this cert, perhaps only for apache. Didn't check the output myself on other versions of OSes/openssl.

Alon, what do you think?

[1] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.5/html/Administration_Guide/appe-Red_Hat_Enterprise_Virtualization_and_SSL.html

[2] http://www.ovirt.org/Features/PKI#User--SSL--.3Eapache--AJP--.3Eovirt-engine

[3] http://www.ovirt.org/Features/PKI#ovirt-engine

Comment 4 Alon Bar-Lev 2015-09-08 12:43:07 UTC
We already testing if CA is our CA when reading the apache.p12 before renew - indeed if password is incorrect we should ignore this as well.

In perfect world, we would not have touched apache ssl configuration, but as the legacy setup had done so, we left it as is, and instructed people to replace the key/cert/p12 if they desire to modify ssl settings to avoid further touch of apache configuration.

Comment 7 Yedidyah Bar David 2015-11-30 08:14:54 UTC
Note to QE:

Current patch makes engine-setup output the following in such a case:

 [WARNING] Failed to read or parse '/etc/pki/ovirt-engine/keys/apache.p12'
           Perhaps it was changed since last Setup.
           Error was:
           Mac verify error: invalid password?

It will do that twice per file - once during customization, right before asking whether to renew the CA, and again while actually renewing.

I personally verified that by just changing the passphrase of apache.p12 and websocket-proxy.p12 .

Please try also other combinations, e.g. also engine.p12 or jboss.p12 (might break stuff, doc (link above) mentions only apache.p12), and also by using a key/cert signed by a 3rd party CA, both using passphrase 'mypass' and something different.

Comment 10 Gonza 2016-01-11 16:14:54 UTC
Verified with:
rhevm-3.6.2-0.1.el6.noarch

# engine-setup
...
[WARNING] Failed to read or parse '/etc/pki/ovirt-engine/keys/apache.p12'
          Perhaps it was changed since last Setup.
          Error was:
          Mac verify error: invalid password?
...
[WARNING] Failed to read or parse '/etc/pki/ovirt-engine/keys/jboss.p12'
          Perhaps it was changed since last Setup.
          Error was:
          139818997311304:error:0D07207B:asn1 encoding routines:ASN1_get_object:header too long:asn1_lib.c:150:
...

Engine upgraded successfully.

Comment 12 errata-xmlrpc 2016-03-09 21:13:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0376.html


Note You need to log in before you can comment on or make changes to this bug.