Bug 1733438

Summary: [downstream clone - 4.3.6] engine-setup fails to upgrade to 4.3 with Unicode characters in CA subject
Product: Red Hat Enterprise Virtualization Manager Reporter: RHV bug bot <rhv-bugzilla-bot>
Component: ovirt-engineAssignee: Yedidyah Bar David <didi>
Status: CLOSED ERRATA QA Contact: Petr Matyáš <pmatyas>
Severity: high Docs Contact:
Priority: high    
Version: 4.3.0CC: didi, emarcus, lleistne, mtessun, Rhev-m-bugs, stirabos
Target Milestone: ovirt-4.3.6Keywords: ZStream
Target Release: 4.3.6   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: ovirt-engine-4.3.6 Doc Type: Bug Fix
Doc Text:
During installation or upgrade to Red Had Virtualization 4.3, engine-setup failed if the PKI Organization Name in the CA certificate included non-ASCII characters. In this release, the upgrade engine-setup process completes successfully.
Story Points: ---
Clone Of: 1729511 Environment:
Last Closed: 2019-10-10 15:36:58 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Integration RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1729511    
Bug Blocks:    

Description RHV bug bot 2019-07-26 05:45:15 UTC
+++ This bug is a downstream clone. The original bug is: +++
+++   bug 1729511 +++
======================================================================

Description of problem:
While upgrading the engine from 4.1 to 4.3, engine-setup fails with:

2019-07-12 08:12:41,623+0100 DEBUG otopi.context context._executeMethod:145 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 132, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-common/base/core/postinstall.py", line 89, in _misc
    otopicons.CoreEnv.MODIFIED_FILES
  File "/usr/lib/python2.7/site-packages/otopi/filetransaction.py", line 151, in __init__
    self._content = self._content.encode("utf-8")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 415: ordinal not in range(128)
2019-07-12 08:12:41,627+0100 ERROR otopi.context context._executeMethod:154 Failed to execute stage 'Misc configuration': 'ascii' codec can't decode byte 0xc3 in position 415: ordinal not in range(128)


The CA certificate has unicode characters in the organization field of the subject:

# openssl x509 -in /etc/pki/ovirt-engine/ca.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 4096 (0x1000)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Inval\xC3\xADd, CN=rhvm.example.com.17156
        Validity
            Not Before: Apr 10 10:57:10 2018 GMT
            Not After : Apr  7 10:57:10 2028 GMT
        Subject: C=US, O=Inval\xC3\xADd, CN=rhvm.example.com.17156
[...]


Version-Release number of selected component (if applicable):
ovirt-engine-setup-base-4.3.4.3-0.1.el7.noarch

How reproducible:
Always

Steps to Reproduce:
1. Install 4.1 or older engine with a CA subject with unicode characters
2. Upgrade to 4.3

Actual results:
The upgrade process fail with the mentioned error. The upgrade to 4.2 has also been tested with similar errors.

Expected results:
The engine-setup process should identify this CA subject as invalid and do the renewal automatically.

Additional info:
A host has been upgraded to latest RHVH 4.3 and libvirtd failed to start with this error:

error : virNetTLSContextLoadCACertListFromFile:550 : Unable to import CA certificate list /etc/pki/vdsm/certs/cacert.pem

(Originally by Juan Orti Alcaine)

Comment 1 RHV bug bot 2019-07-26 05:45:18 UTC
(In reply to Juan Orti Alcaine from comment #0)
> Steps to Reproduce:
> 1. Install 4.1 or older engine with a CA subject with unicode characters
> 2. Upgrade to 4.3
> 
> Actual results:
> The upgrade process fail with the mentioned error. The upgrade to 4.2 has
> also been tested with similar errors.
> 
> Expected results:
> The engine-setup process should identify this CA subject as invalid and do
> the renewal automatically.

Not sure what you mean here. If it's invalid, perhaps we should prevent upgrade?
Please explain what you want.

I'll try to reproduce and check logs later on, then might have better insights.
Worst case, we might have to recreate all PKI with new certs and a new organization
name.

(Originally by didi)

Comment 2 RHV bug bot 2019-07-26 05:45:19 UTC
For other cases of invalid certificates, engine-setup asks to renew them:

https://access.redhat.com/solutions/1572983

I was speaking with Miguel Martín, and in theory, the X520OrganizationName field can be a utf8String, so maybe it's a problem with the SSL library that should accept those certificates.
https://tools.ietf.org/html/rfc5280#appendix-A

So, if we refuse to accept UTF-8 strings, ideally all the PKI should be recreated and the hosts re-enrolled. If not, the necessary changes should be done to accept the certificates.

(Originally by Juan Orti Alcaine)

Comment 3 RHV bug bot 2019-07-26 05:45:21 UTC
Now reproduced also on clean setup - exactly the same failure happens, during postinstall generation, when supplying a unicode organization.

(Originally by didi)

Comment 4 RHV bug bot 2019-07-26 05:45:23 UTC
And the failure is in python code, unrelated to openssl or anything like that. I wonder how it was created so originally...

(Originally by didi)

Comment 5 RHV bug bot 2019-07-26 05:45:24 UTC
OK, found the reason. It was broken only on otopi-1.8 (oVirt/RHV 4.3), due to this patch:

https://gerrit.ovirt.org/92435

I think the patch is correct in principle, and will probably need to patch the code generating postinstall to pass binary data.

(Originally by didi)

Comment 6 RHV bug bot 2019-07-26 05:45:26 UTC
Should I open a different bug for libvirtd not starting with that CA cert?

(Originally by Juan Orti Alcaine)

Comment 7 RHV bug bot 2019-07-26 05:45:28 UTC
(In reply to Juan Orti Alcaine from comment #6)
> Should I open a different bug for libvirtd not starting with that CA cert?

Probably, but I am still not sure what the exact problem was. The specific error in comment 0 is unrelated to ssl or any 3rd-party library - it's all due to oVirt code. Once we fix that, we can try to reproduce the libvirt error. Or, you can try to reproduce with an 4.2 engine (I didn't try). If you do reproduce, and indeed upgrading a host (where the engine is still 4.2) fails as noted, please open a bug. Thanks.

(Originally by didi)

Comment 10 Petr Matyáš 2019-08-09 14:19:14 UTC
Verified on ovirt-engine-4.3.6-0.1.el7.noarch

The engine was updated from 4.1 where I set organization to 'Inval\xC3\xADd\xC3\xA1\xC5\xA1\x08'

Comment 11 RHV bug bot 2019-08-29 13:05:21 UTC
INFO: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Open patch attached]

For more info please contact: rhv-devops

Comment 12 RHV bug bot 2019-09-25 08:46:41 UTC
INFO: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Tag 'ovirt-engine-4.3.5.6' doesn't contain patch 'https://gerrit.ovirt.org/102204']
gitweb: https://gerrit.ovirt.org/gitweb?p=ovirt-engine.git;a=shortlog;h=refs/tags/ovirt-engine-4.3.5.6

For more info please contact: rhv-devops

Comment 13 Yedidyah Bar David 2019-09-25 09:17:42 UTC
Hello bugzilla bot, and hoping a human will eventually see this:

Any idea why we got comment 12? change 102204, aka git commit 60fde0a, is included in all 4.3.6 tags, and the bug is targetted to 4.3.6. Why does the bot check 4.3.5.6?

Comment 18 errata-xmlrpc 2019-10-10 15:36:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:3010