Bug 1607800

Summary: [UPGRADES][14] UndercloudPostDeployment failed: bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate' certificate verify failed')
Product: Red Hat OpenStack Reporter: Yurii Prokulevych <yprokule>
Component: openstack-tripleo-heat-templatesAssignee: Jose Luis Franco <jfrancoa>
Status: CLOSED ERRATA QA Contact: Yurii Prokulevych <yprokule>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 14.0 (Rocky)CC: augol, ccamacho, jfrancoa, josorior, jstransk, mbracho, mburns, pveiga, slinaber, yprokule
Target Milestone: betaKeywords: Triaged
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-9.0.0-0.20180919080946.0rc1.0rc1.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1712551 (view as bug list) Environment:
Last Closed: 2019-01-11 11:50:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1712551    

Description Yurii Prokulevych 2018-07-24 09:55:23 UTC
Description of problem:
-----------------------
Upgrade of RHOS-13 undercloud to containerized RHOS-14 undercloud failed:

openstack undercloud upgrade -y --use-heat
...
                "++ export CLOUDPROMPT_ENABLED=1",
                "++ CLOUDPROMPT_ENABLED=1",
                "+ '[' '!' -f /home/stack/.ssh/authorized_keys ']'",
                "+ '[' '!' -f /home/stack/.ssh/id_rsa ']'",
                "++ cat /home/stack/.ssh/id_rsa.pub",
                "+ grep 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAoilfFFTGFASMZXm8pOuj3W7TmDltehxTcPRXEdOnkh7RMUKU0gNEaNvdR4VPjVk6BIZ9XHaCxvLxGryYbED2s8iCXl0rYGe9qPM1E1WE0ZUL00BL3gmFOVozvr5Denf+Stb9cZXxHsaOBWOi5R+GRu
IwjmLmNlGZcvkfV4R4bA8y2lNifxi0LBoY65vvWSymL5ZxptxzBPEZpoDk+/3tavx55dZ8L8ibbMIILl8RhS7eqj0VTtP4Z591w61x/Q7nOufCcSXuyY76KvGz1xaLG9h0CpLKbc1k8/9bGEmBABOIbvuajiKKN/a/vv0XewwyehhjtcKssoJSbOoXa/wskQ==' /home/stack/.ss
h/authorized_keys",
                "+ chown -R stack:stack /home/stack/.ssh",
                "++ hiera nova_api_enabled",
                "+ '[' true = true ']'",
                "++ openstack project show admin",
                "++ awk '$2==\"id\" {print $4}'",
                "+ openstack quota set --cores -1 --instances -1 --ram -1 dcc36d7c2ccf4ab8b469492b24c84165",
                "SSL exception connecting to https://192.168.24.2:13774/v2.1/os-quota-sets/dcc36d7c2ccf4ab8b469492b24c84165: (\"bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate 
verify failed')],)\",)",
                "",
                "[2018-07-24 05:39:54,279] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-script/88a2dd32-e2d0-498b-806d-9bec43ba9c9b. [1]",
                "",
                "",
                "[2018-07-24 05:39:54,283] (heat-config) [INFO] Completed /usr/libexec/heat-config/hooks/script",
                "[2018-07-24 05:39:54,283] (heat-config) [DEBUG] Running heat-config-notify /var/lib/heat-config/deployed/88a2dd32-e2d0-498b-806d-9bec43ba9c9b.json < /var/lib/heat-config/deployed/88a2dd32-e2d0-4
98b-806d-9bec43ba9c9b.notify.json",
                "[2018-07-24 05:39:54,894] (heat-config) [INFO] ",
                "[2018-07-24 05:39:54,894] (heat-config) [DEBUG] "
            ]
        },
        {
            "status_code": "1"
        }
    ]
}



Version-Release number of selected component (if applicable):
-------------------------------------------------------------
openstack-tripleo-common-containers-9.1.1-0.20180710151736.8e8dabd.el7ost.noarch
openstack-tripleo-heat-templates-9.0.0-0.20180710202746.d2994ca.el7ost.noarch
python-tripleoclient-heat-installer-10.2.1-0.20180709114814.e5ce9a8.el7ost.noarch
python-tripleoclient-10.2.1-0.20180709114814.e5ce9a8.el7ost.noarch


How reproducible:
-----------------
100%

Steps to Reproduce:
-------------------
1. Deploy RHOS-13 undercloud with SSL
2. Setup RHOS-14 repos
3. Upgrade openstack-tripleo-common-containers and generate container-images for RHOS-14 uc
4. Upgrade python-tripleoclient
5. Start undercloud upgrade:
    openstack undercloud upgrade -y --use-heat

Comment 2 Juan Antonio Osorio 2018-07-24 10:22:05 UTC
what's the output of:

openssl s_client -connect 192.168.24.2:13774

Comment 3 Yurii Prokulevych 2018-07-24 11:08:55 UTC
[stack@undercloud-0 ~ ] $ openssl s_client -connect 192.168.24.2:13774
CONNECTED(00000003)
depth=1 CN = Local Signing Authority, CN = 2b8bd999-02514113-9a56ff86-874336f9
verify return:1
depth=0 CN = 192.168.24.2
verify return:1
---
Certificate chain
 0 s:/CN=192.168.24.2
   i:/CN=Local Signing Authority/CN=2b8bd999-02514113-9a56ff86-874336f9
 1 s:/CN=Local Signing Authority/CN=2b8bd999-02514113-9a56ff86-874336f9
   i:/CN=Local Signing Authority/CN=2b8bd999-02514113-9a56ff86-874336f9
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDfDCCAmSgAwIBAgIQK4vZmQJRQROaVv+Gh0M2+jANBgkqhkiG9w0BAQsFADBQ
MSAwHgYDVQQDDBdMb2NhbCBTaWduaW5nIEF1dGhvcml0eTEsMCoGA1UEAwwjMmI4
YmQ5OTktMDI1MTQxMTMtOWE1NmZmODYtODc0MzM2ZjkwHhcNMTgwNzI0MDkzMDI2
WhcNMTkwNzI0MDkzMDI0WjAXMRUwEwYDVQQDEwwxOTIuMTY4LjI0LjIwggEiMA0G
CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDKQcEn+rVPt6m/mGWQECQovXKIzi+n
E+BZHklfks6AvbKUpIz5xRA3FYlXYRIPdSIt5qgTAmJKgoJZQnuvjPH8g5nYx9s6
r6R31Q+c+ZRFa2b7LPvL3qFQMrFN5Swp7NA6Vg7DgMQn+X+ynPome9MyEy2b17I1
LJpxOY3GlPSjzg1NZ21jdZy5AhZEjWc5WlMbbWwNMiyf2XZ2X0qKO5HZx+G75YGl
wEyd4Xsyggi+gTbduC/XNw8otiHS+9Fr41PcIyKv786TmFRtWIozh0lkL2dIPb7F
jWVxB3cXz0ivokWKhFrx7AkAjpsVsHVYWTI7/X3/ynpHzdcZCWDBVDvlAgMBAAGj
gYowgYcwEgYDVR0RAQEABAgwBocEwKgYAjAgBgNVHSUBAQAEFjAUBggrBgEFBQcD
AgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAgBgNVHQ4BAQAEFgQU6An4N2gDFQrQ
EZeoHVQRjqplZo0wHwYDVR0jBBgwFoAUuiqHqPSdagrCHTSJ8EMx3oDp8+0wDQYJ
KoZIhvcNAQELBQADggEBACKCzlvuXQPBOTFcYO72d9i3LqYwRFKZSQL8bqy+LYMk
glK5j/5tVwvGH2NZ5L44lGmkKn5V0HYxqU5WAT3nzy1QbdA1HMxjsekJFnx0oCJw
8Xm7DQZAQaNgWtgt4Wr86bUyDOWt3ZQTvG/OSECSDGzPLBxn/4TFgGHibbo3oDUJ
GeAOyLSmhgUmfrFluK0lUfdahKbsQFXgw8X5lFz8zYXY5bJwB3PsURFe4m2WN9AM
RbxPTIf93yoD81HLhcrviocffkpyhLTSowAxGcc3A+22ygGuse6XQfdUAZopUim0
/V3NG2RHM6Z5qLqtEfTS/Ha+Ok/EnIjx9nwg9HOtE1E=
-----END CERTIFICATE-----
subject=/CN=192.168.24.2
issuer=/CN=Local Signing Authority/CN=2b8bd999-02514113-9a56ff86-874336f9
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2471 bytes and written 415 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 85DDCE66CA8F47D35EC5BE1C808EC977C5519509C574467CF9A4F7E3DA235B07
    Session-ID-ctx: 
    Master-Key: 248F1CFBA212491855D2AB09492711530672A0F1A6DE89837EFF8C462D4B610E80505EED33B99EA785F9C90C3B112941
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 6b 3f 19 88 aa 9c c6 90-35 a8 87 fe a6 78 ff 4a   k?......5....x.J
    0010 - e4 21 65 18 be c4 c4 06-63 99 e9 76 fe 41 e6 ca   .!e.....c..v.A..
    0020 - 9a 0a fd ce 73 ef 59 eb-a8 5e 38 d8 c0 17 72 2e   ....s.Y..^8...r.
    0030 - 6c b7 d4 45 a6 0e 9a bf-1e 74 67 ff 82 4a 1c c9   l..E.....tg..J..
    0040 - 58 c6 9a f6 19 1e 53 41-8b 87 f6 69 88 0e e8 8a   X.....SA...i....
    0050 - 59 30 6f 06 8b 99 8f 00-c0 fd df a1 8d 30 0a da   Y0o..........0..
    0060 - 1a 5e eb 20 f0 50 ff 70-c6 f7 e1 06 2b 38 bd 1e   .^. .P.p....+8..
    0070 - 79 cb de da f2 ec c4 7b-80 37 ab 43 55 17 d4 ad   y......{.7.CU...
    0080 - d0 cb 6f 93 a5 40 66 5e-20 5b 80 a5 41 b9 ce 69   ..o..@f^ [..A..i
    0090 - 9c 28 ca c8 3c 6e b3 b7-0d a1 14 db 88 b2 2d 51   .(..<n........-Q

    Start Time: 1532430389
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
/HTTP/1.0 408 Request Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html><body><h1>408 Request Time-out</h1>
Your browser didn't send a complete request in time.
</body></html>
closed

Comment 14 errata-xmlrpc 2019-01-11 11:50:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045