Bug 1701426 - [OSP-14] TLS for Manila Internal services
Summary: [OSP-14] TLS for Manila Internal services
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: puppet-manila
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: z3
: 14.0 (Rocky)
Assignee: Goutham Pacha Ravi
QA Contact: Jason Grosso
Laura Marsh
URL:
Whiteboard:
Depends On:
Blocks: 1701427
TreeView+ depends on / blocked
 
Reported: 2019-04-18 23:25 UTC by Goutham Pacha Ravi
Modified: 2019-07-22 10:42 UTC (History)
8 users (show)

Fixed In Version: puppet-manila-13.3.2-0.20190420081603.f969ee6.el7ost
Doc Type: Enhancement
Doc Text:
Prior to this release, the communication between hapoxy and the Shared File Systems service (Manila) API was not secured when deployed with TLS everywhere. Support has been added for the Manila API to configured with SSL certificates, allowing TLS on the internal API network. This feature is now automatically configured when TLS everywhere is enabled.
Clone Of:
: 1701427 (view as bug list)
Environment:
Last Closed: 2019-07-22 10:42:53 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 652810 0 None None None 2019-04-22 17:49:26 UTC

Description Goutham Pacha Ravi 2019-04-18 23:25:52 UTC
This bug was initially created as a copy of Bug #1484601

I am copying this bug because a backport of this feature has been requested to OSP 14 and OSP 13

Description of problem:
TLS Support for Manila internal services: 
 Following flows:
 - client to HAProxy
  - HAProxy to server instance

DFG is requested to test basic flows with TLS enabled and verify encryption by doing a tcpdump.

Comment 2 Goutham Pacha Ravi 2019-04-27 05:40:35 UTC
This change has merged upstream. It is ready to be imported downstream.

Comment 4 Lon Hohberger 2019-07-10 10:41:21 UTC
According to our records, this should be resolved by puppet-manila-13.3.2-0.20190420081603.f969ee6.el7ost.  This build is available now.

Comment 5 Jason Grosso 2019-07-19 14:20:58 UTC
Tested on 14z3 
 
(overcloud) [stack@undercloud-0 ~]$ rpm -qa | grep puppet-manila
puppet-manila-13.3.2-0.20190420081603.f969ee6.el7ost.noarch



stack@undercloud-0 ~]$ source stackrc 
(undercloud) [stack@undercloud-0 ~]$ openstack endpoint list
+----------------------------------+-----------+------------------+-------------------------+---------+-----------+--------------------------------------------------+
| ID                               | Region    | Service Name     | Service Type            | Enabled | Interface | URL                                              |
+----------------------------------+-----------+------------------+-------------------------+---------+-----------+--------------------------------------------------+
| 0e35af7f080745739d6ae93bffbe1491 | regionOne | swift            | object-store            | True    | public    | https://192.168.24.2:13808/v1/AUTH_%(tenant_id)s |
| 148405c8bf7d47dab2d9c022cb73be60 | regionOne | keystone         | identity                | True    | internal  | http://192.168.24.3:5000                         |
| 1a6f1d4c6e5b40c1b0bfdf7f02326ef3 | regionOne | ironic           | baremetal               | True    | admin     | http://192.168.24.3:6385                         |
| 2748e5ef2f6e4b4e8d3575c762e2034c | regionOne | glance           | image                   | True    | internal  | http://192.168.24.3:9292                         |
| 27b3649ac3ba4b75a993d85da742914f | regionOne | glance           | image                   | True    | public    | https://192.168.24.2:13292                       |
| 355f1990c64c48ad999252d79d2f02e5 | regionOne | nova             | compute                 | True    | internal  | http://192.168.24.3:8774/v2.1                    |
| 3852951ff4c047f780f346cab0baf9f4 | regionOne | zaqar-websocket  | messaging-websocket     | True    | public    | wss://192.168.24.2:9000                          |
| 40e90a16280641dd8a341e9cdc580004 | regionOne | heat             | orchestration           | True    | internal  | http://192.168.24.3:8004/v1/%(tenant_id)s        |
| 45907c6e6687457aa3972b9d8ca2a8a2 | regionOne | heat-cfn         | cloudformation          | True    | public    | https://192.168.24.2:13005/v1                    |
| 489584a4e89b49efadcaa73fff9cb0dd | regionOne | placement        | placement               | True    | internal  | http://192.168.24.3:8778/placement               |
| 5d0e6f5a0a8744a980ac6881f49fe46b | regionOne | zaqar            | messaging               | True    | internal  | http://192.168.24.3:8888                         |
| 674f734115904bcc943e08041646b46e | regionOne | zaqar            | messaging               | True    | admin     | http://192.168.24.3:8888                         |
| 6910adad30864939850fb97af6a0e927 | regionOne | swift            | object-store            | True    | admin     | http://192.168.24.3:8080                         |
| 6c72e11630a94467946fc2e23650829a | regionOne | ironic           | baremetal               | True    | internal  | http://192.168.24.3:6385                         |
| 6e6788b03fab4576ae958e7461847428 | regionOne | zaqar            | messaging               | True    | public    | https://192.168.24.2:13888                       |
| 736dd40f4d9447ae84c2f54ffa9a2fb4 | regionOne | zaqar-websocket  | messaging-websocket     | True    | admin     | ws://192.168.24.3:9000                           |
| 7a611d38caa7432689351e22c0a4c13f | regionOne | ironic-inspector | baremetal-introspection | True    | admin     | http://192.168.24.3:5050                         |
| 7ead7264a19947759ed0ce02ae0a2dd8 | regionOne | placement        | placement               | True    | public    | https://192.168.24.2:13778/placement             |
| 8331b99bef964fba81113da3e940574f | regionOne | neutron          | network                 | True    | internal  | http://192.168.24.3:9696                         |
| 8d204801ea214e9880fe0cd4427d7bb1 | regionOne | nova             | compute                 | True    | public    | https://192.168.24.2:13774/v2.1                  |
| 9e88d97494fe4f2db2a3b1fadb57a762 | regionOne | zaqar-websocket  | messaging-websocket     | True    | internal  | ws://192.168.24.3:9000                           |
| 9eb748b187ef41d58e6880078039e85f | regionOne | swift            | object-store            | True    | internal  | http://192.168.24.3:8080/v1/AUTH_%(tenant_id)s   |
| a58a56d7b3bc4c64a24bf9ed7d3476da | regionOne | heat-cfn         | cloudformation          | True    | admin     | http://192.168.24.3:8000/v1                      |
| ab1155c714a84598ad8273730030cbf5 | regionOne | glance           | image                   | True    | admin     | http://192.168.24.3:9292                         |
| ba16d7ddb8a1463fb31265aa575c5795 | regionOne | neutron          | network                 | True    | public    | https://192.168.24.2:13696                       |
| bf15b55815974eb2ab4762de0720a48f | regionOne | keystone         | identity                | True    | public    | https://192.168.24.2:13000                       |
| c1f40325e3224696a9f38b7f10507c3d | regionOne | ironic-inspector | baremetal-introspection | True    | internal  | http://192.168.24.3:5050                         |
| c5aec0de9b3f410fa2df2833e117d494 | regionOne | neutron          | network                 | True    | admin     | http://192.168.24.3:9696                         |
| caa73b40873d40d1baeacc98ca4765c3 | regionOne | heat             | orchestration           | True    | public    | https://192.168.24.2:13004/v1/%(tenant_id)s      |
| ccaeec2f17f24322922af09e625f67d2 | regionOne | nova             | compute                 | True    | admin     | http://192.168.24.3:8774/v2.1                    |
| d1a5e7b2d2d94691b55a833e6d7891e3 | regionOne | ironic-inspector | baremetal-introspection | True    | public    | https://192.168.24.2:13050                       |
| d530c96010c54fd89be50d574df0ba5b | regionOne | mistral          | workflowv2              | True    | public    | https://192.168.24.2:13989/v2                    |
| d89b59479d80454ba96fc2edac4caa11 | regionOne | keystone         | identity                | True    | admin     | http://192.168.24.3:35357                        |
| e801764b1e6c43a2b43bd81d4c55fc20 | regionOne | mistral          | workflowv2              | True    | admin     | http://192.168.24.3:8989/v2                      |
| e8a583bfac6f4bb49a7e47ab6ff9bccb | regionOne | placement        | placement               | True    | admin     | http://192.168.24.3:8778/placement               |
| eb2aafe065474299b1b70baa16cb544a | regionOne | mistral          | workflowv2              | True    | internal  | http://192.168.24.3:8989/v2                      |
| eff89459e50c498d8b6067d619d55277 | regionOne | ironic           | baremetal               | True    | public    | https://192.168.24.2:13385                       |
| f075bd92b28b405288dbf947816bfd85 | regionOne | heat-cfn         | cloudformation          | True    | internal  | http://192.168.24.3:8000/v1                      |
| fde6a66f1b234f958ab0892387cf5536 | regionOne | heat             | orchestration           | True    | admin     | http://192.168.24.3:8004/v1/%(tenant_id)s        |
+----------------------------------+-----------+------------------+-------------------------+---------+-----------+--------------------------------------------------+
(undercloud) [stack@undercloud-0 ~]$ source overcloudrc
(overcloud) [stack@undercloud-0 ~]$ openstack endpoint list
+----------------------------------+-----------+--------------+----------------+---------+-----------+-------------------------------------------------------------------+
| ID                               | Region    | Service Name | Service Type   | Enabled | Interface | URL                                                               |
+----------------------------------+-----------+--------------+----------------+---------+-----------+-------------------------------------------------------------------+
| 006194f3346544f8a51e8208ca32552b | regionOne | neutron      | network        | True    | internal  | https://overcloud.internalapi.redhat.local:9696                   |
| 023b2faf451446d69ba998405b136139 | regionOne | glance       | image          | True    | public    | https://overcloud.redhat.local:13292                              |
| 0452fc3d60f44dd4aa6ffa7a5ac01b83 | regionOne | heat         | orchestration  | True    | admin     | https://overcloud.internalapi.redhat.local:8004/v1/%(tenant_id)s  |
| 10b96db7a792486096f1f6270a68c0cf | regionOne | manilav2     | sharev2        | True    | admin     | https://overcloud.internalapi.redhat.local:8786/v2/%(tenant_id)s  |
| 1349cff27c8949dc807759e103f54f23 | regionOne | keystone     | identity       | True    | public    | https://overcloud.redhat.local:13000                              |
| 14a6580469ed4e48ad009502dc26bf1c | regionOne | manila       | share          | True    | public    | https://overcloud.redhat.local:13786/v1/%(tenant_id)s             |
| 1ff6fec4ffef4cab86821786baee1981 | regionOne | neutron      | network        | True    | public    | https://overcloud.redhat.local:13696                              |
| 2188d4de71a9429f93734e18b01799c7 | regionOne | placement    | placement      | True    | internal  | https://overcloud.internalapi.redhat.local:8778/placement         |
| 25fe251e8fdd4244946d85cb26c71112 | regionOne | panko        | event          | True    | admin     | https://overcloud.internalapi.redhat.local:8977                   |
| 28841ff0e4e24ddda8a0ed52a98188c2 | regionOne | gnocchi      | metric         | True    | internal  | https://overcloud.internalapi.redhat.local:8041                   |
| 33557497506040eb90341e5194305f64 | regionOne | manilav2     | sharev2        | True    | internal  | https://overcloud.internalapi.redhat.local:8786/v2/%(tenant_id)s  |
| 3391bd2d8e434df1ad66131197d21d7f | regionOne | cinderv3     | volumev3       | True    | internal  | https://overcloud.internalapi.redhat.local:8776/v3/%(tenant_id)s  |
| 3cf8c7a1127146bea55f54489ede96f7 | regionOne | glance       | image          | True    | admin     | https://overcloud.internalapi.redhat.local:9292                   |
| 483a6b69e357434cb69e0d86b2a7d109 | regionOne | swift        | object-store   | True    | public    | https://overcloud.redhat.local:13808/v1/AUTH_%(tenant_id)s        |
| 4f93933feda54918b89e753251b63841 | regionOne | gnocchi      | metric         | True    | public    | https://overcloud.redhat.local:13041                              |
| 4fffab10743a418f87ca6d70f1e35867 | regionOne | manila       | share          | True    | internal  | https://overcloud.internalapi.redhat.local:8786/v1/%(tenant_id)s  |
| 51d83185115c4e5a880f8c88532c814a | regionOne | heat         | orchestration  | True    | internal  | https://overcloud.internalapi.redhat.local:8004/v1/%(tenant_id)s  |
| 5720698131fc4243b1a96dc637f96baa | regionOne | nova         | compute        | True    | public    | https://overcloud.redhat.local:13774/v2.1                         |
| 592ac835276d44c18e161b16a7a6223a | regionOne | panko        | event          | True    | internal  | https://overcloud.internalapi.redhat.local:8977                   |
| 5ab17713f54f40fa84a0ea6611d0e98b | regionOne | placement    | placement      | True    | admin     | https://overcloud.internalapi.redhat.local:8778/placement         |
| 5cbd8ef7d7bb4be1b30e1bc96fa52ca9 | regionOne | keystone     | identity       | True    | admin     | https://overcloud.ctlplane.redhat.local:35357                     |
| 5ff6b069fbee4564a7e3cc30c1bc83bb | regionOne | aodh         | alarming       | True    | admin     | https://overcloud.internalapi.redhat.local:8042                   |
| 61f33ebbacdd460d8ffd042762b3aa18 | regionOne | manilav2     | sharev2        | True    | public    | https://overcloud.redhat.local:13786/v2/%(tenant_id)s             |
| 67bc9f65e0c047bdb290a93912ddfe0d | regionOne | placement    | placement      | True    | public    | https://overcloud.redhat.local:13778/placement                    |
| 6dddc5e462644e97a7f94b18bc9f8eae | regionOne | aodh         | alarming       | True    | internal  | https://overcloud.internalapi.redhat.local:8042                   |
| 71efff7f4f994ed7b0fbd779dc76d926 | regionOne | cinderv2     | volumev2       | True    | internal  | https://overcloud.internalapi.redhat.local:8776/v2/%(tenant_id)s  |
| 756ca5db99e242f784366749fea7d9a1 | regionOne | heat-cfn     | cloudformation | True    | admin     | https://overcloud.internalapi.redhat.local:8000/v1                |
| 7a3cd5450964420ebd063d7cdb3ecb77 | regionOne | nova         | compute        | True    | internal  | https://overcloud.internalapi.redhat.local:8774/v2.1              |
| 7f03f257320f4d2bb6e3efcf759070df | regionOne | cinderv2     | volumev2       | True    | admin     | https://overcloud.internalapi.redhat.local:8776/v2/%(tenant_id)s  |
| 7fef1a330bfd4941b630245f3960dfa1 | regionOne | cinderv2     | volumev2       | True    | public    | https://overcloud.redhat.local:13776/v2/%(tenant_id)s             |
| 8853bdfa3bc14b6c8aa8245bbaddca52 | regionOne | keystone     | identity       | True    | internal  | https://overcloud.internalapi.redhat.local:5000                   |
| 945e397c21934598b6040e496bda2382 | regionOne | cinderv3     | volumev3       | True    | admin     | https://overcloud.internalapi.redhat.local:8776/v3/%(tenant_id)s  |
| 95abe087de8f4bb793f55aca14e5b917 | regionOne | heat-cfn     | cloudformation | True    | public    | https://overcloud.redhat.local:13005/v1                           |
| 9abf47baae9649818706e0fa7a345370 | regionOne | nova         | compute        | True    | admin     | https://overcloud.internalapi.redhat.local:8774/v2.1              |
| aa351d0df9ef40e5b7b0195d6004072a | regionOne | heat         | orchestration  | True    | public    | https://overcloud.redhat.local:13004/v1/%(tenant_id)s             |
| aa77c078ed7d4620892f2908aa2bc245 | regionOne | glance       | image          | True    | internal  | https://overcloud.internalapi.redhat.local:9292                   |
| ae73cd1882f240b1a10af68b5c45a195 | regionOne | swift        | object-store   | True    | admin     | https://overcloud.storage.redhat.local:8080                       |
| b024d88cfc084adfb25791f23d1956e5 | regionOne | gnocchi      | metric         | True    | admin     | https://overcloud.internalapi.redhat.local:8041                   |
| c0b55ad0b66a4acbb0ad92708bff6253 | regionOne | neutron      | network        | True    | admin     | https://overcloud.internalapi.redhat.local:9696                   |
| c408a530676b4e5e9c653ae5856cbe0b | regionOne | heat-cfn     | cloudformation | True    | internal  | https://overcloud.internalapi.redhat.local:8000/v1                |
| c74f48efc721476687672a9bcf288cff | regionOne | aodh         | alarming       | True    | public    | https://overcloud.redhat.local:13042                              |
| e5d8fbb192a04e7d9e63d071f9c34602 | regionOne | manila       | share          | True    | admin     | https://overcloud.internalapi.redhat.local:8786/v1/%(tenant_id)s  |
| e7099379dea64b95805bd9bfcc4f74cd | regionOne | panko        | event          | True    | public    | https://overcloud.redhat.local:13977                              |
| f99b8f520bf64678b9aa1773cbaed0c4 | regionOne | swift        | object-store   | True    | internal  | https://overcloud.storage.redhat.local:8080/v1/AUTH_%(tenant_id)s |
| fda5372943e84585ac2085bf0cb6a51c | regionOne | cinderv3     | volumev3       | True    | public    | https://overcloud.redhat.local:13776/v3/%(tenant_id)s             |
+----------------------------------+-----------+--------------+----------------+---------+-----------+-------------------------------------------------------------------+

from controller-0
(undercloud) [stack@undercloud-0 ~]$ ssh heat-admin.24.12
Warning: Permanently added '192.168.24.12' (ECDSA) to the list of known hosts.
Last login: Thu Jul 18 20:18:12 2019 from 192.168.24.254
[heat-admin@controller-0 ~]$ openssl s_client -crlf -connect overcloud.internalapi.redhat.local:8786
CONNECTED(00000003)
depth=1 O = REDHAT.LOCAL, CN = Certificate Authority
verify return:1
depth=0 O = REDHAT.LOCAL, CN = controller-0.internalapi.redhat.local
verify return:1
---
Certificate chain
 0 s:/O=REDHAT.LOCAL/CN=controller-0.internalapi.redhat.local
   i:/O=REDHAT.LOCAL/CN=Certificate Authority
 1 s:/O=REDHAT.LOCAL/CN=Certificate Authority
   i:/O=REDHAT.LOCAL/CN=Certificate Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFJzCCBA+gAwIBAgIBOTANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQKDAxSRURI
QVQuTE9DQUwxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xOTA3
MTgxOTM5MzlaFw0yMTA3MTgxOTM5MzlaMEcxFTATBgNVBAoMDFJFREhBVC5MT0NB
TDEuMCwGA1UEAwwlY29udHJvbGxlci0wLmludGVybmFsYXBpLnJlZGhhdC5sb2Nh
bDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMskdGKcnVow2EIaRFnT
KSg7tJ8pLfb96JOPLnYDAhz9qAQVeOZvqjv90nv1qOBQeQaUBeGTw2KSShZi77WD
l5SUxKrlPHs4vgqZbmlB3X3xiEs2iUgAnMbpF05rfHzSf0UZXMdmChA1PQ6c7TEp
RENNN4kiFRq/vKfcPMa/EH+pRCfCfIG6tHAH7FJcNHrTggAwD0ERzUHCpdfY4SIR
iyTieTh2I3KUiMxnuQh729A6QP+ZPKw94+KmD9yAgwNiiUz94Q6MaKhstN+C6Vr7
qVvkfr2JVLg6lqtxT60rzsTwTmtwWqlR6/DQMKFlaWJGLIwQfmzitBLB92ntrkaf
46kCAwEAAaOCAiwwggIoMB8GA1UdIwQYMBaAFCnx1H6IuNLDrVy+BOvtonw9WH1u
MD4GCCsGAQUFBwEBBDIwMDAuBggrBgEFBQcwAYYiaHR0cDovL2lwYS1jYS5yZWRo
YXQubG9jYWwvY2Evb2NzcDAOBgNVHQ8BAf8EBAMCBPAwHQYDVR0lBBYwFAYIKwYB
BQUHAwEGCCsGAQUFBwMCMHcGA1UdHwRwMG4wbKA0oDKGMGh0dHA6Ly9pcGEtY2Eu
cmVkaGF0LmxvY2FsL2lwYS9jcmwvTWFzdGVyQ1JMLmJpbqI0pDIwMDEOMAwGA1UE
CgwFaXBhY2ExHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAdBgNVHQ4E
FgQUFajayefXyDfltCnMxcP+YXy07Ukwgf0GA1UdEQSB9TCB8oIib3ZlcmNsb3Vk
LmludGVybmFsYXBpLnJlZGhhdC5sb2NhbIIlY29udHJvbGxlci0wLmludGVybmFs
YXBpLnJlZGhhdC5sb2NhbKBKBgorBgEEAYI3FAIDoDwMOmhhcHJveHkvY29udHJv
bGxlci0wLmludGVybmFsYXBpLnJlZGhhdC5sb2NhbEBSRURIQVQuTE9DQUygWQYG
KwYBBQICoE8wTaAOGwxSRURIQVQuTE9DQUyhOzA5oAMCAQGhMjAwGwdoYXByb3h5
GyVjb250cm9sbGVyLTAuaW50ZXJuYWxhcGkucmVkaGF0LmxvY2FsMA0GCSqGSIb3
DQEBCwUAA4IBAQCtUCya4PX+ne/R+KXgz9L2SxAfYpomw5c2aRmOuIUUgjO9IixS
OziA3Ew5B5GPiFGstliePYEqdDczQlnWVdYlaPKymGVKG3vgdbS1e6APdRXFGxvm
7C4xMxjnL9rPdKPvoaZDIS7qups0fTMtfvmUq+zpFa091iEekQ/uyvZGIhDO91bs
7VjOMyDIUWiBcgfddlZA45lMEEnSXjnwF0UqPBu7k91SZrsAyHH1pumdOUajVZ4D
+p9J6e4Z0agQ7JfKwBC6drCxQ+J04ejw+NDn32ZbU2D4pX58ZfkX+uFUtwSJMm27
daYhTUSyd0YibrMcMFl8M8XIyRLKv/oc6Dls
-----END CERTIFICATE-----
subject=/O=REDHAT.LOCAL/CN=controller-0.internalapi.redhat.local
issuer=/O=REDHAT.LOCAL/CN=Certificate Authority
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2899 bytes and written 415 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 396F15433184204D00B768C449CE239EEFDDE16121F140764A57FB976649F876
    Session-ID-ctx: 
    Master-Key: 5816324960631581F109CEF8905E8676A3B150E7D3E3823B28D4977D540DCC644E1C43F73291C732987FC2D696C11F63
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - c4 94 84 d1 41 7a 36 78-82 cd 6a 80 92 e8 6e 68   ....Az6x..j...nh
    0010 - 51 02 20 45 e5 7c 60 b3-85 f4 64 3b 69 f3 d0 4c   Q. E.|`...d;i..L
    0020 - 37 01 c5 94 82 e7 02 5d-10 35 f6 fd 26 79 ab 52   7......].5..&y.R
    0030 - 1e 5a 72 9c ab b3 55 ca-3c f9 6b bd 56 fc 6d b8   .Zr...U.<.k.V.m.
    0040 - 10 04 2e ab f4 92 4b 1b-61 40 5f bb d0 12 77 02   ......K.a@_...w.
    0050 - a1 6e c2 dc c7 e4 7e d1-20 d1 83 10 e1 32 cd 66   .n....~. ....2.f
    0060 - af 2e b0 25 ee 53 21 8a-69 7e 90 5c 5a 45 7a 64   ...%.S!.i~.\ZEzd
    0070 - 78 6c c2 b5 e6 86 31 c1-e4 b8 50 1d ad e5 11 13   xl....1...P.....
    0080 - 62 aa 58 9e b4 6f 6e 27-a5 e1 8e ba 2e bd d0 53   b.X..on'.......S
    0090 - f7 e8 ae c6 da 25 f8 58-ea 3c 3c c5 e2 de a3      .....%.X.<<....
    00a0 - <SPACES/NULS>

    Start Time: 1563545415
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
HTTP/1.0 408 Request Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html><body><h1>408 Request Time-out</h1>
Your browser didn't send a complete request in time.
</body></html>
closed


from controller-1 


[heat-admin@controller-1 ~]$  openssl s_client -crlf -connect overcloud.internalapi.redhat.local:8786
CONNECTED(00000003)
depth=1 O = REDHAT.LOCAL, CN = Certificate Authority
verify return:1
depth=0 O = REDHAT.LOCAL, CN = controller-0.internalapi.redhat.local
verify return:1
---
Certificate chain
 0 s:/O=REDHAT.LOCAL/CN=controller-0.internalapi.redhat.local
   i:/O=REDHAT.LOCAL/CN=Certificate Authority
 1 s:/O=REDHAT.LOCAL/CN=Certificate Authority
   i:/O=REDHAT.LOCAL/CN=Certificate Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFJzCCBA+gAwIBAgIBOTANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQKDAxSRURI
QVQuTE9DQUwxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xOTA3
MTgxOTM5MzlaFw0yMTA3MTgxOTM5MzlaMEcxFTATBgNVBAoMDFJFREhBVC5MT0NB
TDEuMCwGA1UEAwwlY29udHJvbGxlci0wLmludGVybmFsYXBpLnJlZGhhdC5sb2Nh
bDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMskdGKcnVow2EIaRFnT
KSg7tJ8pLfb96JOPLnYDAhz9qAQVeOZvqjv90nv1qOBQeQaUBeGTw2KSShZi77WD
l5SUxKrlPHs4vgqZbmlB3X3xiEs2iUgAnMbpF05rfHzSf0UZXMdmChA1PQ6c7TEp
RENNN4kiFRq/vKfcPMa/EH+pRCfCfIG6tHAH7FJcNHrTggAwD0ERzUHCpdfY4SIR
iyTieTh2I3KUiMxnuQh729A6QP+ZPKw94+KmD9yAgwNiiUz94Q6MaKhstN+C6Vr7
qVvkfr2JVLg6lqtxT60rzsTwTmtwWqlR6/DQMKFlaWJGLIwQfmzitBLB92ntrkaf
46kCAwEAAaOCAiwwggIoMB8GA1UdIwQYMBaAFCnx1H6IuNLDrVy+BOvtonw9WH1u
MD4GCCsGAQUFBwEBBDIwMDAuBggrBgEFBQcwAYYiaHR0cDovL2lwYS1jYS5yZWRo
YXQubG9jYWwvY2Evb2NzcDAOBgNVHQ8BAf8EBAMCBPAwHQYDVR0lBBYwFAYIKwYB
BQUHAwEGCCsGAQUFBwMCMHcGA1UdHwRwMG4wbKA0oDKGMGh0dHA6Ly9pcGEtY2Eu
cmVkaGF0LmxvY2FsL2lwYS9jcmwvTWFzdGVyQ1JMLmJpbqI0pDIwMDEOMAwGA1UE
CgwFaXBhY2ExHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAdBgNVHQ4E
FgQUFajayefXyDfltCnMxcP+YXy07Ukwgf0GA1UdEQSB9TCB8oIib3ZlcmNsb3Vk
LmludGVybmFsYXBpLnJlZGhhdC5sb2NhbIIlY29udHJvbGxlci0wLmludGVybmFs
YXBpLnJlZGhhdC5sb2NhbKBKBgorBgEEAYI3FAIDoDwMOmhhcHJveHkvY29udHJv
bGxlci0wLmludGVybmFsYXBpLnJlZGhhdC5sb2NhbEBSRURIQVQuTE9DQUygWQYG
KwYBBQICoE8wTaAOGwxSRURIQVQuTE9DQUyhOzA5oAMCAQGhMjAwGwdoYXByb3h5
GyVjb250cm9sbGVyLTAuaW50ZXJuYWxhcGkucmVkaGF0LmxvY2FsMA0GCSqGSIb3
DQEBCwUAA4IBAQCtUCya4PX+ne/R+KXgz9L2SxAfYpomw5c2aRmOuIUUgjO9IixS
OziA3Ew5B5GPiFGstliePYEqdDczQlnWVdYlaPKymGVKG3vgdbS1e6APdRXFGxvm
7C4xMxjnL9rPdKPvoaZDIS7qups0fTMtfvmUq+zpFa091iEekQ/uyvZGIhDO91bs
7VjOMyDIUWiBcgfddlZA45lMEEnSXjnwF0UqPBu7k91SZrsAyHH1pumdOUajVZ4D
+p9J6e4Z0agQ7JfKwBC6drCxQ+J04ejw+NDn32ZbU2D4pX58ZfkX+uFUtwSJMm27
daYhTUSyd0YibrMcMFl8M8XIyRLKv/oc6Dls
-----END CERTIFICATE-----
subject=/O=REDHAT.LOCAL/CN=controller-0.internalapi.redhat.local
issuer=/O=REDHAT.LOCAL/CN=Certificate Authority
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2899 bytes and written 415 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: C7796F1FB5E12714A712BDFC42B7239323D974A1147A0B03667FA49C1FA84486
    Session-ID-ctx: 
    Master-Key: E753D7C2C099216BD0ECAFF4519F998DC83456B58056E7400F467E637F5102237B6E406210ABED4C2375D4B19240B773
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - c4 94 84 d1 41 7a 36 78-82 cd 6a 80 92 e8 6e 68   ....Az6x..j...nh
    0010 - 1f 9b 16 9c 2b 43 8e c0-8d cc ba 25 2d e0 39 0f   ....+C.....%-.9.
    0020 - 88 79 cf c7 c0 80 e9 59-06 af 1f c5 2f 1f e6 d9   .y.....Y..../...
    0030 - 0b 3e 37 16 89 e5 03 92-c9 53 e5 c7 4b c6 c8 88   .>7......S..K...
    0040 - 78 c3 8b d0 0d f0 af 21-5f ff c1 9a f1 d2 b4 d6   x......!_.......
    0050 - 80 05 11 d5 70 b3 aa 85-04 04 0b 64 57 3e 08 28   ....p......dW>.(
    0060 - 9b 0f a1 93 c1 22 31 12-ef b9 ef 59 9d 43 1b e3   ....."1....Y.C..
    0070 - c9 b9 00 29 c0 f5 65 58-d4 3e 6a 00 9e 15 bb a4   ...)..eX.>j.....
    0080 - fd 10 9f c4 47 d4 22 44-4e da 9b d0 57 70 bd e7   ....G."DN...Wp..
    0090 - 70 bb 50 ee 4b 8f ab b2-9b f6 40 12 d7 73 4c 7d   p.P.K.....@..sL}

    Start Time: 1563545709
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
HTTP/1.0 408 Request Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html><body><h1>408 Request Time-out</h1>
Your browser didn't send a complete request in time.
</body></html>
closed


from controller-2[heat-admin@controller-2 ~]$ openssl s_client -crlf -connect overcloud.internalapi.redhat.local:8786
CONNECTED(00000003)
depth=1 O = REDHAT.LOCAL, CN = Certificate Authority
verify return:1
depth=0 O = REDHAT.LOCAL, CN = controller-0.internalapi.redhat.local
verify return:1
---
Certificate chain
 0 s:/O=REDHAT.LOCAL/CN=controller-0.internalapi.redhat.local
   i:/O=REDHAT.LOCAL/CN=Certificate Authority
 1 s:/O=REDHAT.LOCAL/CN=Certificate Authority
   i:/O=REDHAT.LOCAL/CN=Certificate Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFJzCCBA+gAwIBAgIBOTANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQKDAxSRURI
QVQuTE9DQUwxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xOTA3
MTgxOTM5MzlaFw0yMTA3MTgxOTM5MzlaMEcxFTATBgNVBAoMDFJFREhBVC5MT0NB
TDEuMCwGA1UEAwwlY29udHJvbGxlci0wLmludGVybmFsYXBpLnJlZGhhdC5sb2Nh
bDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMskdGKcnVow2EIaRFnT
KSg7tJ8pLfb96JOPLnYDAhz9qAQVeOZvqjv90nv1qOBQeQaUBeGTw2KSShZi77WD
l5SUxKrlPHs4vgqZbmlB3X3xiEs2iUgAnMbpF05rfHzSf0UZXMdmChA1PQ6c7TEp
RENNN4kiFRq/vKfcPMa/EH+pRCfCfIG6tHAH7FJcNHrTggAwD0ERzUHCpdfY4SIR
iyTieTh2I3KUiMxnuQh729A6QP+ZPKw94+KmD9yAgwNiiUz94Q6MaKhstN+C6Vr7
qVvkfr2JVLg6lqtxT60rzsTwTmtwWqlR6/DQMKFlaWJGLIwQfmzitBLB92ntrkaf
46kCAwEAAaOCAiwwggIoMB8GA1UdIwQYMBaAFCnx1H6IuNLDrVy+BOvtonw9WH1u
MD4GCCsGAQUFBwEBBDIwMDAuBggrBgEFBQcwAYYiaHR0cDovL2lwYS1jYS5yZWRo
YXQubG9jYWwvY2Evb2NzcDAOBgNVHQ8BAf8EBAMCBPAwHQYDVR0lBBYwFAYIKwYB
BQUHAwEGCCsGAQUFBwMCMHcGA1UdHwRwMG4wbKA0oDKGMGh0dHA6Ly9pcGEtY2Eu
cmVkaGF0LmxvY2FsL2lwYS9jcmwvTWFzdGVyQ1JMLmJpbqI0pDIwMDEOMAwGA1UE
CgwFaXBhY2ExHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAdBgNVHQ4E
FgQUFajayefXyDfltCnMxcP+YXy07Ukwgf0GA1UdEQSB9TCB8oIib3ZlcmNsb3Vk
LmludGVybmFsYXBpLnJlZGhhdC5sb2NhbIIlY29udHJvbGxlci0wLmludGVybmFs
YXBpLnJlZGhhdC5sb2NhbKBKBgorBgEEAYI3FAIDoDwMOmhhcHJveHkvY29udHJv
bGxlci0wLmludGVybmFsYXBpLnJlZGhhdC5sb2NhbEBSRURIQVQuTE9DQUygWQYG
KwYBBQICoE8wTaAOGwxSRURIQVQuTE9DQUyhOzA5oAMCAQGhMjAwGwdoYXByb3h5
GyVjb250cm9sbGVyLTAuaW50ZXJuYWxhcGkucmVkaGF0LmxvY2FsMA0GCSqGSIb3
DQEBCwUAA4IBAQCtUCya4PX+ne/R+KXgz9L2SxAfYpomw5c2aRmOuIUUgjO9IixS
OziA3Ew5B5GPiFGstliePYEqdDczQlnWVdYlaPKymGVKG3vgdbS1e6APdRXFGxvm
7C4xMxjnL9rPdKPvoaZDIS7qups0fTMtfvmUq+zpFa091iEekQ/uyvZGIhDO91bs
7VjOMyDIUWiBcgfddlZA45lMEEnSXjnwF0UqPBu7k91SZrsAyHH1pumdOUajVZ4D
+p9J6e4Z0agQ7JfKwBC6drCxQ+J04ejw+NDn32ZbU2D4pX58ZfkX+uFUtwSJMm27
daYhTUSyd0YibrMcMFl8M8XIyRLKv/oc6Dls
-----END CERTIFICATE-----
subject=/O=REDHAT.LOCAL/CN=controller-0.internalapi.redhat.local
issuer=/O=REDHAT.LOCAL/CN=Certificate Authority
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2899 bytes and written 415 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 4E21E08DACE163B42C79078FFBB2440D1AA35F4A8D4AE9A083DAD8376D9A0302
    Session-ID-ctx: 
    Master-Key: 4490ADF75469FBA6921B8E73586751C5A8BE879CBEE0CDCCC44D1420519AD29D1D5F30EE1FDC3585BDA81BD334295C71
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - c4 94 84 d1 41 7a 36 78-82 cd 6a 80 92 e8 6e 68   ....Az6x..j...nh
    0010 - c8 71 d6 1d 3c ea bf bb-a6 91 2f 5d 5a 50 f3 29   .q..<...../]ZP.)
    0020 - 68 44 b6 89 08 6e c9 80-a7 54 6d 10 8d 19 6b e7   hD...n...Tm...k.
    0030 - 2e 78 f6 66 aa 33 27 4a-69 f1 78 b9 d6 2a 6a b3   .x.f.3'Ji.x..*j.
    0040 - ce 79 0a 47 4c 25 af 24-d9 9d 8f ea d0 1c 60 f6   .y.GL%.$......`.
    0050 - 7e 16 38 75 ed 40 a6 7a-9d bc bc 9b c4 d8 cd 54   ~.8u.@.z.......T
    0060 - 80 42 39 11 51 c4 3b 07-0e 8b 00 10 85 1d 3e 79   .B9.Q.;.......>y
    0070 - cd 24 46 f8 a9 d9 0b 78-9d b7 e1 86 9d 2c 26 21   .$F....x.....,&!
    0080 - f3 15 34 58 d3 2d 03 fd-5c 1e 29 5a 34 be 2c 33   ..4X.-..\.)Z4.,3
    0090 - 24 d8 1a f4 9c 85 65 09-d8 97 73 f9 39 c9 b4 9e   $.....e...s.9...

    Start Time: 1563545787
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
HTTP/1.0 408 Request Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html><body><h1>408 Request Time-out</h1>
Your browser didn't send a complete request in time.
</body></html>
closed

[heat-admin@controller-0 ~]$ sudo podman ps | grep tls
sudo: podman: command not found
[heat-admin@controller-0 ~]$ sudo docker ps | grep tls
d266c77cab6e        192.168.24.1:8787/rhosp14/openstack-glance-api:2019-06-24.2                  "kolla_start"            18 hours ago        Up 18 hours                                 glance_api_tls_proxy
e936ce4f5eef        192.168.24.1:8787/rhosp14/openstack-neutron-server:2019-06-24.2              "kolla_start"            18 hours ago        Up 18 hours                                 neutron_server_tls_proxy
81361a59b385        192.168.24.1:8787/rhosp14/openstack-swift-proxy-server:2019-06-24.2          "kolla_start"            18 hours ago        Up 18 hours                                 swift_proxy_tls_proxy
75a25b1bf8ba        192.168.24.1:8787/rhosp14/openstack-redis:pcmklatest                         "kolla_start"            18 hours ago        Up 18 hours                                 redis_tls_proxy
[heat-admin@controller-0 ~]$ exit
logout
Connection to 192.168.24.12 closed.
(overcloud) [stack@undercloud-0 ~]$ ssh heat-admin.24.20
Warning: Permanently added '192.168.24.20' (ECDSA) to the list of known hosts.
Last login: Fri Jul 19 14:13:39 2019 from 192.168.24.1
[heat-admin@controller-1 ~]$  sudo docker ps | grep tls
5d38fb1e9fcc        192.168.24.1:8787/rhosp14/openstack-glance-api:2019-06-24.2                  "kolla_start"            18 hours ago        Up 18 hours                                 glance_api_tls_proxy
455dfc90cceb        192.168.24.1:8787/rhosp14/openstack-neutron-server:2019-06-24.2              "kolla_start"            18 hours ago        Up 18 hours                                 neutron_server_tls_proxy
4c3de1974885        192.168.24.1:8787/rhosp14/openstack-swift-proxy-server:2019-06-24.2          "kolla_start"            18 hours ago        Up 18 hours                                 swift_proxy_tls_proxy
1b15648f4a06        192.168.24.1:8787/rhosp14/openstack-redis:pcmklatest                         "kolla_start"            18 hours ago        Up 18 hours                                 redis_tls_proxy
[heat-admin@controller-1 ~]$ exit
logout
Connection to 192.168.24.20 closed.
(overcloud) [stack@undercloud-0 ~]$ ssh heat-admin.24.7
Warning: Permanently added '192.168.24.7' (ECDSA) to the list of known hosts.
Last login: Fri Jul 19 14:16:02 2019 from 192.168.24.1
[heat-admin@controller-2 ~]$  sudo docker ps | grep tls
6824d2eba02c        192.168.24.1:8787/rhosp14/openstack-glance-api:2019-06-24.2                  "kolla_start"            18 hours ago        Up 18 hours                                 glance_api_tls_proxy
292c0ad500fd        192.168.24.1:8787/rhosp14/openstack-neutron-server:2019-06-24.2              "kolla_start"            18 hours ago        Up 18 hours                                 neutron_server_tls_proxy
5908a464c1a4        192.168.24.1:8787/rhosp14/openstack-swift-proxy-server:2019-06-24.2          "kolla_start"            18 hours ago        Up 18 hours                                 swift_proxy_tls_proxy
7c69d2a05e53        192.168.24.1:8787/rhosp14/openstack-redis:pcmklatest                         "kolla_start"            18 hours ago        Up 18 hours                                 redis_tls_proxy


Note You need to log in before you can comment on or make changes to this bug.