Bug 1934440

Summary: STF | In the latest OSP13 client side can not connect to the server side. SSL errors seen in the logs.
Product: Red Hat OpenStack Reporter: Leonid Natapov <lnatapov>
Component: puppet-tripleoAssignee: Martin Magr <mmagr>
Status: CLOSED ERRATA QA Contact: Leonid Natapov <lnatapov>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 13.0 (Queens)CC: ahyder, gregraka, jamsmith, jbadiapa, jjoyce, jschluet, lmadsen, m.andre, mmagr, mrunge, slinaber, spower, tvignaud
Target Milestone: z16Keywords: Regression, Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: puppet-tripleo-8.5.1-23.el7ost Doc Type: Bug Fix
Doc Text:
Before this update, the Service Telemetry Framework (STF) client could not connect to the STF server, because the latest version of Red Hat AMQ Interconnect does not allow TLS connections without a CA certificate. + This update corrects this problem by providing a new Orchestration service (heat) parameter, `MetricsQdrSSLProfiles`. + To obtain a Red Hat OpenShift TLS certificate, enter these commands: + ---- $ oc get secrets $ oc get secret/default-interconnect-selfsigned -o jsonpath='{.data.ca\.crt}' | base64 -d ---- + Add the `MetricsQdrSSLProfiles` parameter with the contents of your Red Hat OpenShift TLS certificate to a custom environment file: + ---- MetricsQdrSSLProfiles: - name: sslProfile caCertFileContent: | -----BEGIN CERTIFICATE----- ... TOpbgNlPcz0sIoNK3Be0jUcYHVMPKGMR2kk= -----END CERTIFICATE----- ---- + Then, redeploy your overcloud with the `openstack overcloud deploy` command.
 Story Points: ---
Clone Of:
: 1949168 (view as bug list) Environment:
Last Closed: 2021-06-16 10:58:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1949168, 1954722, 1982764    

Description Leonid Natapov 2021-03-03 09:24:54 UTC 
STF | In the latest OSP13 client side can not connect to the server side. SSL errors seen in the logs.

Errors from tne metrics_qdr.log:
2021-03-03 00:02:42.681359 +0000 SERVER (info) [C3788] Connection to default-interconnect-5671-service-telemetry.apps.leonidcluster.lab.upshift.rdu2.redhat.com:443 failed: amqp:connection:framing-error SSL Failure: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed
2021-03-03 00:02:48.055460 +0000 SERVER (info) [C3789] Connection to default-interconnect-5671-service-telemetry.apps.leonidcluster.lab.upshift.rdu2.redhat.com:443 failed: amqp:connection:framing-error SSL Failure: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed
2021-03-03 00:02:53.422553 +0000 SERVER (info) [C3790] Connection to default-interconnect-5671-service-telemetry.apps.leonidcluster.lab.upshift.rdu2.redhat.com:443 failed: amqp:connection:framing-error SSL Failure: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed


conf file that is used:

[root@undercloud-0 virt]# cat stf-connectors.yaml 
    parameter_defaults:
        CeilometerQdrPublishEvents: true

        CollectdAmqpInstances:
            cloud3-notify:
                format: JSON
                notify: true
                presettle: false
            cloud3-telemetry:
                format: JSON
                presettle: false
        CollectdAmqpInterval: 5
        CollectdConnectionType: amqp1
        CollectdDefaultPollingInterval: 5

        CollectdDefaultPlugins:
            - cpu
            - df
            - load
            - connectivity
            - intel_rdt
            - ipmi
            - procevent

        MetricsQdrSSLProfiles:
            - name: sslProfile

        MetricsQdrConnectors:
            - host: default-interconnect-5671-service-telemetry.apps.leonidcluster.lab.upshift.rdu2.redhat.com
              port: 443
              role: edge
              verifyHostname: false
              sslProfile: sslProfile
Comment 9 Martin Magr 2021-03-23 11:15:12 UTC 
*** Bug 1936560 has been marked as a duplicate of this bug. ***
Comment 18 Martin Magr 2021-04-07 12:55:32 UTC 
Attached patch enables TripleO to distribute SSL certificates and hence enables STF client side message bus to connect to server side message bus even with latest AMQ Interconnect versions
Comment 35 Leonid Natapov 2021-05-07 10:23:15 UTC 
To extract OCP certificate run the following command on your OCP:

1.oc get secrets  - (you will get a list)

You should see in the list  default-interconnect-selfsigned 


2.oc get secret/default-interconnect-selfsigned -o jsonpath='{.data.ca\.crt}' | base64 -d

You will get a decode of the certificate that will look like:

-----BEGIN CERTIFICATE-----
MIIDpjCCAo6gAwIBAgIQFE4Z7BXVg+paMsLqKl4fnzANBgkqhkiG9w0BAQsFADBf
MRUwEwYDVQQKEwxjZXJ0LW1hbmFnZXIxQTA/BgNVBAMTOGRlZmF1bHQtaW50ZXJj
b25uZWN0LnNlcnZpY2UtdGVsZW1ldHJ5LnN2Yy5jbHVzdGVyLmxvYeFsMB4XDTIx
MDQwODEzMTAwMFoXDTIxMDcwNzEzMTAwMFowWjEVMBMGA1UEChMMY2VydC1tYW5h
Z2VyMUEwPwYDVQQDEzhkZWZhdWx0LWludGVyY29ubmVjdC5zZXJ2aWNlLXRlbGVt
ZXRyeS5zdmMuY2x1c3Rlci5sb2NhbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC
AQoCggEBALB8oTZJuLT0jy2wf4tkiLuxY58Uo0KiHh+dMXWscZ1voLEeATLIoyrK
eUa2sBmPBsHdLt4nO34nEeBnNzoKQR8XLc5F8x4WM8mQPv0KjpWJPeskPvvAfeWT
XcXBxf5af7HvtzeA+zL2onucaBeEcAbNIEm4Elz3d/BT70w1z235J3a2JMACFGcy
kZVS74PxXjx65b57rPbtf0Gnlf0cSObHfJ8n3N7tlAMgfErjGByHLDuEQ8nxzh0N
EZaC3yhbLE0IdhLM0V+WPvkUQvrflL2x2K0zboKpcLbGdRc99UjbViLTM6E3prx+
GzEqcExDyMWX4kdhd+rBJHpuZ2AVhPMCAwEAAaNoMGYwDgYDVR0PAQH/BAQDAgKk
MA8GA1UdEwEB/wQFMAMBAf8wQwYDVR0RBDwwOoI4ZGVmYXVsdC1pbnRlcmNvbm5l
Y3Quc2VydmljZS10ZWxlbWV0cnkuc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcN
AQELBQADggEBAIi1P31PtEk9nCSKBiPtOGl+RBHZlhoIMPZhLAs1BCmPnjUdBjpq
wS6IjihePVSX7mfb5o4TJTz3qlx/OfiyfWZ2+jKttJ8hOEjaxdspEnJ9n4ska9BP
eEHM5Xu5djIXRJKHihcrzmecFCMTc9R9kkHqQI8cithG4aXeg8I/KsZ1Isfa4w8y
lDNS0Zwk7mf40b5kwG83VB+dfiqvsO/ODTw7xu7aAr8TtnnMoHFfnw/wqr3XEJzw
pwBOkI9C7cq7unApiPBfomQFEXIWdTaU/7sy1Dt238rt/sWbznwcXu3M673oq0JV
TOpbgNlPcz0sIoNK3Be0jUcYHVMPKGMR2kk=
-----END CERTIFICATE-----


3 Edit stf custom template e.g. (stf-connectors.yaml) and add the following content:

MetricsQdrSSLProfiles:
    -   name: sslProfile
        caCertFileContent: |
           -----BEGIN CERTIFICATE-----
           MIIDpjCCAo6gAwIBAgIQFE4Z7BXVg+paMsLqKl4fnzANBgkqhkiG9w0BAQsFADBa
           MRUwEwYDVQQKEwxjZXJ0LW1hbmFnZXIxQTA/BgNVBAMTOGRlZmF1bHQtaW50ZXJj
           b25uZWN0LnNlcnZpY2UtdGVsZW1ldHJ5LnN2Yy5jbHVzdGVyLmxvY2FsMB4XDTIx
           MDQwODEzMTAwMFoXDTIxMDcwNzEzMTAwMFowWjEVMBMGA1UEChMMY2VydC1tYW5h
           Z2VyMUEwPwYDVQQDEzhkZWZhdWx0LWludGVyY29ubmVjdC5zZXJ2aWNlLXRlbGVt
           ZXRyeS5zdmMuY2x1c3Rlci5sb2NhbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC
           AQoCggEBALB8oTZJuLT0jy2wf4tkiLuxY58Uo0KiHh+dMXWbcZ1voLEeATLIoyrK
           eUa2sBmPBsHdLt4nO34nEeBnNzoKQR8XLc5F8x4WM8mQPv0KsdfJPeskPvvAfeWT
           XcXBxf5af7HvtzeA+zL2onucaBeEcAbNIEm4Elz3d/BT70w1z235J3a2JMACFGcy
           kZVS74PxXjx65b57rPbtf0Gnlf0cSObHfJ8n3N7tlAMgfErjGByHLDuEQ8nxzh0N
           EZaC3yhbLE0IdhLM0V+WPvkUQvrflL2x2K0zboKpcLbGdRc9ertbViLTM6E3prx+
           GzEqcExDyMWX4kdhd+rBJHpuZ2AVhPMCAwEAAaNoMGYwDgYDVR0PAQH/BAQDAgKk
           MA8GA1UdEwEB/wQFMAMBAf8wQwYDVR0RBDwwOoI4ZGVh7tVsdC1pbnRlcmNvbm5l
           Y3Quc2VydmljZS10ZWxlbWV0cnkuc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcN
           AQELBQADggEBAIi1P31PtEk9nCSKBiPtOGl+RBHZlhoIMPZhLAs1BCmPnjUdBjpq
           wS6IjihePVSX7mfb5o4TJTz3qlx/OfiyfWZ2+jKttJ8hOEjaxdspEnJ9n4ska9BP
           eEHM5Xu5djIXRJKHihcrzmecFCMTc9R9kkHqQI8cithG4aXeg8I/KsZ1Isfa4w8y
           lDNS0Zwk7mf40b5kwG83VB+dfiqvsO/ODTw7xu7aAr8TtnnMoHFfnw/wqr3XEJzw
           pwBOkI9C7cq7unApiPBfomQFEXIWdTaU/7sy1Dt238rt/sWbznwcXu3M673oq0JV
           TOpbgNlPcz0sIoNK3Be0jUcYHVMPKGMR2kk=
           -----END CERTIFICATE-----


4. Deploy overcloud.



-------------

BZ verified:

OSP was able to connect to the OCP

2021-05-07 09:40:32.390553 +0000 CONN_MGR (info) Created SSL Profile with name sslProfile
2021-05-07 09:40:32.399273 +0000 CONN_MGR (info) Configured Connector: default-interconnect-5671-service-telemetry.apps.leonidcluster.lab.upshift.rdu2.redhat.com:443 proto=any, role=edge, sslProfile=sslProfile
2021-05-07 09:40:32.401534 +0000 CONN_MGR (info) Configured Listener: 172.17.1.106:5666 proto=any, role=normal
2021-05-07 09:40:32.403365 +0000 SERVER (notice) Operational, 4 Threads Running (process ID 9)
2021-05-07 09:40:32.404051 +0000 SERVER (notice) Process VmSize 207.77 MiB (31.26 GiB available memory)
2021-05-07 09:40:32.404493 +0000 SERVER (notice) Listening on 172.17.1.106:5666
2021-05-07 09:40:33.297709 +0000 ROUTER (info) [C1] Connection Opened: dir=out host=default-interconnect-5671-service-telemetry.apps.leonidcluster.lab.upshift.rdu2.redhat.com:443 vhost= encrypted=TLSv1/SSLv3 auth=ANONYMOUS user=(null) container_id=default-interconnect-69d77454b5-wqbp4 props={:product="qpid-dispatch-router", :version="Red Hat Interconnect 1.10.0 (qpid-dispatch 1.14.0)", :"qd.conn-id"=953349}
2021-05-07 09:40:33.297818 +0000 ROUTER_CORE (info) Edge connection (id=1) to interior established
2021-05-07 09:40:33.297997 +0000 ROUTER_CORE (info) [C1][L6] Link attached: dir=out source={<none> expire:link} target={<none> expire:link}
2021-05-07 09:40:33.298085 +0000 ROUTER_CORE (info) [C1][L7] Link attached: dir=in source={Router.controller-0.redhat.local expire:link caps::"qd.router-edge-downlink"} target={<none> expire:link caps::"qd.router-edge-downlink"}
2021-05-07 09:40:33.298124 +0000 ROUTER_CORE (info) [C1][L8] Link attached: dir=in source={_$qd.edge_addr_tracking expire:link} target={<none> expire:link}
2021-05-07 09:40:33.298177 +0000 ROUTER_CORE (info) [C1][L9] Link attached: dir=out source={} target={$management expire:link}
2021-05-07 09:40:33.298190 +0000 ROUTER_CORE (info) [C1][L10] Link attached: dir=in source={<dynamic> expire:link} target={}
2021-05-07 09:40:33.298208 +0000 ROUTER_CORE (info) [C1][L11] Link attached: dir=out source={} target={_$qd.addr_lookup expire:link}
2021-05-07 09:40:33.298219 +0000 ROUTER_CORE (info) [C1][L12] Link attached: dir=in source={<dynamic> expire:link} target={}
2021-05-07 09:40:34.894964 +0000 SERVER (info) [C2] Accepted connection to 172.17.1.106:5666 from 172.17.1.106:46324
2021-05-07 09:40:34.901265 +0000 ROUTER (info) [C2] Connection Opened: dir=in host=172.17.1.106:46324 vhost= encrypted=no auth=no user=anonymous container_id=openstack.org/om/container/controller-0/ceilometer-agent-notification/28/2ad8e44547724d6db23838a290f15a64 props={:process=b"ceilometer-agent-notification", :node=b"controller-0", :pid=28}
2021-05-07 09:40:34.908614 +0000 ROUTER_CORE (info) [C2][L13] Link attached: dir=out source={<dynamic> expire:sess} target={rpc-response expire:sess}
2021-05-07 09:40:34.917686 +0000 ROUTER_CORE (info) [C2][L14] Link attached: dir=in source={/anycast/ceilometer/cloud2-metering.sample expire:sess} target={/anycast/ceilometer/cloud2-metering.sample expire:sess}
2021-05-07 09:40:35.060918 +0000 ROUTER_CORE (info) [C1][L15] Link attached: dir=out source={<none> expire:link} target={anycast/ceilometer/cloud2-metering.sample expire:link}
2021-05-07 09:40:39.314158 +0000 SERVER (info) [C3] Accepted connection to 172.17.1.106:5666 from 172.17.1.106:46590
2021-05-07 09:40:39.315462 +0000 ROUTER (info) [C3] Connection Opened: dir=in host=172.17.1.106:46590 vhost= encrypted=no auth=ANONYMOUS user=anonymous container_id=metrics props=
2021-05-07 09:40:39.315656 +0000 ROUTER_CORE (info) [C3][L16] Link attached: dir=in source={<none> expire:sess} target={<none> expire:sess}
Comment 41 errata-xmlrpc 2021-06-16 10:58:58 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 13.0 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2385