Bug 1811635
| Summary: | [Octavia] "sni_container_refs" empty after updating a Terminated HTTPS listener | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Bruna Bonguardo <bbonguar> |
| Component: | openstack-octavia | Assignee: | Carlos Goncalves <cgoncalves> |
| Status: | CLOSED ERRATA | QA Contact: | Bruna Bonguardo <bbonguar> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 16.0 (Train) | CC: | gthiemon, ihrachys, lpeer, majopela, michjohn, njohnston, scohen |
| Target Milestone: | z2 | Keywords: | Triaged |
| Target Release: | 16.1 (Train on RHEL 8.2) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-octavia-5.0.3-0.20200717203413.b20bdf1.el8ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-10-28 15:36:49 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Bruna Bonguardo
2020-03-09 11:57:52 UTC
Also happening when updating an already functioning TLS terminated listener with SNI: [2020-03-09 12:22:19] (tester) [stack@undercloud-0 ~]$ openstack loadbalancer listener show listener1 +-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | True | | connection_limit | -1 | | created_at | 2020-03-09T15:36:56 | | default_pool_id | 091267c5-7f6a-4d64-bdc9-97d4a6dec4f0 | | default_tls_container_ref | http://10.0.0.129:9311/v1/secrets/4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac | | description | | | id | 255f8d78-a427-40bd-8ee6-b070309bd44d | | insert_headers | None | | l7policies | | | loadbalancers | 252acde2-1fa0-407f-bf83-86844f3880d1 | | name | listener1 | | operating_status | ONLINE | | project_id | b6cd9962adde4b4fbb4c63206f561f9e | | protocol | TERMINATED_HTTPS | | protocol_port | 443 | | provisioning_status | ACTIVE | | sni_container_refs | ['http://10.0.0.129:9311/v1/secrets/3019549d-6be2-474a-a919-50f0ca955ff6', 'http://10.0.0.129:9311/v1/secrets/4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac'] | | timeout_client_data | 50000 | | timeout_member_connect | 5000 | | timeout_member_data | 50000 | | timeout_tcp_inspect | 0 | | updated_at | 2020-03-09T15:45:11 | | client_ca_tls_container_ref | None | | client_authentication | NONE | | client_crl_container_ref | None | | allowed_cidrs | None | +-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ [2020-03-09 12:22:43] (tester) [stack@undercloud-0 ~]$ [2020-03-09 12:23:04] (tester) [stack@undercloud-0 ~]$ [2020-03-09 12:23:04] (tester) [stack@undercloud-0 ~]$ [2020-03-09 12:23:04] (tester) [stack@undercloud-0 ~]$ [2020-03-09 12:23:04] (tester) [stack@undercloud-0 ~]$ [2020-03-09 12:23:05] (tester) [stack@undercloud-0 ~]$ [2020-03-09 12:23:05] (tester) [stack@undercloud-0 ~]$ openstack secret list +------------------------------------------------------------------------+-------------+---------------------------+--------+-----------------------------------------+-----------+------------+-------------+------+------------+ | Secret href | Name | Created | Status | Content types | Algorithm | Bit length | Secret type | Mode | Expiration | +------------------------------------------------------------------------+-------------+---------------------------+--------+-----------------------------------------+-----------+------------+-------------+------+------------+ | http://10.0.0.129:9311/v1/secrets/3019549d-6be2-474a-a919-50f0ca955ff6 | tls_secret2 | 2020-03-09T09:54:30+00:00 | ACTIVE | {'default': 'application/octet-stream'} | aes | 256 | opaque | cbc | None | | http://10.0.0.129:9311/v1/secrets/4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac | tls_secret1 | 2020-03-08T09:24:28+00:00 | ACTIVE | {'default': 'application/octet-stream'} | aes | 256 | opaque | cbc | None | | http://10.0.0.129:9311/v1/secrets/972f6889-e4df-4adc-9e80-a0483c521916 | tls_secret3 | 2020-03-09T16:22:01+00:00 | ACTIVE | {'default': 'application/octet-stream'} | aes | 256 | opaque | cbc | None | +------------------------------------------------------------------------+-------------+---------------------------+--------+-----------------------------------------+-----------+------------+-------------+------+------------+ [2020-03-09 12:23:12] (tester) [stack@undercloud-0 ~]$ openstack loadbalancer listener set listener1 --sni-container-refs $(openstack secret list | awk '/ tls_secret1 / {print $2}') $(openstack secret list | awk '/ tls_secret3 / {print $2}') [2020-03-09 12:27:58] (tester) [stack@undercloud-0 ~]$ [2020-03-09 12:27:58] (tester) [stack@undercloud-0 ~]$ [2020-03-09 12:28:03] (tester) [stack@undercloud-0 ~]$ openstack loadbalancer listener show listener1 +-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | True | | connection_limit | -1 | | created_at | 2020-03-09T15:36:56 | | default_pool_id | 091267c5-7f6a-4d64-bdc9-97d4a6dec4f0 | | default_tls_container_ref | http://10.0.0.129:9311/v1/secrets/4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac | | description | | | id | 255f8d78-a427-40bd-8ee6-b070309bd44d | | insert_headers | None | | l7policies | | | loadbalancers | 252acde2-1fa0-407f-bf83-86844f3880d1 | | name | listener1 | | operating_status | ONLINE | | project_id | b6cd9962adde4b4fbb4c63206f561f9e | | protocol | TERMINATED_HTTPS | | protocol_port | 443 | | provisioning_status | ACTIVE | | sni_container_refs | ['http://10.0.0.129:9311/v1/secrets/3019549d-6be2-474a-a919-50f0ca955ff6', 'http://10.0.0.129:9311/v1/secrets/4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac'] | | timeout_client_data | 50000 | | timeout_member_connect | 5000 | | timeout_member_data | 50000 | | timeout_tcp_inspect | 0 | | updated_at | 2020-03-09T16:28:00 | | client_ca_tls_container_ref | None | | client_authentication | NONE | | client_crl_container_ref | None | | allowed_cidrs | None | +-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ From logs: #Grep tls_secret3 id: [root@controller-1 octavia]# cat octavia.log | grep a0483c521916 2020-03-09 16:27:55.029 27 INFO barbicanclient.base [req-71d380f8-00c1-4a2b-a93a-2c915cd67c1d - b6cd9962adde4b4fbb4c63206f561f9e - default default] Calculated Containers uuid ref: containers/972f6889-e4df-4adc-9e80-a0483c521916 2020-03-09 16:27:55.046 27 INFO octavia.certificates.manager.barbican [req-71d380f8-00c1-4a2b-a93a-2c915cd67c1d - b6cd9962adde4b4fbb4c63206f561f9e - default default] Loading certificate secret http://10.0.0.129:9311/v1/secrets/972f6889-e4df-4adc-9e80-a0483c521916 from Barbican. 2020-03-09 16:27:55.046 27 INFO barbicanclient.base [req-71d380f8-00c1-4a2b-a93a-2c915cd67c1d - b6cd9962adde4b4fbb4c63206f561f9e - default default] Calculated Secrets uuid ref: secrets/972f6889-e4df-4adc-9e80-a0483c521916 2020-03-09 16:27:55.075 27 INFO barbicanclient.base [req-71d380f8-00c1-4a2b-a93a-2c915cd67c1d - b6cd9962adde4b4fbb4c63206f561f9e - default default] Calculated Secrets uuid ref: secrets/972f6889-e4df-4adc-9e80-a0483c521916 2020-03-09 16:27:55.212 27 INFO octavia.certificates.manager.barbican [req-628ced30-6b4b-431c-87ef-d6e9ea58e4ed - - - - -] Loading certificate secret http://10.0.0.129:9311/v1/secrets/972f6889-e4df-4adc-9e80-a0483c521916 from Barbican. 2020-03-09 16:27:55.214 27 INFO barbicanclient.base [req-628ced30-6b4b-431c-87ef-d6e9ea58e4ed - - - - -] Calculated Secrets uuid ref: secrets/972f6889-e4df-4adc-9e80-a0483c521916 2020-03-09 16:27:55.245 27 INFO barbicanclient.base [req-628ced30-6b4b-431c-87ef-d6e9ea58e4ed - - - - -] Calculated Secrets uuid ref: secrets/972f6889-e4df-4adc-9e80-a0483c521916 #Grep listenerID: [root@controller-0 octavia]# cat worker.log | grep 255f8d78-a427-40bd-8ee6-b070309bd44d 2020-03-09 16:27:55.507 23 INFO octavia.controller.queue.v1.endpoints [-] Updating listener '255f8d78-a427-40bd-8ee6-b070309bd44d'... [root@controller-1 octavia]# cat octavia.log | grep 255f8d78-a427-40bd-8ee6-b070309bd44d 2020-03-09 16:27:55.476 27 INFO octavia.api.v2.controllers.listener [req-079831b3-e3ab-4b62-a123-2a059357a9fe - b6cd9962adde4b4fbb4c63206f561f9e - - -] Sending update Listener 255f8d78-a427-40bd-8ee6-b070309bd44d to provider amphora I was able to reproduce this issue in OSP 16.0 z1 and upstream master branch (devstack). Tested in:
16.1 -p RHOS-16.1-RHEL-8-20200930.n.0
First test:
1) Created environment:1 load balancer, 1 terminated https listener, 1 pool with 2 member servers, 1 health monitor.
The terminated https listener points to a barbican container called tls_secret1.
The clients access the VIP with the address www.example.com (added in /etc/hosts file)
2)Another barbican secret was added, called tls_secret2, for the address www1.example.com
I also added www1.example.com to point to the VIP address in /etc/hosts file.
Added tls_secret2 as a sni_container for the listener:
[2020-10-05 12:42:51] (tester) [stack@undercloud-0 ~]$ openstack loadbalancer listener set listener1 --sni-container-refs $(openstack secret list | awk '/ tls_secret1 / {print $2}') $(openstack secret list | awk '/ tls_secret13 / {print $2}')
3) Tested traffic to both urls:
[2020-10-05 12:43:22] (tester) [stack@undercloud-0 ~]$ req='curl -k https://www.example.com'; for i in {1..10}; do $req; echo; donesrv-server2-jcxedzurzrw6
srv-server1-dyistuzpwrz4
srv-server2-jcxedzurzrw6
srv-server1-dyistuzpwrz4
srv-server2-jcxedzurzrw6
srv-server1-dyistuzpwrz4
srv-server2-jcxedzurzrw6
srv-server1-dyistuzpwrz4
srv-server2-jcxedzurzrw6
srv-server1-dyistuzpwrz4
[2020-10-05 12:43:26] (tester) [stack@undercloud-0 ~]$ req='curl -k https://www1.example.com'; for i in {1..10}; do $req; echo; done
srv-server2-jcxedzurzrw6
srv-server1-dyistuzpwrz4
srv-server2-jcxedzurzrw6
srv-server1-dyistuzpwrz4
srv-server2-jcxedzurzrw6
srv-server1-dyistuzpwrz4
srv-server2-jcxedzurzrw6
srv-server1-dyistuzpwrz4
srv-server2-jcxedzurzrw6
srv-server1-dyistuzpwrz4
4) "Listener Show" command now shows the new barbican secret under "sni_container_refs":
[2020-10-05 12:43:32] (tester) [stack@undercloud-0 ~]$ openstack loadbalancer listener show listener1
+-----------------------------+----------------------------------------------------------------------------+
| Field | Value |
+-----------------------------+----------------------------------------------------------------------------+
| admin_state_up | True |
| connection_limit | -1 |
| created_at | 2020-10-05T16:04:23 |
| default_pool_id | 9b2aeb31-2cb4-41de-9724-e7728b2b2071 |
| default_tls_container_ref | http://10.0.0.131:9311/v1/secrets/66133473-e42a-4313-8ec4-a6760a662bcf |
| description | |
| id | 6af78ef1-0eef-463b-8dee-dbd7d3d3185f |
| insert_headers | None |
| l7policies | |
| loadbalancers | b27b318a-0262-4607-a92c-76dc26ce4411 |
| name | listener1 |
| operating_status | ONLINE |
| project_id | c6dcb43ccaf34ca7a77d6c35ae5e3230 |
| protocol | TERMINATED_HTTPS |
| protocol_port | 443 |
| provisioning_status | ACTIVE |
| sni_container_refs | ['http://10.0.0.131:9311/v1/secrets/dc914afb-ec05-40e9-906c-06be14445114'] |
| timeout_client_data | 50000 |
| timeout_member_connect | 5000 |
| timeout_member_data | 50000 |
| timeout_tcp_inspect | 0 |
| updated_at | 2020-10-05T16:43:20 |
| client_ca_tls_container_ref | None |
| client_authentication | NONE |
| client_crl_container_ref | None |
| allowed_cidrs | None |
+-----------------------------+----------------------------------------------------------------------------+
5) Also, the certificates are the right ones:
[2020-10-05 12:44:46] (tester) [stack@undercloud-0 ~]$ openssl x509 -inform pem -noout -text -in testca/testcert.pem
Certificate:
Data:
Version: 1 (0x0)
Serial Number: 1 (0x1)
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, ST = Denial, L = Springfield, O = Dis, CN = www.example.com
Validity
Not Before: Oct 5 15:54:32 2020 GMT
Not After : Oct 3 15:54:32 2030 GMT
Subject: C = US, ST = Denial, O = Dis, CN = www.example.com
Subject Public Key Info:
[...]
[2020-10-05 12:44:57] (tester) [stack@undercloud-0 ~]$ echo | openssl s_client -showcerts -servername www.example.com -connect www.example.com:443 2>/dev/null | openssl x509 -inform pem -noout -text
Certificate:
Data:
Version: 1 (0x0)
Serial Number: 1 (0x1)
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, ST = Denial, L = Springfield, O = Dis, CN = www.example.com
Validity
Not Before: Oct 5 15:54:32 2020 GMT
Not After : Oct 3 15:54:32 2030 GMT
Subject: C = US, ST = Denial, O = Dis, CN = www.example.com
Subject Public Key Info:
[...]
[2020-10-05 12:45:14] (tester) [stack@undercloud-0 ~]$
[2020-10-05 12:46:07] (tester) [stack@undercloud-0 ~]$ openssl x509 -inform pem -noout -text -in newcert2/testca/testcert.pem
Certificate:
Data:
Version: 1 (0x0)
Serial Number: 1 (0x1)
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, ST = Denial, L = Springfield, O = Dis, CN = www1.example.com
Validity
Not Before: Oct 5 16:35:39 2020 GMT
Not After : Oct 3 16:35:39 2030 GMT
Subject: C = US, ST = Denial, O = Dis, CN = www1.example.com
Subject Public Key Info:
[...]
[2020-10-05 12:46:13] (tester) [stack@undercloud-0 ~]$ echo | openssl s_client -showcerts -servername www1.example.com -connect www.example.com:443 2>/dev/null | openssl x509 -inform pem -noout -text
Certificate:
Data:
Version: 1 (0x0)
Serial Number: 1 (0x1)
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, ST = Denial, L = Springfield, O = Dis, CN = www1.example.com
Validity
Not Before: Oct 5 16:35:39 2020 GMT
Not After : Oct 3 16:35:39 2030 GMT
Subject: C = US, ST = Denial, O = Dis, CN = www1.example.com
Subject Public Key Info:
[...]
Second test:
1) Updating a sni container to a new one (from tls_secret13 to tls_secret14):
[2020-10-05 12:56:25] (tester) [stack@undercloud-0 testca]$ openstack loadbalancer listener set listener1 --sni-container-refs $(openstack secret list | awk '/ tls_secret1 / {print $2}') $(openstack secret list | awk '/ tls_secret14 / {print $2}')
2) Listener is updated as expected:
[2020-10-05 12:57:15] (tester) [stack@undercloud-0 testca]$ openstack loadbalancer listener show listener1
+-----------------------------+----------------------------------------------------------------------------+
| Field | Value |
+-----------------------------+----------------------------------------------------------------------------+
| admin_state_up | True |
| connection_limit | -1 |
| created_at | 2020-10-05T16:04:23 |
| default_pool_id | 9b2aeb31-2cb4-41de-9724-e7728b2b2071 |
| default_tls_container_ref | http://10.0.0.131:9311/v1/secrets/66133473-e42a-4313-8ec4-a6760a662bcf |
| description | |
| id | 6af78ef1-0eef-463b-8dee-dbd7d3d3185f |
| insert_headers | None |
| l7policies | |
| loadbalancers | b27b318a-0262-4607-a92c-76dc26ce4411 |
| name | listener1 |
| operating_status | ONLINE |
| project_id | c6dcb43ccaf34ca7a77d6c35ae5e3230 |
| protocol | TERMINATED_HTTPS |
| protocol_port | 443 |
| provisioning_status | ACTIVE |
| sni_container_refs | ['http://10.0.0.131:9311/v1/secrets/22223562-a8d9-46bd-b03e-ff476aaefb58'] |
| timeout_client_data | 50000 |
| timeout_member_connect | 5000 |
| timeout_member_data | 50000 |
| timeout_tcp_inspect | 0 |
| updated_at | 2020-10-05T16:57:05 |
| client_ca_tls_container_ref | None |
| client_authentication | NONE |
| client_crl_container_ref | None |
| allowed_cidrs | None |
+-----------------------------+----------------------------------------------------------------------------+
Moving the bug to VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:4284 |