"sni_container_refs" empty after updating a Terminated HTTPS listener. Version: 16-trunk -p RHOS_TRUNK-16.0-RHEL-8-20200226.n.1 Environment: 1 load balancer, 1 terminated https listener, 1 pool with 2 member servers, 1 health monitor. The terminated https listener points to a barbican container called tls_secret1. The clients access the VIP with the address www.example.com (added in /etc/hosts file) Another barbican secret was added, called tls_secret2, for the address www1.example.com I also added www1.example.com to point to the VIP address in /etc/hosts file. I want to add tls_secret2 as a sni_container for the listener. When doing so, I don't receive any error message from the API. Also, tls_secret2 URI is not showing under "sni_containers_ref" in listener show command: Before: (tester) [stack@undercloud-0 ~]$ openstack secret list +------------------------------------------------------------------------+-------------+---------------------------+--------+-----------------------------------------+-----------+------------+-------------+------+------------+ | Secret href | Name | Created | Status | Content types | Algorithm | Bit length | Secret type | Mode | Expiration | +------------------------------------------------------------------------+-------------+---------------------------+--------+-----------------------------------------+-----------+------------+-------------+------+------------+ | http://10.0.0.129:9311/v1/secrets/4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac | tls_secret1 | 2020-03-08T09:24:28+00:00 | ACTIVE | {'default': 'application/octet-stream'} | aes | 256 | opaque | cbc | None | | http://10.0.0.129:9311/v1/secrets/3019549d-6be2-474a-a919-50f0ca955ff6 | tls_secret2 | 2020-03-09T09:54:30+00:00 | ACTIVE | {'default': 'application/octet-stream'} | aes | 256 | opaque | cbc | None | +------------------------------------------------------------------------+-------------+---------------------------+--------+-----------------------------------------+-----------+------------+-------------+------+------------+ (tester) [stack@undercloud-0 ~]$ openstack loadbalancer listener show listener1 +-----------------------------+------------------------------------------------------------------------+ | Field | Value |+-----------------------------+------------------------------------------------------------------------+ | admin_state_up | True || connection_limit | -1 || created_at | 2020-03-08T13:58:39 || default_pool_id | 091267c5-7f6a-4d64-bdc9-97d4a6dec4f0 || default_tls_container_ref | http://10.0.0.129:9311/v1/secrets/4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac || description | || id | 03365412-ad59-4333-b28e-bb46f9110ac9 || insert_headers | None || l7policies | || loadbalancers | 252acde2-1fa0-407f-bf83-86844f3880d1 || name | listener1 || operating_status | ONLINE || project_id | b6cd9962adde4b4fbb4c63206f561f9e || protocol | TERMINATED_HTTPS || protocol_port | 443 || provisioning_status | ACTIVE || sni_container_refs | [] || timeout_client_data | 50000 || timeout_member_connect | 5000 || timeout_member_data | 50000 || timeout_tcp_inspect | 0 || updated_at | 2020-03-08T15:33:20 || client_ca_tls_container_ref | None || client_authentication | NONE || client_crl_container_ref | None || allowed_cidrs | None |+-----------------------------+------------------------------------------------------------------------+ Updating the listener: (tester) [stack@undercloud-0 ~]$ openstack loadbalancer listener set listener1 --sni-container-refs $(openstack secret list | awk '/ tls_secret1 / {print $2}') $(openstack secret list | awk '/ tls_secret2 / {print $2}') (tester) [stack@undercloud-0 ~]$ After: Load balancer still works: #Connecting to www.example.com (tls_secret1): (tester) [stack@undercloud-0 ~]$ req='curl -k https://www.example.com'; for i in {1..10}; do $req; echo; done serverstack-server2-gixxtgltmvud serverstack-server1-tg74ymitoupy serverstack-server2-gixxtgltmvud serverstack-server1-tg74ymitoupy serverstack-server2-gixxtgltmvud serverstack-server1-tg74ymitoupy serverstack-server2-gixxtgltmvud serverstack-server1-tg74ymitoupy serverstack-server2-gixxtgltmvud serverstack-server1-tg74ymitoupy #Connecting to www1.example.com (tls_secret2) - MAYBE it works because www1.example.com also points to the same VIP as www.example.com in the /etc/hosts file: (tester) [stack@undercloud-0 ~]$ req='curl -k https://www1.example.com'; for i in {1..10}; do $req; echo; done serverstack-server2-gixxtgltmvud serverstack-server1-tg74ymitoupy serverstack-server2-gixxtgltmvud serverstack-server1-tg74ymitoupy serverstack-server2-gixxtgltmvud serverstack-server1-tg74ymitoupy serverstack-server2-gixxtgltmvud serverstack-server1-tg74ymitoupy serverstack-server2-gixxtgltmvud serverstack-server1-tg74ymitoupy But the SNI containers don't show when running "openstack loadbalancer listener show", the "sni_container_refs" is empty. (tester) [stack@undercloud-0 ~]$ openstack loadbalancer listener show listener1 +-----------------------------+------------------------------------------------------------------------+ | Field | Value | +-----------------------------+------------------------------------------------------------------------+ | admin_state_up | True | | connection_limit | -1 | | created_at | 2020-03-08T13:58:39 | | default_pool_id | 091267c5-7f6a-4d64-bdc9-97d4a6dec4f0 | | default_tls_container_ref | http://10.0.0.129:9311/v1/secrets/4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac | | description | | | id | 03365412-ad59-4333-b28e-bb46f9110ac9 | | insert_headers | None | | l7policies | | | loadbalancers | 252acde2-1fa0-407f-bf83-86844f3880d1 | | name | listener1 | | operating_status | ONLINE | | project_id | b6cd9962adde4b4fbb4c63206f561f9e | | protocol | TERMINATED_HTTPS | | protocol_port | 443 | | provisioning_status | ACTIVE | | sni_container_refs | [] | | timeout_client_data | 50000 | | timeout_member_connect | 5000 | | timeout_member_data | 50000 | | timeout_tcp_inspect | 0 | | updated_at | 2020-03-09T11:18:41 | | client_ca_tls_container_ref | None | | client_authentication | NONE | | client_crl_container_ref | None | | allowed_cidrs | None | +-----------------------------+------------------------------------------------------------------------+ When connecting to www.example.com, we get the desired certificate as seen below: #Server certificate: (tester) [stack@undercloud-0 ~]$ openssl x509 -inform pem -noout -text -in cert1/testca/testcert.pem Certificate: Data: Version: 1 (0x0) Serial Number: 1 (0x1) Signature Algorithm: sha256WithRSAEncryption Issuer: C = US, ST = Denial, L = Springfield, O = Dis, CN = www.example.com Validity Not Before: Mar 5 11:40:31 2020 GMT Not After : Mar 3 11:40:31 2030 GMT Subject: C = US, ST = Denial, O = Dis, CN = www.example.com Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public-Key: (2048 bit) Modulus: 00:bf:b4:ed:0d:57:b6:2e:f4:02:5d:09:e7:10:fc: e1:6f:a4:75:70:9c:1d:9c:55:cc:70:17:67:1c:b2: 36:bc:45:d6:32:23:fd:64:fb:4b:97:ec:dd:36:60: 13:7e:aa:f0:28:ef:a5:b1:fb:df:6b:13:5f:36:ff: 31:2f:f0:79:ef:4a:b6:2e:1b:c9:aa:f3:1c:9e:02: 66:67:4b:5a:f5:27:9b:cc:0b:5b:30:38:61:ee:bc: 2d:ba:a5:65:c0:c9:68:da:bc:f9:ea:35:bf:b9:e3: cb:60:b6:a9:f8:8e:f8:6b:54:c5:06:d7:94:c4:5d: f8:89:fb:95:85:16:c3:c3:95:05:eb:f5:bd:a4:8b: eb:5f:e2:cb:7a:a9:27:a1:8e:3a:2c:ca:81:0b:60: 8f:54:3b:f1:f2:59:2a:69:b1:79:e4:8a:af:9b:10: ca:9b:9f:0e:40:cc:69:df:84:e3:ae:e8:7c:75:67: c3:6f:81:3b:36:c0:14:ea:6a:be:fd:26:4e:c9:71: a9:db:95:b7:e4:c6:e4:a7:9c:4c:7a:2f:ac:7f:2e: f5:ad:c1:db:e2:80:ac:92:e8:2a:2e:31:41:1f:82: e8:f4:b5:b2:b8:44:12:6e:6c:ce:af:82:2c:f3:9f: 3b:86:bd:ae:55:9b:08:a5:0c:2e:3f:d2:72:f4:e1: 7a:67 Exponent: 65537 (0x10001) Signature Algorithm: sha256WithRSAEncryption 7d:cf:bb:2c:61:e9:d7:10:48:a3:b1:77:8c:f8:b9:25:53:25: cf:71:5b:52:9c:b1:e2:2e:01:32:44:ac:f6:1b:d3:3d:1c:72: d6:4d:de:78:c3:12:db:63:6b:7b:79:ec:d9:da:47:d8:74:04: f6:a3:29:02:08:a5:a8:a4:b5:94:bc:23:c8:82:99:1b:9a:3d: ee:3e:79:c7:30:21:4e:4f:ea:70:ae:05:55:6b:7c:4e:23:ec: fe:dd:56:0f:8a:af:70:88:70:5f:42:d4:28:ca:26:0b:3a:cc: f7:48:3d:c1:e8:58:99:7b:00:c1:f7:71:06:ad:e2:9e:db:00: 8c:03:9c:56:02:6b:4a:6b:d1:a3:7d:b5:e6:99:e0:03:8b:46: 0b:ed:ee:ba:af:c1:0d:29:0e:eb:83:11:3b:f1:11:f6:bc:a5: db:6b:4a:f2:10:11:44:b9:01:b3:5f:c0:1e:7f:99:0b:08:4d: 7f:07:19:19:d6:fd:10:3c:93:f8:6f:a3:2e:7c:7f:ca:94:eb: 92:5d:fe:8e:6c:7c:9c:9a:f6:1d:04:45:c4:ad:a2:88:26:24: 86:07:2b:65:bc:96:d8:12:4d:ee:37:8c:4c:9d:9d:a7:de:35: 13:d0:e7:b4:2d:8d:8b:2c:37:25:e8:48:f6:5a:b4:5f:0f:5c: b4:f1:62:90 #Client certificate - Is the same as the server certificate: (tester) [stack@undercloud-0 ~]$ echo | openssl s_client -showcerts -servername www.example.com -connect www.example.com:443 2>/dev/null | openssl x509 -inform pem -noout -text Certificate: Data: Version: 1 (0x0) Serial Number: 1 (0x1) Signature Algorithm: sha256WithRSAEncryption Issuer: C = US, ST = Denial, L = Springfield, O = Dis, CN = www.example.com Validity Not Before: Mar 5 11:40:31 2020 GMT Not After : Mar 3 11:40:31 2030 GMT Subject: C = US, ST = Denial, O = Dis, CN = www.example.com Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public-Key: (2048 bit) Modulus: 00:bf:b4:ed:0d:57:b6:2e:f4:02:5d:09:e7:10:fc: e1:6f:a4:75:70:9c:1d:9c:55:cc:70:17:67:1c:b2: 36:bc:45:d6:32:23:fd:64:fb:4b:97:ec:dd:36:60: 13:7e:aa:f0:28:ef:a5:b1:fb:df:6b:13:5f:36:ff: 31:2f:f0:79:ef:4a:b6:2e:1b:c9:aa:f3:1c:9e:02: 66:67:4b:5a:f5:27:9b:cc:0b:5b:30:38:61:ee:bc: 2d:ba:a5:65:c0:c9:68:da:bc:f9:ea:35:bf:b9:e3: cb:60:b6:a9:f8:8e:f8:6b:54:c5:06:d7:94:c4:5d: f8:89:fb:95:85:16:c3:c3:95:05:eb:f5:bd:a4:8b: eb:5f:e2:cb:7a:a9:27:a1:8e:3a:2c:ca:81:0b:60: 8f:54:3b:f1:f2:59:2a:69:b1:79:e4:8a:af:9b:10: ca:9b:9f:0e:40:cc:69:df:84:e3:ae:e8:7c:75:67: c3:6f:81:3b:36:c0:14:ea:6a:be:fd:26:4e:c9:71: a9:db:95:b7:e4:c6:e4:a7:9c:4c:7a:2f:ac:7f:2e: f5:ad:c1:db:e2:80:ac:92:e8:2a:2e:31:41:1f:82: e8:f4:b5:b2:b8:44:12:6e:6c:ce:af:82:2c:f3:9f: 3b:86:bd:ae:55:9b:08:a5:0c:2e:3f:d2:72:f4:e1: 7a:67 Exponent: 65537 (0x10001) Signature Algorithm: sha256WithRSAEncryption 7d:cf:bb:2c:61:e9:d7:10:48:a3:b1:77:8c:f8:b9:25:53:25: cf:71:5b:52:9c:b1:e2:2e:01:32:44:ac:f6:1b:d3:3d:1c:72: d6:4d:de:78:c3:12:db:63:6b:7b:79:ec:d9:da:47:d8:74:04: f6:a3:29:02:08:a5:a8:a4:b5:94:bc:23:c8:82:99:1b:9a:3d: ee:3e:79:c7:30:21:4e:4f:ea:70:ae:05:55:6b:7c:4e:23:ec: fe:dd:56:0f:8a:af:70:88:70:5f:42:d4:28:ca:26:0b:3a:cc: f7:48:3d:c1:e8:58:99:7b:00:c1:f7:71:06:ad:e2:9e:db:00: 8c:03:9c:56:02:6b:4a:6b:d1:a3:7d:b5:e6:99:e0:03:8b:46: 0b:ed:ee:ba:af:c1:0d:29:0e:eb:83:11:3b:f1:11:f6:bc:a5: db:6b:4a:f2:10:11:44:b9:01:b3:5f:c0:1e:7f:99:0b:08:4d: 7f:07:19:19:d6:fd:10:3c:93:f8:6f:a3:2e:7c:7f:ca:94:eb: 92:5d:fe:8e:6c:7c:9c:9a:f6:1d:04:45:c4:ad:a2:88:26:24: 86:07:2b:65:bc:96:d8:12:4d:ee:37:8c:4c:9d:9d:a7:de:35: 13:d0:e7:b4:2d:8d:8b:2c:37:25:e8:48:f6:5a:b4:5f:0f:5c: b4:f1:62:90 But when doing the same for www1.example.com, it is expected to get the SNI certificate (tls_secret2), but it seems like it is defaulting to the "default container certificate" (tls_secret1): #Server certificate: (tester) [stack@undercloud-0 ~]$ openssl x509 -inform pem -noout -text -in cert2/testca/testcert.pemCertificate: Data: Version: 1 (0x0) Serial Number: 1 (0x1) Signature Algorithm: sha256WithRSAEncryption Issuer: C = US, ST = Denial, L = Springfield, O = Dis, CN = www1.example.com Validity Not Before: Mar 9 09:01:42 2020 GMT Not After : Mar 7 09:01:42 2030 GMT Subject: C = US, ST = Denial, O = Dis, CN = www1.example.com Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public-Key: (2048 bit) Modulus: 00:c6:55:33:5f:bd:61:0f:9d:1d:29:88:dc:4c:a0: fa:a5:f7:4d:69:6b:65:d6:b0:9c:da:b8:63:4e:21: fd:e5:06:f0:cb:b6:60:c5:23:70:1f:ab:a2:ed:d8: eb:68:f9:36:ad:77:dc:ae:cd:06:a1:21:a6:ed:8b: 19:37:8c:54:fd:d7:e2:37:33:3d:88:54:b9:28:9b: ae:49:72:39:e4:31:de:bc:f2:3b:81:9b:d0:cf:ea: 6c:4a:e0:7e:fb:77:69:88:63:5a:9c:12:88:45:00: eb:f3:19:99:6b:2f:de:2b:e7:e6:e1:08:6c:c4:e1: 71:90:63:4c:40:0f:c2:09:85:85:5a:05:f8:84:e2: 52:1d:dd:0e:e0:c1:10:10:f1:fe:b5:5a:aa:30:73: e8:dc:98:6a:77:60:62:c0:1f:f0:2d:70:9a:0d:e9: 3e:52:98:b5:7e:0f:08:ab:86:b6:7c:1a:7e:44:ee: 32:22:de:8f:da:20:8a:63:d5:25:14:58:56:88:8d: 74:ea:2c:2b:02:9f:e1:e1:d5:5e:e9:46:91:39:03: fd:a9:8e:9a:b2:7c:cc:93:70:89:fd:eb:d5:7f:9e: 70:02:36:c7:1b:ab:be:21:ea:7a:86:3f:18:f9:01: cd:67:47:20:2a:13:64:60:b2:f8:d9:e1:34:43:c6: e9:6d Exponent: 65537 (0x10001) Signature Algorithm: sha256WithRSAEncryption 74:4d:b8:76:10:52:5b:55:bb:c7:e7:45:be:c6:3a:36:5f:8c: ac:47:10:b5:7f:24:55:0d:2c:2f:98:bc:0c:38:fb:b2:0e:3f: ca:a3:81:33:84:d0:c0:f4:f9:c7:b4:30:1f:9e:50:d6:91:12: 1c:d6:9b:1a:fd:7f:6e:1b:c1:de:d4:aa:53:ae:d8:96:7d:d6: 43:c6:21:fb:88:a9:b0:0e:6d:36:88:86:d7:3c:2b:11:31:ad: 64:e8:1d:81:1c:de:eb:a6:94:1d:e7:52:6b:f8:17:cf:84:93: 8c:c8:2b:40:3f:6e:cd:21:13:5c:c4:c3:2b:5b:9e:d4:a4:d2: ea:b5:64:6f:16:fe:75:76:91:b7:0b:42:86:be:2a:ca:80:96: 5e:db:15:98:3d:27:fa:7a:d6:7c:b9:2a:24:3e:f3:18:ba:d7: 35:d7:ef:1e:95:a0:41:e1:c5:3b:0e:7a:7e:c5:21:da:91:47: c2:0c:88:f8:a8:66:ce:d2:bf:36:79:cb:cf:34:e2:54:f0:4c: 4b:c5:c8:81:3b:d8:32:13:f0:2c:22:43:1d:ba:bc:15:81:a6: e3:fd:12:77:55:6c:0a:2a:dd:05:f9:e3:0a:71:d8:4c:12:33: 69:24:3d:6b:c1:9e:e3:bc:47:ae:d3:62:f3:3e:28:bf:40:ba: e1:28:c3:29 #Client certificate. As you can see, it is for www.example.com: (tester) [stack@undercloud-0 ~]$ echo | openssl s_client -showcerts -servername www1.example.com -connect www1.example.com:443 2>/dev/null | openssl x509 -inform pem -noout -text Certificate: Data: Version: 1 (0x0) Serial Number: 1 (0x1) Signature Algorithm: sha256WithRSAEncryption Issuer: C = US, ST = Denial, L = Springfield, O = Dis, CN = www.example.com Validity Not Before: Mar 5 11:40:31 2020 GMT Not After : Mar 3 11:40:31 2030 GMT Subject: C = US, ST = Denial, O = Dis, CN = www.example.com Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public-Key: (2048 bit) Modulus: 00:bf:b4:ed:0d:57:b6:2e:f4:02:5d:09:e7:10:fc: e1:6f:a4:75:70:9c:1d:9c:55:cc:70:17:67:1c:b2: 36:bc:45:d6:32:23:fd:64:fb:4b:97:ec:dd:36:60: 13:7e:aa:f0:28:ef:a5:b1:fb:df:6b:13:5f:36:ff: 31:2f:f0:79:ef:4a:b6:2e:1b:c9:aa:f3:1c:9e:02: 66:67:4b:5a:f5:27:9b:cc:0b:5b:30:38:61:ee:bc: 2d:ba:a5:65:c0:c9:68:da:bc:f9:ea:35:bf:b9:e3: cb:60:b6:a9:f8:8e:f8:6b:54:c5:06:d7:94:c4:5d: f8:89:fb:95:85:16:c3:c3:95:05:eb:f5:bd:a4:8b: eb:5f:e2:cb:7a:a9:27:a1:8e:3a:2c:ca:81:0b:60: 8f:54:3b:f1:f2:59:2a:69:b1:79:e4:8a:af:9b:10: ca:9b:9f:0e:40:cc:69:df:84:e3:ae:e8:7c:75:67: c3:6f:81:3b:36:c0:14:ea:6a:be:fd:26:4e:c9:71: a9:db:95:b7:e4:c6:e4:a7:9c:4c:7a:2f:ac:7f:2e: f5:ad:c1:db:e2:80:ac:92:e8:2a:2e:31:41:1f:82: e8:f4:b5:b2:b8:44:12:6e:6c:ce:af:82:2c:f3:9f: 3b:86:bd:ae:55:9b:08:a5:0c:2e:3f:d2:72:f4:e1: 7a:67 Exponent: 65537 (0x10001) Signature Algorithm: sha256WithRSAEncryption 7d:cf:bb:2c:61:e9:d7:10:48:a3:b1:77:8c:f8:b9:25:53:25: cf:71:5b:52:9c:b1:e2:2e:01:32:44:ac:f6:1b:d3:3d:1c:72: d6:4d:de:78:c3:12:db:63:6b:7b:79:ec:d9:da:47:d8:74:04: f6:a3:29:02:08:a5:a8:a4:b5:94:bc:23:c8:82:99:1b:9a:3d: ee:3e:79:c7:30:21:4e:4f:ea:70:ae:05:55:6b:7c:4e:23:ec: fe:dd:56:0f:8a:af:70:88:70:5f:42:d4:28:ca:26:0b:3a:cc: f7:48:3d:c1:e8:58:99:7b:00:c1:f7:71:06:ad:e2:9e:db:00: 8c:03:9c:56:02:6b:4a:6b:d1:a3:7d:b5:e6:99:e0:03:8b:46: 0b:ed:ee:ba:af:c1:0d:29:0e:eb:83:11:3b:f1:11:f6:bc:a5: db:6b:4a:f2:10:11:44:b9:01:b3:5f:c0:1e:7f:99:0b:08:4d: 7f:07:19:19:d6:fd:10:3c:93:f8:6f:a3:2e:7c:7f:ca:94:eb: 92:5d:fe:8e:6c:7c:9c:9a:f6:1d:04:45:c4:ad:a2:88:26:24: 86:07:2b:65:bc:96:d8:12:4d:ee:37:8c:4c:9d:9d:a7:de:35: 13:d0:e7:b4:2d:8d:8b:2c:37:25:e8:48:f6:5a:b4:5f:0f:5c: b4:f1:62:90 Error messages in octavia.log: Reminder: tls_secret1 is 4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac tls_secret2 is 3019549d-6be2-474a-a919-50f0ca955ff6 [root@controller-0 octavia]# cat octavia.log 2020-03-09 11:18:31.909 26 INFO barbicanclient.base [req-91b54236-bf18-4adf-b8a0-6d7b9842058b - b6cd9962adde4b4fbb4c63206f561f9e - default default] Calculated Containers uuid ref: containers/4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac 2020-03-09 11:18:33.358 26 ERROR barbicanclient.client [req-91b54236-bf18-4adf-b8a0-6d7b9842058b - b6cd9962adde4b4fbb4c63206f561f9e - default default] 4xx Client error: Not Found: Not Found. Sorry but your container is in another castle. 2020-03-09 11:18:33.359 26 INFO octavia.certificates.manager.barbican [req-91b54236-bf18-4adf-b8a0-6d7b9842058b - b6cd9962adde4b4fbb4c63206f561f9e - default default] Loading certificate secret http://10.0.0.129:9311/v1/secrets/4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac from Barbican. 2020-03-09 11:18:33.360 26 INFO barbicanclient.base [req-91b54236-bf18-4adf-b8a0-6d7b9842058b - b6cd9962adde4b4fbb4c63206f561f9e - default default] Calculated Secrets uuid ref: secrets/4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac 2020-03-09 11:18:33.392 26 INFO barbicanclient.base [req-91b54236-bf18-4adf-b8a0-6d7b9842058b - b6cd9962adde4b4fbb4c63206f561f9e - default default] Calculated Secrets uuid ref: secrets/4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac 2020-03-09 11:18:35.139 26 INFO barbicanclient.base [req-91b54236-bf18-4adf-b8a0-6d7b9842058b - b6cd9962adde4b4fbb4c63206f561f9e - default default] Calculated Containers uuid ref: containers/3019549d-6be2-474a-a919-50f0ca955ff6 2020-03-09 11:18:35.162 26 ERROR barbicanclient.client [req-91b54236-bf18-4adf-b8a0-6d7b9842058b - b6cd9962adde4b4fbb4c63206f561f9e - default default] 4xx Client error: Not Found: Not Found. Sorry but your container is in another castle. 2020-03-09 11:18:35.163 26 INFO octavia.certificates.manager.barbican [req-91b54236-bf18-4adf-b8a0-6d7b9842058b - b6cd9962adde4b4fbb4c63206f561f9e - default default] Loading certificate secret http://10.0.0.129:9311/v1/secrets/3019549d-6be2-474a-a919-50f0ca955ff6 from Barbican. 2020-03-09 11:18:35.163 26 INFO barbicanclient.base [req-91b54236-bf18-4adf-b8a0-6d7b9842058b - b6cd9962adde4b4fbb4c63206f561f9e - default default] Calculated Secrets uuid ref: secrets/3019549d-6be2-474a-a919-50f0ca955ff6 2020-03-09 11:18:35.210 26 INFO barbicanclient.base [req-91b54236-bf18-4adf-b8a0-6d7b9842058b - b6cd9962adde4b4fbb4c63206f561f9e - default default] Calculated Secrets uuid ref: secrets/3019549d-6be2-474a-a919-50f0ca955ff6 2020-03-09 11:18:35.283 26 INFO octavia.certificates.manager.barbican [req-0b508be5-cad4-4bef-a7b1-87e127f929a9 - - - - -] Loading certificate secret http://10.0.0.129:9311/v1/secrets/4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac from Barbican. 2020-03-09 11:18:35.283 26 INFO barbicanclient.base [req-0b508be5-cad4-4bef-a7b1-87e127f929a9 - - - - -] Calculated Secrets uuid ref: secrets/4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac 2020-03-09 11:18:35.311 26 INFO barbicanclient.base [req-0b508be5-cad4-4bef-a7b1-87e127f929a9 - - - - -] Calculated Secrets uuid ref: secrets/4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac 2020-03-09 11:18:35.389 26 INFO octavia.certificates.manager.barbican [req-0b508be5-cad4-4bef-a7b1-87e127f929a9 - - - - -] Loading certificate secret http://10.0.0.129:9311/v1/secrets/3019549d-6be2-474a-a919-50f0ca955ff6 from Barbican. 2020-03-09 11:18:35.390 26 INFO barbicanclient.base [req-0b508be5-cad4-4bef-a7b1-87e127f929a9 - - - - -] Calculated Secrets uuid ref: secrets/3019549d-6be2-474a-a919-50f0ca955ff6 2020-03-09 11:18:35.440 26 INFO barbicanclient.base [req-0b508be5-cad4-4bef-a7b1-87e127f929a9 - - - - -] Calculated Secrets uuid ref: secrets/3019549d-6be2-474a-a919-50f0ca955ff6 2020-03-09 11:18:35.520 26 INFO octavia.certificates.manager.barbican [req-29c8588d-5227-4ae6-b097-d9f836d9d792 - b6cd9962adde4b4fbb4c63206f561f9e - - -] Loading certificate secret http://10.0.0.129:9311/v1/secrets/4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac from Barbican. 2020-03-09 11:18:35.521 26 INFO barbicanclient.base [req-29c8588d-5227-4ae6-b097-d9f836d9d792 - b6cd9962adde4b4fbb4c63206f561f9e - - -] Calculated Secrets uuid ref: secrets/4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac 2020-03-09 11:18:35.552 26 INFO barbicanclient.base [req-29c8588d-5227-4ae6-b097-d9f836d9d792 - b6cd9962adde4b4fbb4c63206f561f9e - - -] Calculated Secrets uuid ref: secrets/4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac 2020-03-09 11:18:35.591 26 INFO octavia.api.v2.controllers.listener [req-29c8588d-5227-4ae6-b097-d9f836d9d792 - b6cd9962adde4b4fbb4c63206f561f9e - - -] Sending update Listener 03365412-ad59-4333-b28e-bb46f9110ac9 to provider amphora
Also happening when updating an already functioning TLS terminated listener with SNI: [2020-03-09 12:22:19] (tester) [stack@undercloud-0 ~]$ openstack loadbalancer listener show listener1 +-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | True | | connection_limit | -1 | | created_at | 2020-03-09T15:36:56 | | default_pool_id | 091267c5-7f6a-4d64-bdc9-97d4a6dec4f0 | | default_tls_container_ref | http://10.0.0.129:9311/v1/secrets/4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac | | description | | | id | 255f8d78-a427-40bd-8ee6-b070309bd44d | | insert_headers | None | | l7policies | | | loadbalancers | 252acde2-1fa0-407f-bf83-86844f3880d1 | | name | listener1 | | operating_status | ONLINE | | project_id | b6cd9962adde4b4fbb4c63206f561f9e | | protocol | TERMINATED_HTTPS | | protocol_port | 443 | | provisioning_status | ACTIVE | | sni_container_refs | ['http://10.0.0.129:9311/v1/secrets/3019549d-6be2-474a-a919-50f0ca955ff6', 'http://10.0.0.129:9311/v1/secrets/4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac'] | | timeout_client_data | 50000 | | timeout_member_connect | 5000 | | timeout_member_data | 50000 | | timeout_tcp_inspect | 0 | | updated_at | 2020-03-09T15:45:11 | | client_ca_tls_container_ref | None | | client_authentication | NONE | | client_crl_container_ref | None | | allowed_cidrs | None | +-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ [2020-03-09 12:22:43] (tester) [stack@undercloud-0 ~]$ [2020-03-09 12:23:04] (tester) [stack@undercloud-0 ~]$ [2020-03-09 12:23:04] (tester) [stack@undercloud-0 ~]$ [2020-03-09 12:23:04] (tester) [stack@undercloud-0 ~]$ [2020-03-09 12:23:04] (tester) [stack@undercloud-0 ~]$ [2020-03-09 12:23:05] (tester) [stack@undercloud-0 ~]$ [2020-03-09 12:23:05] (tester) [stack@undercloud-0 ~]$ openstack secret list +------------------------------------------------------------------------+-------------+---------------------------+--------+-----------------------------------------+-----------+------------+-------------+------+------------+ | Secret href | Name | Created | Status | Content types | Algorithm | Bit length | Secret type | Mode | Expiration | +------------------------------------------------------------------------+-------------+---------------------------+--------+-----------------------------------------+-----------+------------+-------------+------+------------+ | http://10.0.0.129:9311/v1/secrets/3019549d-6be2-474a-a919-50f0ca955ff6 | tls_secret2 | 2020-03-09T09:54:30+00:00 | ACTIVE | {'default': 'application/octet-stream'} | aes | 256 | opaque | cbc | None | | http://10.0.0.129:9311/v1/secrets/4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac | tls_secret1 | 2020-03-08T09:24:28+00:00 | ACTIVE | {'default': 'application/octet-stream'} | aes | 256 | opaque | cbc | None | | http://10.0.0.129:9311/v1/secrets/972f6889-e4df-4adc-9e80-a0483c521916 | tls_secret3 | 2020-03-09T16:22:01+00:00 | ACTIVE | {'default': 'application/octet-stream'} | aes | 256 | opaque | cbc | None | +------------------------------------------------------------------------+-------------+---------------------------+--------+-----------------------------------------+-----------+------------+-------------+------+------------+ [2020-03-09 12:23:12] (tester) [stack@undercloud-0 ~]$ openstack loadbalancer listener set listener1 --sni-container-refs $(openstack secret list | awk '/ tls_secret1 / {print $2}') $(openstack secret list | awk '/ tls_secret3 / {print $2}') [2020-03-09 12:27:58] (tester) [stack@undercloud-0 ~]$ [2020-03-09 12:27:58] (tester) [stack@undercloud-0 ~]$ [2020-03-09 12:28:03] (tester) [stack@undercloud-0 ~]$ openstack loadbalancer listener show listener1 +-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | True | | connection_limit | -1 | | created_at | 2020-03-09T15:36:56 | | default_pool_id | 091267c5-7f6a-4d64-bdc9-97d4a6dec4f0 | | default_tls_container_ref | http://10.0.0.129:9311/v1/secrets/4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac | | description | | | id | 255f8d78-a427-40bd-8ee6-b070309bd44d | | insert_headers | None | | l7policies | | | loadbalancers | 252acde2-1fa0-407f-bf83-86844f3880d1 | | name | listener1 | | operating_status | ONLINE | | project_id | b6cd9962adde4b4fbb4c63206f561f9e | | protocol | TERMINATED_HTTPS | | protocol_port | 443 | | provisioning_status | ACTIVE | | sni_container_refs | ['http://10.0.0.129:9311/v1/secrets/3019549d-6be2-474a-a919-50f0ca955ff6', 'http://10.0.0.129:9311/v1/secrets/4727685b-4c88-4dc6-bb8e-d8cb30f3c8ac'] | | timeout_client_data | 50000 | | timeout_member_connect | 5000 | | timeout_member_data | 50000 | | timeout_tcp_inspect | 0 | | updated_at | 2020-03-09T16:28:00 | | client_ca_tls_container_ref | None | | client_authentication | NONE | | client_crl_container_ref | None | | allowed_cidrs | None | +-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ From logs: #Grep tls_secret3 id: [root@controller-1 octavia]# cat octavia.log | grep a0483c521916 2020-03-09 16:27:55.029 27 INFO barbicanclient.base [req-71d380f8-00c1-4a2b-a93a-2c915cd67c1d - b6cd9962adde4b4fbb4c63206f561f9e - default default] Calculated Containers uuid ref: containers/972f6889-e4df-4adc-9e80-a0483c521916 2020-03-09 16:27:55.046 27 INFO octavia.certificates.manager.barbican [req-71d380f8-00c1-4a2b-a93a-2c915cd67c1d - b6cd9962adde4b4fbb4c63206f561f9e - default default] Loading certificate secret http://10.0.0.129:9311/v1/secrets/972f6889-e4df-4adc-9e80-a0483c521916 from Barbican. 2020-03-09 16:27:55.046 27 INFO barbicanclient.base [req-71d380f8-00c1-4a2b-a93a-2c915cd67c1d - b6cd9962adde4b4fbb4c63206f561f9e - default default] Calculated Secrets uuid ref: secrets/972f6889-e4df-4adc-9e80-a0483c521916 2020-03-09 16:27:55.075 27 INFO barbicanclient.base [req-71d380f8-00c1-4a2b-a93a-2c915cd67c1d - b6cd9962adde4b4fbb4c63206f561f9e - default default] Calculated Secrets uuid ref: secrets/972f6889-e4df-4adc-9e80-a0483c521916 2020-03-09 16:27:55.212 27 INFO octavia.certificates.manager.barbican [req-628ced30-6b4b-431c-87ef-d6e9ea58e4ed - - - - -] Loading certificate secret http://10.0.0.129:9311/v1/secrets/972f6889-e4df-4adc-9e80-a0483c521916 from Barbican. 2020-03-09 16:27:55.214 27 INFO barbicanclient.base [req-628ced30-6b4b-431c-87ef-d6e9ea58e4ed - - - - -] Calculated Secrets uuid ref: secrets/972f6889-e4df-4adc-9e80-a0483c521916 2020-03-09 16:27:55.245 27 INFO barbicanclient.base [req-628ced30-6b4b-431c-87ef-d6e9ea58e4ed - - - - -] Calculated Secrets uuid ref: secrets/972f6889-e4df-4adc-9e80-a0483c521916 #Grep listenerID: [root@controller-0 octavia]# cat worker.log | grep 255f8d78-a427-40bd-8ee6-b070309bd44d 2020-03-09 16:27:55.507 23 INFO octavia.controller.queue.v1.endpoints [-] Updating listener '255f8d78-a427-40bd-8ee6-b070309bd44d'... [root@controller-1 octavia]# cat octavia.log | grep 255f8d78-a427-40bd-8ee6-b070309bd44d 2020-03-09 16:27:55.476 27 INFO octavia.api.v2.controllers.listener [req-079831b3-e3ab-4b62-a123-2a059357a9fe - b6cd9962adde4b4fbb4c63206f561f9e - - -] Sending update Listener 255f8d78-a427-40bd-8ee6-b070309bd44d to provider amphora
I was able to reproduce this issue in OSP 16.0 z1 and upstream master branch (devstack).
Tested in: 16.1 -p RHOS-16.1-RHEL-8-20200930.n.0 First test: 1) Created environment:1 load balancer, 1 terminated https listener, 1 pool with 2 member servers, 1 health monitor. The terminated https listener points to a barbican container called tls_secret1. The clients access the VIP with the address www.example.com (added in /etc/hosts file) 2)Another barbican secret was added, called tls_secret2, for the address www1.example.com I also added www1.example.com to point to the VIP address in /etc/hosts file. Added tls_secret2 as a sni_container for the listener: [2020-10-05 12:42:51] (tester) [stack@undercloud-0 ~]$ openstack loadbalancer listener set listener1 --sni-container-refs $(openstack secret list | awk '/ tls_secret1 / {print $2}') $(openstack secret list | awk '/ tls_secret13 / {print $2}') 3) Tested traffic to both urls: [2020-10-05 12:43:22] (tester) [stack@undercloud-0 ~]$ req='curl -k https://www.example.com'; for i in {1..10}; do $req; echo; donesrv-server2-jcxedzurzrw6 srv-server1-dyistuzpwrz4 srv-server2-jcxedzurzrw6 srv-server1-dyistuzpwrz4 srv-server2-jcxedzurzrw6 srv-server1-dyistuzpwrz4 srv-server2-jcxedzurzrw6 srv-server1-dyistuzpwrz4 srv-server2-jcxedzurzrw6 srv-server1-dyistuzpwrz4 [2020-10-05 12:43:26] (tester) [stack@undercloud-0 ~]$ req='curl -k https://www1.example.com'; for i in {1..10}; do $req; echo; done srv-server2-jcxedzurzrw6 srv-server1-dyistuzpwrz4 srv-server2-jcxedzurzrw6 srv-server1-dyistuzpwrz4 srv-server2-jcxedzurzrw6 srv-server1-dyistuzpwrz4 srv-server2-jcxedzurzrw6 srv-server1-dyistuzpwrz4 srv-server2-jcxedzurzrw6 srv-server1-dyistuzpwrz4 4) "Listener Show" command now shows the new barbican secret under "sni_container_refs": [2020-10-05 12:43:32] (tester) [stack@undercloud-0 ~]$ openstack loadbalancer listener show listener1 +-----------------------------+----------------------------------------------------------------------------+ | Field | Value | +-----------------------------+----------------------------------------------------------------------------+ | admin_state_up | True | | connection_limit | -1 | | created_at | 2020-10-05T16:04:23 | | default_pool_id | 9b2aeb31-2cb4-41de-9724-e7728b2b2071 | | default_tls_container_ref | http://10.0.0.131:9311/v1/secrets/66133473-e42a-4313-8ec4-a6760a662bcf | | description | | | id | 6af78ef1-0eef-463b-8dee-dbd7d3d3185f | | insert_headers | None | | l7policies | | | loadbalancers | b27b318a-0262-4607-a92c-76dc26ce4411 | | name | listener1 | | operating_status | ONLINE | | project_id | c6dcb43ccaf34ca7a77d6c35ae5e3230 | | protocol | TERMINATED_HTTPS | | protocol_port | 443 | | provisioning_status | ACTIVE | | sni_container_refs | ['http://10.0.0.131:9311/v1/secrets/dc914afb-ec05-40e9-906c-06be14445114'] | | timeout_client_data | 50000 | | timeout_member_connect | 5000 | | timeout_member_data | 50000 | | timeout_tcp_inspect | 0 | | updated_at | 2020-10-05T16:43:20 | | client_ca_tls_container_ref | None | | client_authentication | NONE | | client_crl_container_ref | None | | allowed_cidrs | None | +-----------------------------+----------------------------------------------------------------------------+ 5) Also, the certificates are the right ones: [2020-10-05 12:44:46] (tester) [stack@undercloud-0 ~]$ openssl x509 -inform pem -noout -text -in testca/testcert.pem Certificate: Data: Version: 1 (0x0) Serial Number: 1 (0x1) Signature Algorithm: sha256WithRSAEncryption Issuer: C = US, ST = Denial, L = Springfield, O = Dis, CN = www.example.com Validity Not Before: Oct 5 15:54:32 2020 GMT Not After : Oct 3 15:54:32 2030 GMT Subject: C = US, ST = Denial, O = Dis, CN = www.example.com Subject Public Key Info: [...] [2020-10-05 12:44:57] (tester) [stack@undercloud-0 ~]$ echo | openssl s_client -showcerts -servername www.example.com -connect www.example.com:443 2>/dev/null | openssl x509 -inform pem -noout -text Certificate: Data: Version: 1 (0x0) Serial Number: 1 (0x1) Signature Algorithm: sha256WithRSAEncryption Issuer: C = US, ST = Denial, L = Springfield, O = Dis, CN = www.example.com Validity Not Before: Oct 5 15:54:32 2020 GMT Not After : Oct 3 15:54:32 2030 GMT Subject: C = US, ST = Denial, O = Dis, CN = www.example.com Subject Public Key Info: [...] [2020-10-05 12:45:14] (tester) [stack@undercloud-0 ~]$ [2020-10-05 12:46:07] (tester) [stack@undercloud-0 ~]$ openssl x509 -inform pem -noout -text -in newcert2/testca/testcert.pem Certificate: Data: Version: 1 (0x0) Serial Number: 1 (0x1) Signature Algorithm: sha256WithRSAEncryption Issuer: C = US, ST = Denial, L = Springfield, O = Dis, CN = www1.example.com Validity Not Before: Oct 5 16:35:39 2020 GMT Not After : Oct 3 16:35:39 2030 GMT Subject: C = US, ST = Denial, O = Dis, CN = www1.example.com Subject Public Key Info: [...] [2020-10-05 12:46:13] (tester) [stack@undercloud-0 ~]$ echo | openssl s_client -showcerts -servername www1.example.com -connect www.example.com:443 2>/dev/null | openssl x509 -inform pem -noout -text Certificate: Data: Version: 1 (0x0) Serial Number: 1 (0x1) Signature Algorithm: sha256WithRSAEncryption Issuer: C = US, ST = Denial, L = Springfield, O = Dis, CN = www1.example.com Validity Not Before: Oct 5 16:35:39 2020 GMT Not After : Oct 3 16:35:39 2030 GMT Subject: C = US, ST = Denial, O = Dis, CN = www1.example.com Subject Public Key Info: [...] Second test: 1) Updating a sni container to a new one (from tls_secret13 to tls_secret14): [2020-10-05 12:56:25] (tester) [stack@undercloud-0 testca]$ openstack loadbalancer listener set listener1 --sni-container-refs $(openstack secret list | awk '/ tls_secret1 / {print $2}') $(openstack secret list | awk '/ tls_secret14 / {print $2}') 2) Listener is updated as expected: [2020-10-05 12:57:15] (tester) [stack@undercloud-0 testca]$ openstack loadbalancer listener show listener1 +-----------------------------+----------------------------------------------------------------------------+ | Field | Value | +-----------------------------+----------------------------------------------------------------------------+ | admin_state_up | True | | connection_limit | -1 | | created_at | 2020-10-05T16:04:23 | | default_pool_id | 9b2aeb31-2cb4-41de-9724-e7728b2b2071 | | default_tls_container_ref | http://10.0.0.131:9311/v1/secrets/66133473-e42a-4313-8ec4-a6760a662bcf | | description | | | id | 6af78ef1-0eef-463b-8dee-dbd7d3d3185f | | insert_headers | None | | l7policies | | | loadbalancers | b27b318a-0262-4607-a92c-76dc26ce4411 | | name | listener1 | | operating_status | ONLINE | | project_id | c6dcb43ccaf34ca7a77d6c35ae5e3230 | | protocol | TERMINATED_HTTPS | | protocol_port | 443 | | provisioning_status | ACTIVE | | sni_container_refs | ['http://10.0.0.131:9311/v1/secrets/22223562-a8d9-46bd-b03e-ff476aaefb58'] | | timeout_client_data | 50000 | | timeout_member_connect | 5000 | | timeout_member_data | 50000 | | timeout_tcp_inspect | 0 | | updated_at | 2020-10-05T16:57:05 | | client_ca_tls_container_ref | None | | client_authentication | NONE | | client_crl_container_ref | None | | allowed_cidrs | None | +-----------------------------+----------------------------------------------------------------------------+ Moving the bug to VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:4284