Bug 1808795
| Summary: | Octavia HTTP listener in ERROR state after deleting a Barbican secret and triggering a LB failover | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Bruna Bonguardo <bbonguar> |
| Component: | openstack-octavia | Assignee: | Michael Johnson <michjohn> |
| Status: | CLOSED ERRATA | QA Contact: | Bruna Bonguardo <bbonguar> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 16.0 (Train) | CC: | cgoncalves, ihrachys, lpeer, majopela, mgarciac, scohen |
| Target Milestone: | z2 | Keywords: | Triaged |
| Target Release: | 16.0 (Train on RHEL 8.1) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-octavia-5.0.2-0.20200226143008.0c1cb41.el8ost | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-05-14 12:08:10 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This looks like a bug that has been fixed and released a while ago: https://bugzilla.redhat.com/show_bug.cgi?id=1759476 Could you please attach logs? Thank you. I opened this bug because I dont think the POOL should be on operating_status of "OFFLINE", because it impacts the other HTTP listener that is attached to this pool, and it has nothing to do with the Terminated-HTTPS listener.
Before failover:
[2020-03-01 10:25:08] (tester) [stack@undercloud-0 ~]$ openstack loadbalancer status show lb1
{
"loadbalancer": {
"id": "91d4281d-7526-48ed-9262-3c91392bceed",
"name": "lb1",
"operating_status": "ONLINE",
"provisioning_status": "ACTIVE",
"listeners": [
{
"id": "b7130107-b2a6-4f05-848c-1f7b7bc81319",
"name": "listener_http",
"operating_status": "ONLINE",
"provisioning_status": "ACTIVE",
"pools": [
{
"id": "3bc02dc2-1840-4936-9126-ad0c5dea3134",
"name": "pool1",
"provisioning_status": "ACTIVE",
"operating_status": "ONLINE",
"health_monitor": {
"id": "0d537542-bdbc-4f4c-9961-ecfce1d72c10",
"name": "",
"type": "HTTP",
"provisioning_status": "ACTIVE",
"operating_status": "ONLINE"
},
"members": [
{
"id": "f809a7cc-8ce2-46c9-af71-4919251d85d3",
"name": "",
"operating_status": "ONLINE",
"provisioning_status": "ACTIVE",
"address": "10.0.1.199",
"protocol_port": 8080
},
{
"id": "7719c14e-ec5c-4696-9ab7-f7ec02d8340c",
"name": "",
"operating_status": "ONLINE",
"provisioning_status": "ACTIVE",
"address": "10.0.1.253",
"protocol_port": 8080
}
]
}
]
},
{
"id": "a37a4080-2c6a-4d15-b45b-b79cbc2a3113",
"name": "listener1",
"operating_status": "ONLINE",
"provisioning_status": "ACTIVE",
"pools": [
{
"id": "3bc02dc2-1840-4936-9126-ad0c5dea3134",
"name": "pool1",
"provisioning_status": "ACTIVE",
"operating_status": "ONLINE",
"health_monitor": {
"id": "0d537542-bdbc-4f4c-9961-ecfce1d72c10",
"name": "",
"type": "HTTP",
"provisioning_status": "ACTIVE",
"operating_status": "ONLINE"
},
"members": [
{
"id": "f809a7cc-8ce2-46c9-af71-4919251d85d3",
"name": "",
"operating_status": "ONLINE",
"provisioning_status": "ACTIVE",
"address": "10.0.1.199",
"protocol_port": 8080
},
{
"id": "7719c14e-ec5c-4696-9ab7-f7ec02d8340c",
"name": "",
"operating_status": "ONLINE",
"provisioning_status": "ACTIVE",
"address": "10.0.1.253",
"protocol_port": 8080
}
]
}
]
}
]
}
}
[2020-03-01 10:28:05] (tester) [stack@undercloud-0 ~]$ openstack secret list
+------------------------------------------------------------------------+-------------+---------------------------+--------+-----------------------------------------+-----------+------------+-------------+------+------------+
| Secret href | Name | Created | Status | Content types | Algorithm | Bit length | Secret type | Mode | Expiration |
+------------------------------------------------------------------------+-------------+---------------------------+--------+-----------------------------------------+-----------+------------+-------------+------+------------+
| http://10.0.0.115:9311/v1/secrets/448b4fdd-4df5-4c29-bb0b-e02277c7eb01 | tls_secret2 | 2020-02-24T13:01:16+00:00 | ACTIVE | {'default': 'application/octet-stream'} | aes | 256 | opaque | cbc | None |
| http://10.0.0.115:9311/v1/secrets/61f261ad-d577-48eb-8f0e-069664524a08 | tls_secret3 | 2020-03-01T12:17:03+00:00 | ACTIVE | {'default': 'application/octet-stream'} | aes | 256 | opaque | cbc | None |
+------------------------------------------------------------------------+-------------+---------------------------+--------+-----------------------------------------+-----------+------------+-------------+------+------------+
After deletion of tls_secret3 from barbican and "openstack loadbalancer failover lb1":
Attaching SOSReport: http://rhos-release.virt.bos.redhat.com/log/bz1808795
Do you need anything else?
We have confirmed that the fix for this issue (and others) is missing from the build you are using. We will trigger an update in the build system to get it to pull in the patches. Verification will be ready 14/05 - I have been having difficulties with the setup [stack@undercloud-0 ~]$ cat /var/lib/rhos-release/latest-installed 16-trunk -p RHOS_TRUNK-16.0-RHEL-8-20200427.n.0 > Steps to Reproduce: > 1. Deploy OSP16 with one LB, one Terminated-HTTPS(443) listener, one HTTP(80) listener, both connected to a HTTP pool with 2 members servers and a HTTP health monitor. (tester) [stack@undercloud-0 ~]$ openstack loadbalancer status show lb1 { "loadbalancer": { "id": "3ccd33a1-1256-4610-aa66-87c9bc2f77ea", "name": "lb1", "operating_status": "ONLINE", "provisioning_status": "ACTIVE", "listeners": [ { "id": "13936c6f-4271-4947-b454-c64937092cf6", "name": "listener_http", "operating_status": "ONLINE", "provisioning_status": "ACTIVE", "pools": [ { "id": "bb189f32-506b-426b-90f6-b718f66d6a94", "name": "pool1", "provisioning_status": "ACTIVE", "operating_status": "ONLINE", "health_monitor": { "id": "64bcd3a2-cf8b-4b9b-b17c-c8901e6da4ca", "name": "hm", "type": "HTTP", "provisioning_status": "ACTIVE", "operating_status": "ONLINE" }, "members": [ { "id": "37955166-c7f1-4852-9ce8-3460d7575dee", "name": "", "operating_status": "ONLINE", "provisioning_status": "ACTIVE", "address": "10.0.1.203", "protocol_port": 80 }, { "id": "f923b96b-e07a-40f9-8099-f0a651e88816", "name": "", "operating_status": "ONLINE", "provisioning_status": "ACTIVE", "address": "10.0.1.84", "protocol_port": 80 } ] } ] }, { "id": "420893b7-ce06-4e96-a8e3-b506f5b94bdb", "name": "listener_https", "operating_status": "ONLINE", "provisioning_status": "ACTIVE", "pools": [ { "id": "bb189f32-506b-426b-90f6-b718f66d6a94", "name": "pool1", "provisioning_status": "ACTIVE", "operating_status": "ONLINE", "health_monitor": { "id": "64bcd3a2-cf8b-4b9b-b17c-c8901e6da4ca", "name": "hm", "type": "HTTP", "provisioning_status": "ACTIVE", "operating_status": "ONLINE" }, "members": [ { "id": "37955166-c7f1-4852-9ce8-3460d7575dee", "name": "", "operating_status": "ONLINE", "provisioning_status": "ACTIVE", "address": "10.0.1.203", "protocol_port": 80 }, { "id": "f923b96b-e07a-40f9-8099-f0a651e88816", "name": "", "operating_status": "ONLINE", "provisioning_status": "ACTIVE", "address": "10.0.1.84", "protocol_port": 80 } > 2. Delete the barbican secret connected to the Terminated-HTTPS(443) listener. [2020-05-13 08:53:18] (tester) [stack@undercloud-0 ~]$ openstack secret list +------------------------------------------------------------------------+-------------+---------------------------+--------+-----------------------------------------+-----------+------------+-------------+------+------------+ | Secret href | Name | Created | Status | Content types | Algorithm | Bit length | Secret type | Mode | Expiration | +------------------------------------------------------------------------+-------------+---------------------------+--------+-----------------------------------------+-----------+------------+-------------+------+------------+ | http://10.0.0.135:9311/v1/secrets/144c37cf-7aa5-4a91-948a-b7818ca2fd75 | tls_secret1 | 2020-05-13T11:39:15+00:00 | ACTIVE | {'default': 'application/octet-stream'} | aes | 256 | opaque | cbc | None | +------------------------------------------------------------------------+-------------+---------------------------+--------+-----------------------------------------+-----------+------------+-------------+------+------------+ [2020-05-13 08:55:44] (tester) [stack@undercloud-0 ~]$ openstack secret delete http://10.0.0.135:9311/v1/secrets/144c37cf-7aa5-4a91-948a-b7818ca2fd75 [2020-05-13 08:55:53] (tester) [stack@undercloud-0 ~]$ openstack secret list [2020-05-13 08:55:58] (tester) [stack@undercloud-0 ~]$ > 3. Trigger a failover to the LB [2020-05-13 08:56:38] (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer failover lb1 [2020-05-13 08:56:42] (overcloud) [stack@undercloud-0 ~]$ Checking the status of the LB tree: [2020-05-13 08:56:56] (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer status show lb1 { "loadbalancer": { "id": "3ccd33a1-1256-4610-aa66-87c9bc2f77ea", "name": "lb1", "operating_status": "ONLINE", "provisioning_status": "ACTIVE", "listeners": [ { "id": "13936c6f-4271-4947-b454-c64937092cf6", "name": "listener_http", "operating_status": "ONLINE", "provisioning_status": "ACTIVE", "pools": [ { "id": "bb189f32-506b-426b-90f6-b718f66d6a94", "name": "pool1", "provisioning_status": "ACTIVE", "operating_status": "ONLINE", "health_monitor": { "id": "64bcd3a2-cf8b-4b9b-b17c-c8901e6da4ca", "name": "hm", "type": "HTTP", "provisioning_status": "ACTIVE", "operating_status": "ONLINE" }, "members": [ { "id": "37955166-c7f1-4852-9ce8-3460d7575dee", "name": "", "operating_status": "ONLINE", "provisioning_status": "ACTIVE", "address": "10.0.1.203", "protocol_port": 80 }, { "id": "f923b96b-e07a-40f9-8099-f0a651e88816", "name": "", "operating_status": "ONLINE", "provisioning_status": "ACTIVE", "address": "10.0.1.84", "protocol_port": 80 } ] } ] }, { "id": "420893b7-ce06-4e96-a8e3-b506f5b94bdb", "name": "listener_https", "operating_status": "ERROR", "provisioning_status": "ERROR", "pools": [ { "id": "bb189f32-506b-426b-90f6-b718f66d6a94", "name": "pool1", "provisioning_status": "ACTIVE", "operating_status": "ONLINE", "health_monitor": { "id": "64bcd3a2-cf8b-4b9b-b17c-c8901e6da4ca", "name": "hm", "type": "HTTP", "provisioning_status": "ACTIVE", "operating_status": "ONLINE" }, "members": [ { "id": "37955166-c7f1-4852-9ce8-3460d7575dee", "name": "", "operating_status": "ONLINE", "provisioning_status": "ACTIVE", "address": "10.0.1.203", "protocol_port": 80 }, { "id": "f923b96b-e07a-40f9-8099-f0a651e88816", "name": "", "operating_status": "ONLINE", "provisioning_status": "ACTIVE", "address": "10.0.1.84", "protocol_port": 80 Terminated_HTTPS listener is in ERROR - as expected. HTTP listener is working as expected. Moving the bug to VERIFIED. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2164 |
Description of problem: Environment: One Load Balancer, Two listeners: one Terminated-HTTPS(443) connected to a Barbican secret, and the other listener is HTTP(80). Both listeners connected to one HTTP pool (2 member servers). Both listeners working as expected. When deleting the Barbican secret and performing a Failover to the Load Balancer, both listeners enter a ERROR state. The documentation says the LB should enter an ERROR state in this case, but IMO the HTTP listener should continue working as usual, as the Barbican secret is supposed to impact only the Terminated-HTTPS listener. Version-Release number of selected component (if applicable): OSP16 $ cat /var/lib/rhos-release/latest-installed 16-trunk -p RHOS_TRUNK-16.0-RHEL-8-20200213.n.1 How reproducible: 100% Steps to Reproduce: 1. Deploy OSP16 with one LB, one Terminated-HTTPS(443) listener, one HTTP(80) listener, both connected to a HTTP pool with 2 members servers and a HTTP health monitor. 2. Delete the barbican secret connected to the Terminated-HTTPS(443) listener. 3. Trigger a failover to the LB Actual results: Load balancer status: (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer status show lb1 { "loadbalancer": { "id": "1aa8fe32-f99d-49f3-9668-ea7962ab8d58", "name": "lb1", "operating_status": "ONLINE", "provisioning_status": "ERROR", "listeners": [ { "id": "50a38a2d-bcff-437a-91c1-e5ffc26d6637", "name": "listener_http", "operating_status": "OFFLINE", "provisioning_status": "ERROR", "pools": [ { "id": "8cb8cd80-af47-40b2-824b-898e5732d5b0", "name": "pool1", "provisioning_status": "ACTIVE", "operating_status": "OFFLINE", "health_monitor": { "id": "5511b3fd-3b09-4a17-8351-1f098e7ae66a", "name": "", "type": "HTTP", "provisioning_status": "ACTIVE", "operating_status": "ONLINE" }, "members": [ { "id": "6633ce90-34bf-4dc0-8bd3-22d28600263b", "name": "", "operating_status": "ONLINE", "provisioning_status": "ACTIVE", "address": "10.0.1.199", "protocol_port": 8080 }, { "id": "72bfcc95-c901-41b3-95cd-5fba1e6086df", "name": "", "operating_status": "ONLINE", "provisioning_status": "ACTIVE", "address": "10.0.1.253", "protocol_port": 8080 } ] } ] }, { "id": "ab8ecec6-bf01-41f1-b7e2-a7bb2bee605b", "name": "listener1", "operating_status": "OFFLINE", "provisioning_status": "ERROR", "pools": [ { "id": "8cb8cd80-af47-40b2-824b-898e5732d5b0", "name": "pool1", "provisioning_status": "ACTIVE", "operating_status": "OFFLINE", "health_monitor": { "id": "5511b3fd-3b09-4a17-8351-1f098e7ae66a", "name": "", "type": "HTTP", "provisioning_status": "ACTIVE", "operating_status": "ONLINE" }, "members": [ { "id": "6633ce90-34bf-4dc0-8bd3-22d28600263b", "name": "", "operating_status": "ONLINE", "provisioning_status": "ACTIVE", "address": "10.0.1.199", "protocol_port": 8080 }, { "id": "72bfcc95-c901-41b3-95cd-5fba1e6086df", "name": "", "operating_status": "ONLINE", "provisioning_status": "ACTIVE", "address": "10.0.1.253", "protocol_port": 8080 } ] } ] } ] } } Traffic to HTTP: [stack@undercloud-0 ~]$ req='curl -kvvv http://www.example.com'; for i in {1..10}; do $req; echo; done * Rebuilt URL to: http://www.example.com/ * Uses proxy env variable no_proxy == ',10.0.0.115,192.168.24.46' * Trying 10.0.0.219... * TCP_NODELAY set Expected results: [stack@undercloud-0 ~]$ req='curl -k http://www.example.com'; for i in {1..10}; do $req; echo; done novaservers-server1-tu3rjia3mxvr novaservers-server2-iyqw3dftcdd3 novaservers-server1-tu3rjia3mxvr novaservers-server2-iyqw3dftcdd3 novaservers-server1-tu3rjia3mxvr novaservers-server2-iyqw3dftcdd3 novaservers-server1-tu3rjia3mxvr novaservers-server2-iyqw3dftcdd3 novaservers-server1-tu3rjia3mxvr novaservers-server2-iyqw3dftcdd3