Deleted secrets associated with TLS-terminated HTTPS load balancers prevent successful create, update and delete operations on load balancers. In some cases (e.g. additional listener create, see below) the resource goes to ERROR and eventually later the Health Manager detects this inconsistency and tries to repair with an endless amphora failover.


Version-Release number of selected component (if applicable):
OSP 16, 15 and likely but TBC 14 and 13.

How reproducible:
100%

Steps to Reproduce:
$ openstack loadbalancer create --vip-subnet-id private-subnet --name lb-1
$ openstack loadbalancer listener create --protocol-port 443 --protocol TERMINATED_HTTPS --name listener-1 --default-tls-container=$(openstack secret list | awk '/ tls_secret1 / {print $2}') lb-1
$ openstack secret delete $(openstack secret list | awk '/ tls_secret1 / {print $2}')

At this point, there are two different scenarios:

1. Update of load balancer name errors out at Octavia API level and returns immediately.

    $ openstack loadbalancer set --name lb-1-broken lb-1
    Not Found: Not Found. Sorry but your container is in another castle. (HTTP 500) (Request-ID: req-79f67c79-7fdd-4d12-a61d-ccc2fe9a52b7)

    devstack[25057]: HTTPClientError: Not Found: Not Found. Sorry but your container is in another castle.

2. Creation of a new listener. Octavia API accepts the request and forwards the action to the Octavia Worker service. The Worker fails creating the listener with:

    $ openstack loadbalancer listener create --protocol-port 80 --protocol TCP --name listener-2 lb-1
    $ (no exit error)

    ERROR octavia.controller.worker.v1.controller_worker HTTPClientError: Not Found: Not Found. Sorry but your container is in another castle


$ openstack loadbalancer show lb-1
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| created_at          | 2019-10-08T09:03:17                  |
| description         |                                      |
| flavor_id           | None                                 |
| id                  | 02013c9b-bd7e-4862-aa86-3cfd3424d023 |
| listeners           | aaa6ab33-dc82-49a2-a0ee-044958388e64 |
|                     | 40b096ef-ec06-4f54-a79d-60c96cc08e6c |
| name                | lb-1                                 |
| operating_status    | ONLINE                               |
| pools               |                                      |
| project_id          | b87357546ea0444bb9200f34eb491749     |
| provider            | amphora                              |
| provisioning_status | PENDING_UPDATE                       |
| updated_at          | 2019-10-08T09:24:50                  |
| vip_address         | 10.0.0.3                             |
| vip_network_id      | 9cf10ec3-8d3b-4bb7-ac22-eb4a0166f307 |
| vip_port_id         | f27df020-b839-4036-9a54-b8dc9cb04157 |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | 396339d1-8fb8-498e-be21-4502148690e8 |
+---------------------+--------------------------------------+

$ openstack loadbalancer amphora list
+--------------------------------------+--------------------------------------+--------+------------+---------------+----------+
| id                                   | loadbalancer_id                      | status | role       | lb_network_ip | ha_ip    |
+--------------------------------------+--------------------------------------+--------+------------+---------------+----------+
| 36102b6f-0f41-46c4-9421-f0a03a440345 | 02013c9b-bd7e-4862-aa86-3cfd3424d023 | ERROR  | BACKUP     | 192.168.0.58  | 10.0.0.3 |
| ba861ca4-f4ea-4eb6-8a06-0a0b72e4aadf | 02013c9b-bd7e-4862-aa86-3cfd3424d023 | ERROR  | STANDALONE | 192.168.0.5   | 10.0.0.3 |
+--------------------------------------+--------------------------------------+--------+------------+---------------+----------+