Description of problem:

Octavia is struggling with proper handling of DB connectivity issues bringing down all running loadbalancers. 

I've experienced problems during minor RHOSP13 release update but it can be replicated by simple restart of galera-bundle.

Version-Release number of selected component (if applicable):

  DockerOctaviaApiImage: 192.168.111.1:8787/rhosp13/openstack-octavia-api:13.0-43
  DockerOctaviaConfigImage: 192.168.111.1:8787/rhosp13/openstack-octavia-api:13.0-43
  DockerOctaviaHealthManagerImage: 192.168.111.1:8787/rhosp13/openstack-octavia-health-manager:13.0-45
  DockerOctaviaHousekeepingImage: 192.168.111.1:8787/rhosp13/openstack-octavia-housekeeping:13.0-45
  DockerOctaviaWorkerImage: 192.168.111.1:8787/rhosp13/openstack-octavia-worker:13.0-44


How reproducible:

All the time. Results may be different but are always catastrophic. 

Steps to Reproduce:
1. create a few loadbalancers: 

for i in `seq 0 5` ; do openstack loadbalancer create --vip-subnet-id 2a5c5f64-81d1-4742-a4da-6c9706111f6f --name "loadbalancer-${i}"; done


2. Wait a while so everything will settle:

(overcloud) [stack@director ~]$ openstack loadbalancer list
+--------------------------------------+----------------+----------------------------------+---------------+---------------------+----------+
| id                                   | name           | project_id                       | vip_address   | provisioning_status | provider |
+--------------------------------------+----------------+----------------------------------+---------------+---------------------+----------+
| 2cc2524a-72ea-4c02-ab1a-3e37ea674c68 | loadbalancer-0 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.110 | ACTIVE              | octavia  |
| e19d1426-c351-4c01-89f6-a9842be03c19 | loadbalancer-1 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.105 | ACTIVE              | octavia  |
| 8b794304-212b-4be2-b9a8-66ef2a3afb5d | loadbalancer-2 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.115 | ACTIVE              | octavia  |
| c3bf6eb7-788e-40a3-954a-351ef9b94b4a | loadbalancer-3 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.112 | ACTIVE              | octavia  |
| 32003402-216f-4c17-a675-b1442c96598a | loadbalancer-4 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.113 | ACTIVE              | octavia  |
| 102d8b48-9794-42eb-9c9d-7e21c4409ddb | loadbalancer-5 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.103 | ACTIVE              | octavia  |
+--------------------------------------+----------------+----------------------------------+---------------+---------------------+----------+
(overcloud) [stack@director ~]$ openstack loadbalancer amphora list
+--------------------------------------+--------------------------------------+-----------+------------+---------------+---------------+
| id                                   | loadbalancer_id                      | status    | role       | lb_network_ip | ha_ip         |
+--------------------------------------+--------------------------------------+-----------+------------+---------------+---------------+
| 40703e60-c4b8-43df-a9b4-f4e1ddf2237c | e19d1426-c351-4c01-89f6-a9842be03c19 | ALLOCATED | STANDALONE | 172.24.0.17   | 192.168.5.105 |
| 4bec0db5-e381-4c85-b39b-e0b96629060e | 2cc2524a-72ea-4c02-ab1a-3e37ea674c68 | ALLOCATED | STANDALONE | 172.24.0.6    | 192.168.5.110 |
| 5f317218-c0d8-4316-a3cc-863bc130b509 | 8b794304-212b-4be2-b9a8-66ef2a3afb5d | ALLOCATED | STANDALONE | 172.24.0.5    | 192.168.5.115 |
| 94566449-2646-46b5-924c-e44a746ab5f9 | 102d8b48-9794-42eb-9c9d-7e21c4409ddb | ALLOCATED | STANDALONE | 172.24.0.18   | 192.168.5.103 |
| a8155cf9-5594-4402-9ab3-7f9b440c009d | 32003402-216f-4c17-a675-b1442c96598a | ALLOCATED | STANDALONE | 172.24.0.7    | 192.168.5.113 |
| fc7981f7-6c7e-42b0-a724-1a27b0622a8c | c3bf6eb7-788e-40a3-954a-351ef9b94b4a | ALLOCATED | STANDALONE | 172.24.0.19   | 192.168.5.112 |
+--------------------------------------+--------------------------------------+-----------+------------+---------------+---------------+



3. Restart galera-bundle on the controller node

[root@overcloud-controller-2 octavia]# pcs resource restart galera-bundle

4. Observe that various Octavia services are loosing DB connection

2018-08-28 14:23:47.270 24 ERROR octavia.controller.healthmanager.update_db DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8)

5. Observe Octavia trying to failover amphoras:

2018-08-28 14:24:16.216 25 INFO octavia.controller.healthmanager.health_manager [-] Stale amphora's id is: 4bec0db5-e381-4c85-b39b-e0b96629060e
2018-08-28 14:24:16.247 25 INFO octavia.controller.healthmanager.health_manager [-] Stale amphora's id is: 94566449-2646-46b5-924c-e44a746ab5f9
2018-08-28 14:24:16.293 25 INFO octavia.controller.healthmanager.health_manager [-] Stale amphora's id is: fc7981f7-6c7e-42b0-a724-1a27b0622a8c
2018-08-28 14:24:16.352 25 INFO octavia.controller.healthmanager.health_manager [-] Waiting for 3 failovers to finish
2018-08-28 14:24:16.385 25 WARNING octavia.controller.worker.controller_worker [-] Failing over amphora with no spares pool may cause delays in failover times while a new amphora instance boots.
2018-08-28 14:24:16.429 25 WARNING octavia.controller.worker.controller_worker [-] Failing over amphora with no spares pool may cause delays in failover times while a new amphora instance boots.
2018-08-28 14:24:16.498 25 WARNING octavia.controller.worker.controller_worker [-] Failing over amphora with no spares pool may cause delays in failover times while a new amphora instance boots.

6. Multiple things can happen here now, just to name a few:
 - Octavia can't create new amphora because nova isn't ready yet after DB outage. Nova-api throws 500, Octavia nukes amphora instance and won't try to recreate it again

 - Octavia tries to recreate amphora instance but it get stuck in PENDING_CREATE forever

 - Octavia fails completely reporting DB connection issues, leaving some amphoras in error, some in pending_delete as bellow:


(overcloud) [stack@director ~]$ openstack loadbalancer amphora list
+--------------------------------------+--------------------------------------+----------------+------------+---------------+---------------+
| id                                   | loadbalancer_id                      | status         | role       | lb_network_ip | ha_ip         |
+--------------------------------------+--------------------------------------+----------------+------------+---------------+---------------+
| 40703e60-c4b8-43df-a9b4-f4e1ddf2237c | e19d1426-c351-4c01-89f6-a9842be03c19 | ERROR          | STANDALONE | 172.24.0.17   | 192.168.5.105 |
| 4bec0db5-e381-4c85-b39b-e0b96629060e | 2cc2524a-72ea-4c02-ab1a-3e37ea674c68 | ERROR          | STANDALONE | 172.24.0.6    | 192.168.5.110 |
| 5f317218-c0d8-4316-a3cc-863bc130b509 | 8b794304-212b-4be2-b9a8-66ef2a3afb5d | PENDING_DELETE | STANDALONE | 172.24.0.5    | 192.168.5.115 |
| 94566449-2646-46b5-924c-e44a746ab5f9 | 102d8b48-9794-42eb-9c9d-7e21c4409ddb | PENDING_DELETE | STANDALONE | 172.24.0.18   | 192.168.5.103 |
| a8155cf9-5594-4402-9ab3-7f9b440c009d | 32003402-216f-4c17-a675-b1442c96598a | ERROR          | STANDALONE | 172.24.0.7    | 192.168.5.113 |
| fc7981f7-6c7e-42b0-a724-1a27b0622a8c | c3bf6eb7-788e-40a3-954a-351ef9b94b4a | PENDING_DELETE | STANDALONE | 172.24.0.19   | 192.168.5.112 |
+--------------------------------------+--------------------------------------+----------------+------------+---------------+---------------+


5. Restart Octavia containers across the whole cluster won't help. Amphoras are left as they are or at some point being successfully deleted. No option to failover or do anything - LBs get stuck in PENDING_UPDATE:

(undercloud) [stack@director ~]$ ansible Controller -i /usr/bin/tripleo-ansible-inventory -b -m shell -a "docker ps -f name=octavia -q | xargs -P 5 -I{} docker restart {}"
192.168.111.202 | SUCCESS | rc=0 >>
592833395bb2
1f13973ad6cb
bad8340f4752
097e18890327

192.168.111.201 | SUCCESS | rc=0 >>
c6858d3d8181
7f2b304a7224
3ea76b696667
2abc5096abe1

192.168.111.207 | SUCCESS | rc=0 >>
4a73f0b48578
827cc913c74c
0b8b7dcea332
3a8ce9f1e08a


(overcloud) [stack@director ~]$ openstack loadbalancer list
+--------------------------------------+----------------+----------------------------------+---------------+---------------------+----------+
| id                                   | name           | project_id                       | vip_address   | provisioning_status | provider |
+--------------------------------------+----------------+----------------------------------+---------------+---------------------+----------+
| 2cc2524a-72ea-4c02-ab1a-3e37ea674c68 | loadbalancer-0 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.110 | ERROR               | octavia  |
| e19d1426-c351-4c01-89f6-a9842be03c19 | loadbalancer-1 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.105 | ERROR               | octavia  |
| 8b794304-212b-4be2-b9a8-66ef2a3afb5d | loadbalancer-2 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.115 | ACTIVE              | octavia  |
| c3bf6eb7-788e-40a3-954a-351ef9b94b4a | loadbalancer-3 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.112 | PENDING_UPDATE      | octavia  |
| 32003402-216f-4c17-a675-b1442c96598a | loadbalancer-4 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.113 | ERROR               | octavia  |
| 102d8b48-9794-42eb-9c9d-7e21c4409ddb | loadbalancer-5 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.103 | ACTIVE              | octavia  |
+--------------------------------------+----------------+----------------------------------+---------------+---------------------+----------+
(overcloud) [stack@director ~]$ openstack loadbalancer amphora list
+--------------------------------------+--------------------------------------+----------------+------------+---------------+---------------+
| id                                   | loadbalancer_id                      | status         | role       | lb_network_ip | ha_ip         |
+--------------------------------------+--------------------------------------+----------------+------------+---------------+---------------+
| 40703e60-c4b8-43df-a9b4-f4e1ddf2237c | e19d1426-c351-4c01-89f6-a9842be03c19 | ERROR          | STANDALONE | 172.24.0.17   | 192.168.5.105 |
| 4bec0db5-e381-4c85-b39b-e0b96629060e | 2cc2524a-72ea-4c02-ab1a-3e37ea674c68 | ERROR          | STANDALONE | 172.24.0.6    | 192.168.5.110 |
| 94566449-2646-46b5-924c-e44a746ab5f9 | 102d8b48-9794-42eb-9c9d-7e21c4409ddb | PENDING_DELETE | STANDALONE | 172.24.0.18   | 192.168.5.103 |
| a8155cf9-5594-4402-9ab3-7f9b440c009d | 32003402-216f-4c17-a675-b1442c96598a | ERROR          | STANDALONE | 172.24.0.7    | 192.168.5.113 |
| fc7981f7-6c7e-42b0-a724-1a27b0622a8c | c3bf6eb7-788e-40a3-954a-351ef9b94b4a | PENDING_DELETE | STANDALONE | 172.24.0.19   | 192.168.5.112 |
+--------------------------------------+--------------------------------------+----------------+------------+---------------+---------------+

6. Attempt to nuke everything will fail as well due to orphaned ports with associated security groups which can't be deleted anymore by Octavia:

(overcloud) [stack@director ~]$ openstack loadbalancer list -f value -c id | xargs -P 6 -I{} openstack loadbalancer delete {}
(overcloud) [stack@director ~]$ openstack loadbalancer list 
+--------------------------------------+----------------+----------------------------------+---------------+---------------------+----------+
| id                                   | name           | project_id                       | vip_address   | provisioning_status | provider |
+--------------------------------------+----------------+----------------------------------+---------------+---------------------+----------+
| 2cc2524a-72ea-4c02-ab1a-3e37ea674c68 | loadbalancer-0 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.110 | PENDING_DELETE      | octavia  |
| e19d1426-c351-4c01-89f6-a9842be03c19 | loadbalancer-1 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.105 | PENDING_DELETE      | octavia  |
| 8b794304-212b-4be2-b9a8-66ef2a3afb5d | loadbalancer-2 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.115 | PENDING_DELETE      | octavia  |
| c3bf6eb7-788e-40a3-954a-351ef9b94b4a | loadbalancer-3 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.112 | PENDING_DELETE      | octavia  |
| 32003402-216f-4c17-a675-b1442c96598a | loadbalancer-4 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.113 | PENDING_DELETE      | octavia  |
| 102d8b48-9794-42eb-9c9d-7e21c4409ddb | loadbalancer-5 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.103 | PENDING_DELETE      | octavia  |
+--------------------------------------+----------------+----------------------------------+---------------+---------------------+----------+


2018-08-28 14:44:59.418 23 INFO octavia.network.drivers.neutron.allowed_address_pairs [-] Removing security group 231409b7-216b-4825-8818-304a414fa23c from port 5bb9694a-c444-4abc-b018-269b5a41d364
2018-08-28 14:45:01.108 23 WARNING octavia.network.drivers.neutron.allowed_address_pairs [-] Attempt 1 to remove security group 231409b7-216b-4825-8818-304a414fa23c failed.: Conflict: Security Group 231409b7-216b-4825-8818-304a414fa23c in use.

(...)

2018-08-28 14:45:18.609 23 WARNING octavia.network.drivers.neutron.allowed_address_pairs [-] Attempt 16 to remove security group 231409b7-216b-4825-8818-304a414fa23c failed.: Conflict: Security Group 231409b7-216b-4825-8818-304a414fa23c in use.
2018-08-28 14:45:19.612 23 ERROR octavia.network.drivers.neutron.allowed_address_pairs [-] All attempts to remove security group 231409b7-216b-4825-8818-304a414fa23c have failed.: Conflict: Security Group 231409b7-216b-4825-8818-304a414fa23c in use.
2018-08-28 14:45:19.612 23 ERROR octavia.network.drivers.neutron.allowed_address_pairs Conflict: Security Group 231409b7-216b-4825-8818-304a414fa23c in use.
           |__Flow 'octavia-delete-loadbalancer-flow': DeallocateVIPException: All attempts to remove security group 231409b7-216b-4825-8818-304a414fa23c have failed.
2018-08-28 14:45:19.617 23 ERROR octavia.controller.worker.controller_worker DeallocateVIPException: All attempts to remove security group 231409b7-216b-4825-8818-304a414fa23c have failed.
2018-08-28 14:45:19.695 23 ERROR oslo_messaging.rpc.server [-] Exception during message handling: DeallocateVIPException: All attempts to remove security group 231409b7-216b-4825-8818-304a414fa23c have failed.
2018-08-28 14:45:19.695 23 ERROR oslo_messaging.rpc.server DeallocateVIPException: All attempts to remove security group 231409b7-216b-4825-8818-304a414fa23c have failed.

(overcloud) [stack@director ~]$ openstack loadbalancer list 
+--------------------------------------+----------------+----------------------------------+---------------+---------------------+----------+
| id                                   | name           | project_id                       | vip_address   | provisioning_status | provider |
+--------------------------------------+----------------+----------------------------------+---------------+---------------------+----------+
| 2cc2524a-72ea-4c02-ab1a-3e37ea674c68 | loadbalancer-0 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.110 | ERROR               | octavia  |
| e19d1426-c351-4c01-89f6-a9842be03c19 | loadbalancer-1 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.105 | ERROR               | octavia  |
| 8b794304-212b-4be2-b9a8-66ef2a3afb5d | loadbalancer-2 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.115 | ERROR               | octavia  |
| c3bf6eb7-788e-40a3-954a-351ef9b94b4a | loadbalancer-3 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.112 | ERROR               | octavia  |
| 32003402-216f-4c17-a675-b1442c96598a | loadbalancer-4 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.113 | ERROR               | octavia  |
| 102d8b48-9794-42eb-9c9d-7e21c4409ddb | loadbalancer-5 | ac52d9d546584c80b517fad61f449be5 | 192.168.5.103 | ERROR               | octavia  |
+--------------------------------------+----------------+----------------------------------+---------------+---------------------+----------+

7. The only way do delete those loadbalancers is to nuke ports which were associated with amphoras first, then re-try the deletion.


Actual results:

Octavia breaks all LoadBalancers not allowing for easy recovery due to database connectivity issues which may occur not only due to the failure but also during overcloud update.

Expected results:

Octavia properly handles DB outage, ideally not trying to failover amphoras.

Additional info: