Bug 1676467

Summary:	[ACTIVE-STANDBY]- Disabling heartbeat interface for failover, LB goes into error state, New backup amphora is not created
Product:	Red Hat OpenStack	Reporter:	Alexander Stafeyev <astafeye>
Component:	openstack-octavia	Assignee:	Carlos Goncalves <cgoncalves>
Status:	CLOSED WORKSFORME	QA Contact:	Bruna Bonguardo <bbonguar>
Severity:	high	Docs Contact:
Priority:	low
Version:	14.0 (Rocky)	CC:	astafeye, bbonguar, cgoncalves, ihrachys, lpeer, majopela
Target Milestone:	z4	Keywords:	Triaged, ZStream
Target Release:	14.0 (Rocky)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-10-01 17:29:42 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1687321
Bug Blocks:	1698576

Description Alexander Stafeyev 2019-02-12 11:46:28 UTC

Description of problem:
With octavia active standby only: 
On active standby configuration: 
We disabled the MASTER management port in order to block heartbit messages and trigger failover. 

The LB goes to error state, new backup amphora is not created, the backup amphora ( which is now functional ) is still shown as BACKUP 

(overcloud) [stack@undercloud-0 ~]$  openstack loadbalancer amphora list | grep 192.168.1.16
| 5f48afa4-8d58-4960-998e-c714d10ab21f | 7bf8921e-6d37-4667-82c3-6b3ae3410af8 | ALLOCATED | MASTER | 192.168.199.51 | 192.168.1.16 |
| e232e520-4bd5-403c-880a-361a54990df8 | 7bf8921e-6d37-4667-82c3-6b3ae3410af8 | ALLOCATED | BACKUP | 192.168.199.56 | 192.168.1.16 |

(overcloud) [stack@undercloud-0 ~]$ openstack port list | grep 192.168.199.51
| 1671769a-a790-4c56-8d97-d102d2630052 |                                                             | fa:16:3e:62:c7:32 | ip_address='192.168.199.51', subnet_id='06096d99-a0bc-4533-a6c8-f8824dddbf2e' | ACTIVE |


(overcloud) [stack@undercloud-0 ~]$ openstack port set 1671769a-a790-4c56-8d97-d102d2630052 --disable 


After port disabling: 

(overcloud) [stack@undercloud-0 ~]$  openstack loadbalancer amphora list | grep 192.168.1.16
| e232e520-4bd5-403c-880a-361a54990df8 | 7bf8921e-6d37-4667-82c3-6b3ae3410af8 | ALLOCATED | BACKUP | 192.168.199.56 | 192.168.1.16 |




2018-05-14 14:11:51.038 22 ERROR octavia.controller.worker.controller_worker [req-94d8360b-097b-44d0-a9aa-22fb363a353a - 64ba63c12a9a46288fd4623295d81bc0 - - -] Failover exception: failed to detect a valid IP address from None: AddrFormatError: failed to detect a valid IP address from None
2018-05-14 14:11:51.060 22 INFO octavia.controller.healthmanager.health_manager [-] Attempted 1 failovers of amphora
2018-05-14 14:11:51.060 22 INFO octavia.controller.healthmanager.health_manager [-] Failed at 1 failovers of amphora
2018-05-14 14:11:51.061 22 INFO octavia.controller.healthmanager.health_manager [-] Cancelled 0 failovers of amphora
2018-05-14 14:11:51.061 22 INFO octavia.controller.healthmanager.health_manager [-] Successfully completed 0 failovers of amphora


Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1. create a LB
2. set the master amphora management port to disable ( --disabled) 
3. Wait 

Actual results:
The LB goes into error mode . master amphora is deleted. Backup amphora goes to error mode. 

Expected results:
Failover shoul occur properly 

Additional info:
https://bugzilla.redhat.com/show_bug.cgi?id=1655431#c16

Comment 1 Alexander Stafeyev 2019-02-12 11:57:37 UTC

https://bugzilla.redhat.com/show_bug.cgi?id=1655431#c12

Comment 4 Carlos Goncalves 2019-02-13 15:04:58 UTC

Active-standby is community supported only. Could you please file a story upstream?

Comment 5 Alexander Stafeyev 2019-03-04 10:24:10 UTC

https://storyboard.openstack.org/#!/story/2005127

Comment 6 Carlos Goncalves 2019-03-13 15:11:11 UTC

Let's give this another try once https://bugzilla.redhat.com/show_bug.cgi?id=1687321 is fixed and code available.

Comment 7 Carlos Goncalves 2019-04-17 14:46:42 UTC

RHBZ #1687321 is ON_QA. We can resume this work.

Comment 11 Carlos Goncalves 2019-10-01 17:29:42 UTC

I cannot reproduce this with latest OSP 13 z8. I believe it was fixed between the time the issue was reported and now.


(overcloud2) [stack@undercloud-0 ~]$ openstack loadbalancer show lb-1
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| created_at          | 2019-10-01T17:18:01                  |
| description         |                                      |
| flavor              |                                      |
| id                  | 56b606ed-9265-4e7c-94bd-9633eab217a9 |
| listeners           |                                      |
| name                | lb-1                                 |
| operating_status    | ONLINE                               |
| pools               |                                      |
| project_id          | 00f15374552e4ea7a7bb980fb655fb53     |
| provider            | octavia                              |
| provisioning_status | ACTIVE                               |
| updated_at          | 2019-10-01T17:20:17                  |
| vip_address         | 10.99.0.6                            |
| vip_network_id      | 4e61726c-9658-41a6-91a4-8392a1a541b8 |
| vip_port_id         | ea8741eb-3c3b-4635-9959-0befd472c8ae |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | 2c8561f6-f2ce-4274-8cd2-0dbe300cf593 |
+---------------------+--------------------------------------+

(overcloud2) [stack@undercloud-0 ~]$ openstack loadbalancer amphora list
+--------------------------------------+--------------------------------------+-----------+--------+---------------+-----------+
| id                                   | loadbalancer_id                      | status    | role   | lb_network_ip | ha_ip     |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+-----------+
| 072d826d-5ebf-4506-b5cd-f1be19092778 | 56b606ed-9265-4e7c-94bd-9633eab217a9 | ALLOCATED | MASTER | 172.24.0.12   | 10.99.0.6 |
| e87a4825-7337-4f19-b3db-1a7318ba0579 | 56b606ed-9265-4e7c-94bd-9633eab217a9 | ALLOCATED | BACKUP | 172.24.0.27   | 10.99.0.6 |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+-----------+

(overcloud2) [stack@undercloud-0 ~]$ openstack loadbalancer listener create --protocol HTTP --protocol-port 80 --name listener-1 lb-1
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| connection_limit          | -1                                   |
| created_at                | 2019-10-01T17:21:32                  |
| default_pool_id           | None                                 |
| default_tls_container_ref | None                                 |
| description               |                                      |
| id                        | c8357785-37eb-438d-ba17-5a01e5ae052b |
| insert_headers            | None                                 |
| l7policies                |                                      |
| loadbalancers             | 56b606ed-9265-4e7c-94bd-9633eab217a9 |
| name                      | listener-1                           |
| operating_status          | OFFLINE                              |
| project_id                | 00f15374552e4ea7a7bb980fb655fb53     |
| protocol                  | HTTP                                 |
| protocol_port             | 80                                   |
| provisioning_status       | PENDING_CREATE                       |
| sni_container_refs        | []                                   |
| updated_at                | None                                 |
+---------------------------+--------------------------------------+

(overcloud2) [stack@undercloud-0 ~]$ openstack loadbalancer listener show listener-1
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| connection_limit          | -1                                   |
| created_at                | 2019-10-01T17:21:32                  |
| default_pool_id           | None                                 |
| default_tls_container_ref | None                                 |
| description               |                                      |
| id                        | c8357785-37eb-438d-ba17-5a01e5ae052b |
| insert_headers            | None                                 |
| l7policies                |                                      |
| loadbalancers             | 56b606ed-9265-4e7c-94bd-9633eab217a9 |
| name                      | listener-1                           |
| operating_status          | ONLINE                               |
| project_id                | 00f15374552e4ea7a7bb980fb655fb53     |
| protocol                  | HTTP                                 |
| protocol_port             | 80                                   |
| provisioning_status       | ACTIVE                               |
| sni_container_refs        | []                                   |
| updated_at                | 2019-10-01T17:21:40                  |
+---------------------------+--------------------------------------+

(overcloud2) [stack@undercloud-0 ~]$ openstack port list | grep 172.24.0.12
| 26e7270b-a7fa-4043-b587-ceb362b89a1b |                                                              | fa:16:3e:15:92:48 | ip_address='172.24.0.12', subnet_id='22e76954-1b46-470e-a4a3-ad467fdcc518' | ACTIVE |
(overcloud2) [stack@undercloud-0 ~]$ openstack port set 26e7270b-a7fa-4043-b587-ceb362b89a1b --disable

overcloud2) [stack@undercloud-0 ~]$ openstack loadbalancer amphora list
+--------------------------------------+--------------------------------------+-----------+--------+---------------+-----------+
| id                                   | loadbalancer_id                      | status    | role   | lb_network_ip | ha_ip     |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+-----------+
| bb387472-1b7f-424a-9229-7f8561bd143a | None                                 | BOOTING   | None   | None          | None      |
| e87a4825-7337-4f19-b3db-1a7318ba0579 | 56b606ed-9265-4e7c-94bd-9633eab217a9 | ALLOCATED | BACKUP | 172.24.0.27   | 10.99.0.6 |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+-----------+

(overcloud2) [stack@undercloud-0 ~]$ openstack loadbalancer amphora list
+--------------------------------------+--------------------------------------+-----------+--------+---------------+-----------+
| id                                   | loadbalancer_id                      | status    | role   | lb_network_ip | ha_ip     |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+-----------+
| bb387472-1b7f-424a-9229-7f8561bd143a | 56b606ed-9265-4e7c-94bd-9633eab217a9 | ALLOCATED | MASTER | 172.24.0.15   | 10.99.0.6 |
| e87a4825-7337-4f19-b3db-1a7318ba0579 | 56b606ed-9265-4e7c-94bd-9633eab217a9 | ALLOCATED | BACKUP | 172.24.0.27   | 10.99.0.6 |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+-----------