1658688 – Octavia health manager doesn't receive heartbeat message from amp on composable role

Bug 1658688 - Octavia health manager doesn't receive heartbeat message from amp on composable role

Summary: Octavia health manager doesn't receive heartbeat message from amp on composab...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-tripleo-heat-templates
Sub Component:
Version:	14.0 (Rocky)
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	urgent
Target Milestone:	z1
Target Release:	14.0 (Rocky)
Assignee:	Carlos Goncalves
QA Contact:	Alexander Stafeyev
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1659077
TreeView+	depends on / blocked

Reported:	2018-12-12 16:38 UTC by Carlos Goncalves
Modified:	2019-03-18 13:03 UTC (History)
CC List:	5 users (show)
Fixed In Version:	openstack-tripleo-heat-templates-9.0.1-0.20181013060909.el7ost
Doc Type:	Bug Fix
Doc Text:	Previously, the Octavia Health Manager did not receive heartbeat messages from amphorae in Octavia composable role deployments. This caused the firewall on Network nodes to drop packets, and the the load balancers did not return an ONLINE operating status. After this fix, the Octavia Health Manager now receives heartbeat message from amphorae, and the load balancers can go online.
Clone Of:
Clones:	1659077 (view as bug list)
Environment:
Last Closed:	2019-03-18 13:03:25 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Launchpad	1808190	None	None	None	2018-12-12 16:43:41 UTC
OpenStack gerrit	624775	None	MERGED	Fix misnaming of service in firewall rule	2020-03-13 16:28:14 UTC
OpenStack gerrit	625538	None	MERGED	Fix misnaming of service in firewall rule	2020-03-13 16:28:14 UTC
Red Hat Product Errata	RHBA-2019:0446	None	None	None	2019-03-18 13:03:35 UTC

Description Carlos Goncalves 2018-12-12 16:38:12 UTC

Description of problem: the Octavia health manager service does not receive heartbeat messages sent from amphorae. Firewall on networker node drops packets. Firewall rule (protocol UDP, port 5555, interface o-hm0) is missing.

How reproducible:
100%

Steps to Reproduce:
1. create new load balancer
2. wait for provisioning_status to go ACTIVE
3. observe that operating_status will never go ONLINE

Actual results:
operating_status stays OFFLINE forever

Expected results:
operating_status to flip to ONLINE

Additional info:
This only affects composable Octavia deployments

Comment 3 Carlos Goncalves 2018-12-12 16:55:05 UTC

With missing firewall rule in network node:

(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer show lb2
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| created_at          | 2018-12-12T15:51:55                  |
| description         |                                      |
| flavor              |                                      |
| id                  | 4417aca4-9252-46eb-b9a0-e64c85d6b69c |
| listeners           |                                      |
| name                | lb2                                  |
| operating_status    | OFFLINE                              |
| pools               |                                      |
| project_id          | 0bde3075a41d45ef95566863121fec69     |
| provider            | octavia                              |
| provisioning_status | ACTIVE                               |
| updated_at          | 2018-12-12T15:53:26                  |
| vip_address         | 10.0.10.18                           |
| vip_network_id      | 1d128f7e-f180-4c47-81ab-252196d0be8e |
| vip_port_id         | 26461b79-3c8f-4659-b294-7d102896ff8f |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | b4b66a6a-ecf9-4851-87dd-fdbea352d4fc |
+---------------------+--------------------------------------+


After manually creating firewall rule in network node:

(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer show lb2
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| created_at          | 2018-12-12T15:51:55                  |
| description         |                                      |
| flavor              |                                      |
| id                  | 4417aca4-9252-46eb-b9a0-e64c85d6b69c |
| listeners           |                                      |
| name                | lb2                                  |
| operating_status    | ONLINE                               |
| pools               |                                      |
| project_id          | 0bde3075a41d45ef95566863121fec69     |
| provider            | octavia                              |
| provisioning_status | ACTIVE                               |
| updated_at          | 2018-12-12T16:02:41                  |
| vip_address         | 10.0.10.18                           |
| vip_network_id      | 1d128f7e-f180-4c47-81ab-252196d0be8e |
| vip_port_id         | 26461b79-3c8f-4659-b294-7d102896ff8f |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | b4b66a6a-ecf9-4851-87dd-fdbea352d4fc |
+---------------------+--------------------------------------+

Octavia tempest tests ran on an OSP 13 composable Octavia deployment caught this issue [1]. For example, octavia_tempest_plugin.tests.api.v2.test_listener.ListenerAPITest.test_listener_create failed because of this bug.

[1] https://cgoncalves.pt/trash/openstack/octavia/bf02560a-95b2-4b5e-afa6-17d0e3150903.html

Comment 4 Carlos Goncalves 2018-12-12 21:32:23 UTC

OSP 13 composable Octavia with fix applied locally: https://cgoncalves.pt/trash/openstack/octavia/cc8b80a3-8be4-4807-8183-d9324b5415a4.html
Way better now :-)

Comment 9 errata-xmlrpc 2019-03-18 13:03:25 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0446

Note You need to log in before you can comment on or make changes to this bug.