Bug 2003491

Summary:	[Kuryr][3.11] Loadbalancer listener disappearing for an OpenShift services managed by Kuryr
Product:	OpenShift Container Platform	Reporter:	Mohammad <mahmad>
Component:	Networking	Assignee:	Robin Cernin <rcernin>
Networking sub component:	kuryr	QA Contact:	Itzik Brown <itbrown>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	urgent
Priority:	urgent	CC:	chris.smart, itbrown, mdemaced, mdulko, mifiedle, rcernin
Version:	3.11.0	Keywords:	Triaged
Target Milestone:	---
Target Release:	3.11.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-12-02 22:01:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Mohammad 2021-09-13 01:26:08 UTC

Description of problem: Loadbalancer listener disappearing for an OpenShift services managed by Kuryr


Version-Release number of selected component (if applicable): 3.11.394


How reproducible: Build an application that uses a service of type loadbalancer and run it for a while. Not all of them have this issue.


Steps to Reproduce:
1. Build mongodb application that uses a loadbalancer service
2. Wait for the loadbalancer listener to disappear and all the corresponding pools


Expected results: This should not happen or kuryr should be able to recover.

Comment 1 Mohammad 2021-09-13 01:44:55 UTC

Symtpoms:

On the openstack side:

$ openstack loadbalancer list |egrep 'mongodb-teame-myenv-dc-cdcs-0|mongodb-teame-myenv-dc-hdms-0|mongodb-teame-myenv-dc-cd-1|mongodb-teame-myenv-dc-hd-6|mongodb-teame-myenv-dc-hd-3'
| 6af51355-e0bf-4af0-a7a9-9479558566de | myenv-teame-myservices/mongodb-teame-myenv-dc-cdcs-0                      | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | XX.XX.133.170 | ACTIVE              | amphora  |
| 332b99b4-0a2d-408d-a684-115c8775fb02 | myenv-teame-myservices/mongodb-teame-myenv-dc-cd-1                        | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | XX.XX.147.184 | ACTIVE              | amphora  |
| edf0371d-b839-468f-bc70-dc8f8b19e151 | myenv-teame-myservices/mongodb-teame-myenv-dc-hd-3                        | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | XX.XX.141.150 | ACTIVE              | amphora  |
$ openstack loadbalancer pool list --loadbalancer  myenv-teame-myservices/mongodb-teame-myenv-dc-cdcs-0

$
$ openstack loadbalancer pool list --loadbalancer  myenv-teame-myservices/mongodb-teame-myenv-dc-cd-1


$
$
$ openstack loadbalancer pool list --loadbalancer  myenv-teame-myservices/mongodb-teame-myenv-dc-hd-3


$
$ openstack loadbalancer listener list --loadbalancer  myenv-teame-myservices/mongodb-teame-myenv-dc-cdcs-0


$
$  openstack loadbalancer listener list --loadbalancer  myenv-teame-myservices/mongodb-teame-myenv-dc-hd-3


On the openshift side:

[openshift@master-1 mologs]$ oc logs -p kuryr-controller-59d5d4fd5b-65d5l > kuryr-controller-59d5d4fd5b-65d5l_raw.txt
[openshift@master-1 mologs]$ oc get svc -n myenv-teame-myservices |egrep 'mongodb-teame-myenv-dc-cdcs-0|mongodb-teame-myenv-dc-cd-1|mongodb-teame-myenv-dc-hd-3'
mongodb-teame-myenv-dc-cd-1                 LoadBalancer   XX.XX.147.184   XXX.ZZZ.10.126   27018:31590/TCP   16d
mongodb-teame-myenv-dc-cdcs-0               LoadBalancer   XX.XX.133.170   XXX.ZZZ.9.197    27019:30108/TCP   16d
mongodb-teame-myenv-dc-hd-3                 LoadBalancer   XX.XX.141.150   XXX.ZZZ.10.199   27018:30906/TCP   15d
[openshift@master-1 mologs]$ oc get pods -n myenv-teame-myservices |egrep 'mongodb-teame-myenv-dc-cdcs-0|mongodb-teame-myenv-dc-cd-1|mongodb-teame-myenv-dc-hd-3'
mongodb-teame-myenv-dc-cd-1                                  3/3       Running     1          16d
mongodb-teame-myenv-dc-cdcs-0                                3/3       Running     0          16d
mongodb-teame-myenv-dc-hd-3                                  3/3       Running     0          4d

Comment 3 Mohammad 2021-09-13 02:13:40 UTC

From the annotations, here are the listener and pool ids:

"listener_id": "16c903e4-a3fc-47da-90df-27b70040abfb",
"loadbalancer_id": "332b99b4-0a2d-408d-a684-115c8775fb02",
"name": "myenv-teame-myservices/mongodb-teame-myenv-dc-cd-1:TCP:27018"
"pool_id": "4ddd00b2-eda5-44b2-b75b-b54e6b53b8b6",

And

"listener_id": "085aac11-0730-4245-8285-87ff8cc2c33e",
"loadbalancer_id": "6af51355-e0bf-4af0-a7a9-9479558566de",
"name": "myenv-teame-myservices/mongodb-teame-myenv-dc-cdcs-0:TCP:27019"
"pool_id": "c87092d2-8c8a-448e-b0c3-01f2e178d068"

and
"listener_id": "d2d03e35-2b00-496f-8f10-6b916cdf2ea6",
"loadbalancer_id": "edf0371d-b839-468f-bc70-dc8f8b19e151",
"name": "myenv-teame-myservices/mongodb-teame-myenv-dc-hd-3:TCP:27018"
"pool_id": "766a5cab-2e2f-4d28-b82b-34e36e55ebc7",

Comment 17 Michał Dulko 2021-10-13 14:24:30 UTC

Okay, so this is kind of patched with [1]. Moving to ON_QA.

[1] https://github.com/openshift/kuryr-kubernetes/pull/548/commits/67498b27c834344bde2dace8c6b2e73ea39bc02e

Comment 18 Michał Dulko 2021-10-13 14:25:38 UTC

Whoops, it should be POST, just as https://bugzilla.redhat.com/show_bug.cgi?id=2002909.

Comment 19 Michał Dulko 2021-10-20 11:15:37 UTC

Okay, this really got patched with https://github.com/openshift/kuryr-kubernetes/pull/548.

Comment 22 Itzik Brown 2021-11-03 16:12:32 UTC

Verified with 
OCP v3.11.542
OSP13 2021-10-19.2

Comment 25 errata-xmlrpc 2021-12-02 22:01:17 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 3.11.569 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4827