Description of problem: Loadbalancer listener disappearing for an OpenShift services managed by Kuryr Version-Release number of selected component (if applicable): 3.11.394 How reproducible: Build an application that uses a service of type loadbalancer and run it for a while. Not all of them have this issue. Steps to Reproduce: 1. Build mongodb application that uses a loadbalancer service 2. Wait for the loadbalancer listener to disappear and all the corresponding pools Expected results: This should not happen or kuryr should be able to recover.
Symtpoms: On the openstack side: $ openstack loadbalancer list |egrep 'mongodb-teame-myenv-dc-cdcs-0|mongodb-teame-myenv-dc-hdms-0|mongodb-teame-myenv-dc-cd-1|mongodb-teame-myenv-dc-hd-6|mongodb-teame-myenv-dc-hd-3' | 6af51355-e0bf-4af0-a7a9-9479558566de | myenv-teame-myservices/mongodb-teame-myenv-dc-cdcs-0 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | XX.XX.133.170 | ACTIVE | amphora | | 332b99b4-0a2d-408d-a684-115c8775fb02 | myenv-teame-myservices/mongodb-teame-myenv-dc-cd-1 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | XX.XX.147.184 | ACTIVE | amphora | | edf0371d-b839-468f-bc70-dc8f8b19e151 | myenv-teame-myservices/mongodb-teame-myenv-dc-hd-3 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | XX.XX.141.150 | ACTIVE | amphora | $ openstack loadbalancer pool list --loadbalancer myenv-teame-myservices/mongodb-teame-myenv-dc-cdcs-0 $ $ openstack loadbalancer pool list --loadbalancer myenv-teame-myservices/mongodb-teame-myenv-dc-cd-1 $ $ $ openstack loadbalancer pool list --loadbalancer myenv-teame-myservices/mongodb-teame-myenv-dc-hd-3 $ $ openstack loadbalancer listener list --loadbalancer myenv-teame-myservices/mongodb-teame-myenv-dc-cdcs-0 $ $ openstack loadbalancer listener list --loadbalancer myenv-teame-myservices/mongodb-teame-myenv-dc-hd-3 On the openshift side: [openshift@master-1 mologs]$ oc logs -p kuryr-controller-59d5d4fd5b-65d5l > kuryr-controller-59d5d4fd5b-65d5l_raw.txt [openshift@master-1 mologs]$ oc get svc -n myenv-teame-myservices |egrep 'mongodb-teame-myenv-dc-cdcs-0|mongodb-teame-myenv-dc-cd-1|mongodb-teame-myenv-dc-hd-3' mongodb-teame-myenv-dc-cd-1 LoadBalancer XX.XX.147.184 XXX.ZZZ.10.126 27018:31590/TCP 16d mongodb-teame-myenv-dc-cdcs-0 LoadBalancer XX.XX.133.170 XXX.ZZZ.9.197 27019:30108/TCP 16d mongodb-teame-myenv-dc-hd-3 LoadBalancer XX.XX.141.150 XXX.ZZZ.10.199 27018:30906/TCP 15d [openshift@master-1 mologs]$ oc get pods -n myenv-teame-myservices |egrep 'mongodb-teame-myenv-dc-cdcs-0|mongodb-teame-myenv-dc-cd-1|mongodb-teame-myenv-dc-hd-3' mongodb-teame-myenv-dc-cd-1 3/3 Running 1 16d mongodb-teame-myenv-dc-cdcs-0 3/3 Running 0 16d mongodb-teame-myenv-dc-hd-3 3/3 Running 0 4d
From the annotations, here are the listener and pool ids: "listener_id": "16c903e4-a3fc-47da-90df-27b70040abfb", "loadbalancer_id": "332b99b4-0a2d-408d-a684-115c8775fb02", "name": "myenv-teame-myservices/mongodb-teame-myenv-dc-cd-1:TCP:27018" "pool_id": "4ddd00b2-eda5-44b2-b75b-b54e6b53b8b6", And "listener_id": "085aac11-0730-4245-8285-87ff8cc2c33e", "loadbalancer_id": "6af51355-e0bf-4af0-a7a9-9479558566de", "name": "myenv-teame-myservices/mongodb-teame-myenv-dc-cdcs-0:TCP:27019" "pool_id": "c87092d2-8c8a-448e-b0c3-01f2e178d068" and "listener_id": "d2d03e35-2b00-496f-8f10-6b916cdf2ea6", "loadbalancer_id": "edf0371d-b839-468f-bc70-dc8f8b19e151", "name": "myenv-teame-myservices/mongodb-teame-myenv-dc-hd-3:TCP:27018" "pool_id": "766a5cab-2e2f-4d28-b82b-34e36e55ebc7",
Okay, so this is kind of patched with [1]. Moving to ON_QA. [1] https://github.com/openshift/kuryr-kubernetes/pull/548/commits/67498b27c834344bde2dace8c6b2e73ea39bc02e
Whoops, it should be POST, just as https://bugzilla.redhat.com/show_bug.cgi?id=2002909.
Okay, this really got patched with https://github.com/openshift/kuryr-kubernetes/pull/548.
Verified with OCP v3.11.542 OSP13 2021-10-19.2
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 3.11.569 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:4827