Bug 1958093 - kuryr-controller restarting after 3 days cluster running - pools without members
Summary: kuryr-controller restarting after 3 days cluster running - pools without members
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.8
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.6.z
Assignee: Michał Dulko
QA Contact: rlobillo
URL:
Whiteboard:
Depends On: 1949551
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-05-07 07:33 UTC by OpenShift BugZilla Robot
Modified: 2021-06-08 13:54 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-06-08 13:54:19 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift kuryr-kubernetes pull 509 0 None open [release-4.6] [release-4.7] Bug 1958093: kuryr-controller restarting after 3 days cluster running - pools without member... 2021-05-12 10:15:42 UTC
Red Hat Product Errata RHBA-2021:2157 0 None None None 2021-06-08 13:54:39 UTC

Comment 4 rlobillo 2021-05-31 10:53:35 UTC
Verified on 4.6.0-0.nightly-2021-05-29-053555 over OSP16.1 (RHOS-16.1-RHEL-8-20210323.n.0) with OVN-Octavia.

Create a project with 3 pods and a svc, so below members on the loadbalancer will be created:

$ oc get all                                                                                                                                              
NAME                        READY   STATUS    RESTARTS   AGE                                                                                                                                  
pod/demo-56c97d6845-2lg7s   1/1     Running   0          9m58s                                                                                                                                
pod/demo-56c97d6845-qzq7h   1/1     Running   0          9m58s                                                     
pod/demo-56c97d6845-x48q6   1/1     Running   0          9m58s                                                                                                                               
                                                                                                                                                                                              
NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE                                                                                                                       
service/demo   ClusterIP   172.30.155.72   <none>        80/TCP    9m58s                                                                                                                      
                                                                                                                                                                                             
NAME                   READY   UP-TO-DATE   AVAILABLE   AGE                                                                                                                                   
deployment.apps/demo   3/3     3            3           9m58s         
                                                                                                                   
NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/demo-56c97d6845   3         3         3       9m58s          
                                                                                                                
$ oc rsh pod/demo-56c97d6845-2lg7s curl 172.30.155.72
demo-56c97d6845-x48q6: HELLO! I AM ALIVE!!!                                                                                                                                                   

$ openstack loadbalancer member list test/demo:TCP:80
+--------------------------------------+---------------------------------+----------------------------------+---------------------+----------------+---------------+------------------+-------
-+                                                                          
| id                                   | name                            | project_id                       | provisioning_status | address        | protocol_port | operating_status | weight
 |                                                                  
+--------------------------------------+---------------------------------+----------------------------------+---------------------+----------------+---------------+------------------+-------
-+                                                                                                                                          
| 873df845-bff8-461c-bbd2-17239d0d3140 | test/demo-56c97d6845-2lg7s:8080 | 0b46717610424c5a9bc161550a8545db | ACTIVE              | 10.128.108.102 |          8080 | NO_MONITOR       |      1
 |                                     
| f6d4be33-1b51-4241-a6fc-0489011d5ffc | test/demo-56c97d6845-qzq7h:8080 | 0b46717610424c5a9bc161550a8545db | ACTIVE              | 10.128.109.184 |          8080 | NO_MONITOR       |      1
 |                                             
| 174db786-c09d-49c5-92fe-1978aea14bdd | test/demo-56c97d6845-x48q6:8080 | 0b46717610424c5a9bc161550a8545db | ACTIVE              | 10.128.108.79  |          8080 | NO_MONITOR       |      1
 |                     
+--------------------------------------+---------------------------------+----------------------------------+---------------------+----------------+---------------+------------------+-------
-+                                        

$ for i in `openstack loadbalancer member list test/demo:TCP:80 -c name -f value`; do openstack loadbalancer member delete test/demo:TCP:80 $i; done      
(shiftstack) [stack@undercloud-0 ~]$ openstack loadbalancer member list test/demo:TCP:80

$ oc edit klb/demo -n test
kuryrloadbalancer.openstack.org/demo edited
(# ^ Remove only the member section, including the key, to trigger kuryr-controller reconciliation. If the whole status section is removed, you will hit https://bugzilla.redhat.com/show_bug.cgi?id=1933880)

Wait few seconds:

$ openstack loadbalancer member list test/demo:TCP:80
+--------------------------------------+---------------------------------+----------------------------------+---------------------+----------------+---------------+------------------+-------
-+
| id                                   | name                            | project_id                       | provisioning_status | address        | protocol_port | operating_status | weight
 |
+--------------------------------------+---------------------------------+----------------------------------+---------------------+----------------+---------------+------------------+-------
-+
| 3e0b9916-3c8a-4107-bfb7-84ce67768b33 | test/demo-56c97d6845-2lg7s:8080 | 0b46717610424c5a9bc161550a8545db | ACTIVE              | 10.128.108.102 |          8080 | NO_MONITOR       |      1
 |
| 2df69d3a-d60c-4803-8e89-e9ebbc89eb8b | test/demo-56c97d6845-x48q6:8080 | 0b46717610424c5a9bc161550a8545db | ACTIVE              | 10.128.108.79  |          8080 | NO_MONITOR       |      1
 |
| 8d521d5d-3680-4331-b3f3-caa62508c745 | test/demo-56c97d6845-qzq7h:8080 | 0b46717610424c5a9bc161550a8545db | ACTIVE              | 10.128.109.184 |          8080 | NO_MONITOR       |      1
 |
+--------------------------------------+---------------------------------+----------------------------------+---------------------+----------------+---------------+------------------+-------
-+

Kuryr-controller remained stable:

$ oc get pods -n openshift-kuryr
NAME                                READY   STATUS    RESTARTS   AGE
kuryr-cni-5q7nw                     1/1     Running   0          152m
kuryr-cni-dst2l                     1/1     Running   8          94m
kuryr-cni-gpz99                     1/1     Running   12         94m
kuryr-cni-lrsk8                     1/1     Running   0          157m
kuryr-cni-rzplf                     1/1     Running   0          157m
kuryr-controller-5f76f66694-9qjtv   1/1     Running   0          157m

$ oc rsh pod/demo-56c97d6845-2lg7s curl 172.30.155.72
demo-56c97d6845-x48q6: HELLO! I AM ALIVE!!!

Comment 7 errata-xmlrpc 2021-06-08 13:54:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.32 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2157


Note You need to log in before you can comment on or make changes to this bug.