Bug 1958093
| Summary: | kuryr-controller restarting after 3 days cluster running - pools without members | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | OpenShift BugZilla Robot <openshift-bugzilla-robot> |
| Component: | Networking | Assignee: | Michał Dulko <mdulko> |
| Networking sub component: | kuryr | QA Contact: | rlobillo |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | medium | ||
| Priority: | medium | CC: | juriarte, mdulko |
| Version: | 4.8 | Keywords: | Triaged |
| Target Milestone: | --- | ||
| Target Release: | 4.6.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-06-08 13:54:19 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1949551 | ||
| Bug Blocks: | |||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6.32 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2157 |
Verified on 4.6.0-0.nightly-2021-05-29-053555 over OSP16.1 (RHOS-16.1-RHEL-8-20210323.n.0) with OVN-Octavia. Create a project with 3 pods and a svc, so below members on the loadbalancer will be created: $ oc get all NAME READY STATUS RESTARTS AGE pod/demo-56c97d6845-2lg7s 1/1 Running 0 9m58s pod/demo-56c97d6845-qzq7h 1/1 Running 0 9m58s pod/demo-56c97d6845-x48q6 1/1 Running 0 9m58s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/demo ClusterIP 172.30.155.72 <none> 80/TCP 9m58s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/demo 3/3 3 3 9m58s NAME DESIRED CURRENT READY AGE replicaset.apps/demo-56c97d6845 3 3 3 9m58s $ oc rsh pod/demo-56c97d6845-2lg7s curl 172.30.155.72 demo-56c97d6845-x48q6: HELLO! I AM ALIVE!!! $ openstack loadbalancer member list test/demo:TCP:80 +--------------------------------------+---------------------------------+----------------------------------+---------------------+----------------+---------------+------------------+------- -+ | id | name | project_id | provisioning_status | address | protocol_port | operating_status | weight | +--------------------------------------+---------------------------------+----------------------------------+---------------------+----------------+---------------+------------------+------- -+ | 873df845-bff8-461c-bbd2-17239d0d3140 | test/demo-56c97d6845-2lg7s:8080 | 0b46717610424c5a9bc161550a8545db | ACTIVE | 10.128.108.102 | 8080 | NO_MONITOR | 1 | | f6d4be33-1b51-4241-a6fc-0489011d5ffc | test/demo-56c97d6845-qzq7h:8080 | 0b46717610424c5a9bc161550a8545db | ACTIVE | 10.128.109.184 | 8080 | NO_MONITOR | 1 | | 174db786-c09d-49c5-92fe-1978aea14bdd | test/demo-56c97d6845-x48q6:8080 | 0b46717610424c5a9bc161550a8545db | ACTIVE | 10.128.108.79 | 8080 | NO_MONITOR | 1 | +--------------------------------------+---------------------------------+----------------------------------+---------------------+----------------+---------------+------------------+------- -+ $ for i in `openstack loadbalancer member list test/demo:TCP:80 -c name -f value`; do openstack loadbalancer member delete test/demo:TCP:80 $i; done (shiftstack) [stack@undercloud-0 ~]$ openstack loadbalancer member list test/demo:TCP:80 $ oc edit klb/demo -n test kuryrloadbalancer.openstack.org/demo edited (# ^ Remove only the member section, including the key, to trigger kuryr-controller reconciliation. If the whole status section is removed, you will hit https://bugzilla.redhat.com/show_bug.cgi?id=1933880) Wait few seconds: $ openstack loadbalancer member list test/demo:TCP:80 +--------------------------------------+---------------------------------+----------------------------------+---------------------+----------------+---------------+------------------+------- -+ | id | name | project_id | provisioning_status | address | protocol_port | operating_status | weight | +--------------------------------------+---------------------------------+----------------------------------+---------------------+----------------+---------------+------------------+------- -+ | 3e0b9916-3c8a-4107-bfb7-84ce67768b33 | test/demo-56c97d6845-2lg7s:8080 | 0b46717610424c5a9bc161550a8545db | ACTIVE | 10.128.108.102 | 8080 | NO_MONITOR | 1 | | 2df69d3a-d60c-4803-8e89-e9ebbc89eb8b | test/demo-56c97d6845-x48q6:8080 | 0b46717610424c5a9bc161550a8545db | ACTIVE | 10.128.108.79 | 8080 | NO_MONITOR | 1 | | 8d521d5d-3680-4331-b3f3-caa62508c745 | test/demo-56c97d6845-qzq7h:8080 | 0b46717610424c5a9bc161550a8545db | ACTIVE | 10.128.109.184 | 8080 | NO_MONITOR | 1 | +--------------------------------------+---------------------------------+----------------------------------+---------------------+----------------+---------------+------------------+------- -+ Kuryr-controller remained stable: $ oc get pods -n openshift-kuryr NAME READY STATUS RESTARTS AGE kuryr-cni-5q7nw 1/1 Running 0 152m kuryr-cni-dst2l 1/1 Running 8 94m kuryr-cni-gpz99 1/1 Running 12 94m kuryr-cni-lrsk8 1/1 Running 0 157m kuryr-cni-rzplf 1/1 Running 0 157m kuryr-controller-5f76f66694-9qjtv 1/1 Running 0 157m $ oc rsh pod/demo-56c97d6845-2lg7s curl 172.30.155.72 demo-56c97d6845-x48q6: HELLO! I AM ALIVE!!!