Bug 2082599

Summary:	retry logic should have an upper bound on the number of failed attempts
Product:	OpenShift Container Platform	Reporter:	Riccardo Ravaioli <rravaiol>
Component:	Networking	Assignee:	Riccardo Ravaioli <rravaiol>
Networking sub component:	ovn-kubernetes	QA Contact:	Ross Brattain <rbrattai>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	medium
Priority:	medium	CC:	rbrattai, zzhao
Version:	4.11
Target Milestone:	---
Target Release:	4.12.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-01-17 19:48:48 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Riccardo Ravaioli 2022-05-06 14:08:24 UTC

The generic retry logic now used for various resource types in ovnk master attempts to add or delete a given object indefinitely until it succeeds.

Just like with level-driven controllers, we should add an upper bound to number of retries, after which the retry entry should be discarded.

Comment 1 Riccardo Ravaioli 2022-05-06 14:16:13 UTC

upstream PR: https://github.com/ovn-org/ovn-kubernetes/pull/2970

Comment 10 Ross Brattain 2022-08-05 18:59:46 UTC

Scaled a few times up to 123 nodes, unable to get any failed attempts.


4.12.0-0.nightly-2022-08-05-045104


log_ovnkube-master-gk925_ip.eu-central-1.compute.internal:I0805 17:35:39.639802       1 obj_retry.go:1245] Retry successful for *v1.Pod openshift-multus/multus-b2449 after 0 failed attempt(s)
log_ovnkube-master-gk925_ip.eu-central-1.compute.internal:I0805 17:35:39.639896       1 obj_retry.go:1245] Retry successful for *v1.Pod openshift-dns/node-resolver-ktqd7 after 0 failed attempt(s)
log_ovnkube-master-gk925_ip.eu-central-1.compute.internal:I0805 17:35:39.639942       1 obj_retry.go:1245] Retry successful for *v1.Pod openshift-machine-config-operator/machine-config-daemon-grxwt after 0 failed attempt(s)
log_ovnkube-master-gk925_ip.eu-central-1.compute.internal:I0805 17:35:39.642075       1 obj_retry.go:1245] Retry successful for *v1.Pod openshift-network-diagnostics/network-check-target-brvnm after 0 failed attempt(s)
log_ovnkube-master-gk925_ip.eu-central-1.compute.internal:I0805 17:35:39.642480       1 obj_retry.go:1245] Retry successful for *v1.Pod openshift-multus/network-metrics-daemon-d9n6v after 0 failed attempt(s)

Comment 13 errata-xmlrpc 2023-01-17 19:48:48 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7399