Bug 2028964

Summary:	Whereabouts should reconcile stranded IP addresses
Product:	OpenShift Container Platform	Reporter:	Douglas Smith <dosmith>
Component:	Networking	Assignee:	Douglas Smith <dosmith>
Networking sub component:	multus	QA Contact:	Weibin Liang <weliang>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	high	CC:	weliang
Version:	4.10	Keywords:	FastFix
Target Milestone:	---
Target Release:	4.9.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Enhancement
Doc Text:	Feature: Implements an IP reconciliation job for Whereabouts IPAM CNI called "ip-reconciler" which runs as a Kubernetes cronjob. Reason: On occasion events occur where the CNI DEL action will not complete for a given pod (for example, a forcefully powered off node), and in such a case stored IP address allocations may be left stranded and unable to be allocated without manual intervention. Result: Stranded IP address allocations are garbage collected automatically on a periodic basis to free unused IP addresses.	Story Points:	---
Clone Of:	2028963
Clones:	2028966 (view as bug list)		Environment:
Last Closed:	2022-02-10 06:33:21 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2028963
Bug Blocks:	2028966

Description Douglas Smith 2021-12-03 20:25:45 UTC

+++ This bug was initially created as a clone of Bug #2028963 +++

Description of problem: IP reconciliation is a feature in the latest whereabouts, and due to reports, this feature should be backported all the way to 4.6.z. The feature is in the form of a cron job which reconciles the IP addresses.


Version-Release number of selected component (if applicable): 4.6-4.10


How reproducible: Specialized. Customers often experience this when nodes are rebooted, or pods are force deleted, and therefore CNI DEL calls aren't processed in their entirety by Whereabouts


Steps to Reproduce: (We will produce a procedure which produces orphaned IP addresses)

Actual results: IP addresses will remain stranded, and never utilized again.


Expected results: IP addresses that were stranded become available for use again.


Additional info: 4.10 has the reconciliation code but still requires a bug fix from upstream.

Comment 1 Douglas Smith 2022-01-06 18:22:07 UTC

This can be verified using this procedure: https://gist.github.com/dougbtv/599e8a1a747fde300d46e912f573b40f

Thank you!

Comment 4 Weibin Liang 2022-01-11 22:40:03 UTC

PR should be merged in https://amd64.ocp.releases.ci.openshift.org/releasestream/4.9.0-0.nightly/release/4.9.0-0.nightly-2022-01-10-125832

But the verification failed in 4.9.0-0.nightly-2022-01-11-155222

[weliang@weliang ~]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2022-01-11-155222   True        False         56m     Cluster version is 4.9.0-0.nightly-2022-01-11-155222
[weliang@weliang ~]$ oc get cronjobs -n openshift-multus
No resources found in openshift-multus namespace.
[weliang@weliang ~]$

Comment 5 Weibin Liang 2022-01-18 20:58:02 UTC

Verifying still failed in the latest night build:

[weliang@weliang ~]$ oc get cronjobs -n openshift-multus
No resources found in openshift-multus namespace.
[weliang@weliang ~]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2022-01-17-165424   True        False         14m     Cluster version is 4.9.0-0.nightly-2022-01-17-165424
[weliang@weliang ~]$

Comment 6 Weibin Liang 2022-01-20 16:39:37 UTC

/label qe-approved for https://github.com/openshift/cluster-network-operator/pull/1264

Comment 9 errata-xmlrpc 2022-02-10 06:33:21 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.9.19 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0340