Bug 1516782

Summary:	Pods failing to start on a node with due to IP address exhaustion
Product:	OpenShift Container Platform	Reporter:	Jaspreet Kaur <jkaur>
Component:	Networking	Assignee:	Dan Williams <dcbw>
Status:	CLOSED ERRATA	QA Contact:	Meng Bo <bmeng>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	3.6.0	CC:	aos-bugs, bbilgin, dcaldwel, dcbw, glamb, rpuccini, stwalter
Target Milestone:	---	Flags:	rpuccini: needinfo?
Target Release:	3.6.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: A bug in the node container garbage collection and network setup prevented pod sandboxes from being properly garbage collected. Consequence: Nodes could exhaust the available pool of pod IP addresses, especially if they are restarted and/or containers removed while the node was not running. Fix: Nodes now properly garbage collect and tear down pod sandboxes, ensuring that IP addresses are released to the pool for subsequent re-use. Result: Newly installed nodes should no longer experience IP address exhaustion due to pod sandbox teardown errors. Upgraded nodes should remove all files in /var/lib/cni/networks/openshift-sdn/ during the upgrade, or after upgrade when no pods are running on the node.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-01-23 17:58:09 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jaspreet Kaur 2017-11-23 11:26:12 UTC

Description of problem: When creating the application the pod fails to start and is giving below errors :

Nov 23 03:29:26 node.example.com atomic-openshift-node[20732]: W1123 03:29:26.092661   20732 docker_sandbox.go:263] NetworkPlugin cni failed on the status hook for pod "msp-gift-card-discount-2-3txqp_msp": Unexpected command output nsenter: cannot open : No such file or directory
Nov 23 03:29:26 node.example.com atomic-openshift-node[20732]: W1123 03:29:26.120943   20732 docker_sandbox.go:263] NetworkPlugin cni failed on the status hook for pod "msp-gift-card-discount-2-3txqp_msp": Unexpected command output nsenter: cannot open : No such file or directory



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results: It is failing to start


Expected results: It should start


Additional info: Similar to below issue :

https://github.com/openshift/origin/issues/15802
https://github.com/openshift/origin/issues/15210

Comment 11 Meng Bo 2018-01-08 07:29:30 UTC

Tested on v3.6.173.0.94

Delete the containers from the node, there will not have error about the Unexpected command output nsenter: cannot open : No such file or directory

Comment 14 errata-xmlrpc 2018-01-23 17:58:09 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0113