Bug 1516782

Summary: Pods failing to start on a node with due to IP address exhaustion
Product: OpenShift Container Platform Reporter: Jaspreet Kaur <jkaur>
Component: NetworkingAssignee: Dan Williams <dcbw>
Status: CLOSED ERRATA QA Contact: Meng Bo <bmeng>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.6.0CC: aos-bugs, bbilgin, dcaldwel, dcbw, glamb, rpuccini, stwalter
Target Milestone: ---Flags: rpuccini: needinfo?
Target Release: 3.6.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: A bug in the node container garbage collection and network setup prevented pod sandboxes from being properly garbage collected. Consequence: Nodes could exhaust the available pool of pod IP addresses, especially if they are restarted and/or containers removed while the node was not running. Fix: Nodes now properly garbage collect and tear down pod sandboxes, ensuring that IP addresses are released to the pool for subsequent re-use. Result: Newly installed nodes should no longer experience IP address exhaustion due to pod sandbox teardown errors. Upgraded nodes should remove all files in /var/lib/cni/networks/openshift-sdn/ during the upgrade, or after upgrade when no pods are running on the node.
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-01-23 17:58:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Jaspreet Kaur 2017-11-23 11:26:12 UTC
Description of problem: When creating the application the pod fails to start and is giving below errors :

Nov 23 03:29:26 node.example.com atomic-openshift-node[20732]: W1123 03:29:26.092661   20732 docker_sandbox.go:263] NetworkPlugin cni failed on the status hook for pod "msp-gift-card-discount-2-3txqp_msp": Unexpected command output nsenter: cannot open : No such file or directory
Nov 23 03:29:26 node.example.com atomic-openshift-node[20732]: W1123 03:29:26.120943   20732 docker_sandbox.go:263] NetworkPlugin cni failed on the status hook for pod "msp-gift-card-discount-2-3txqp_msp": Unexpected command output nsenter: cannot open : No such file or directory



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results: It is failing to start


Expected results: It should start


Additional info: Similar to below issue :

https://github.com/openshift/origin/issues/15802
https://github.com/openshift/origin/issues/15210

Comment 11 Meng Bo 2018-01-08 07:29:30 UTC
Tested on v3.6.173.0.94

Delete the containers from the node, there will not have error about the Unexpected command output nsenter: cannot open : No such file or directory

Comment 14 errata-xmlrpc 2018-01-23 17:58:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0113