Bug 1516782 - Pods failing to start on a node with due to IP address exhaustion [NEEDINFO]
Summary: Pods failing to start on a node with due to IP address exhaustion
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 3.6.z
Assignee: Dan Williams
QA Contact: Meng Bo
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-23 11:26 UTC by Jaspreet Kaur
Modified: 2018-10-22 21:46 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: A bug in the node container garbage collection and network setup prevented pod sandboxes from being properly garbage collected. Consequence: Nodes could exhaust the available pool of pod IP addresses, especially if they are restarted and/or containers removed while the node was not running. Fix: Nodes now properly garbage collect and tear down pod sandboxes, ensuring that IP addresses are released to the pool for subsequent re-use. Result: Newly installed nodes should no longer experience IP address exhaustion due to pod sandbox teardown errors. Upgraded nodes should remove all files in /var/lib/cni/networks/openshift-sdn/ during the upgrade, or after upgrade when no pods are running on the node.
Clone Of:
Environment:
Last Closed: 2018-01-23 17:58:09 UTC
Target Upstream Version:
Embargoed:
rpuccini: needinfo?


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:0113 0 normal SHIPPED_LIVE OpenShift Container Platform 3.7 and 3.6 bug fix and enhancement update 2018-01-23 22:55:59 UTC

Description Jaspreet Kaur 2017-11-23 11:26:12 UTC
Description of problem: When creating the application the pod fails to start and is giving below errors :

Nov 23 03:29:26 node.example.com atomic-openshift-node[20732]: W1123 03:29:26.092661   20732 docker_sandbox.go:263] NetworkPlugin cni failed on the status hook for pod "msp-gift-card-discount-2-3txqp_msp": Unexpected command output nsenter: cannot open : No such file or directory
Nov 23 03:29:26 node.example.com atomic-openshift-node[20732]: W1123 03:29:26.120943   20732 docker_sandbox.go:263] NetworkPlugin cni failed on the status hook for pod "msp-gift-card-discount-2-3txqp_msp": Unexpected command output nsenter: cannot open : No such file or directory



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results: It is failing to start


Expected results: It should start


Additional info: Similar to below issue :

https://github.com/openshift/origin/issues/15802
https://github.com/openshift/origin/issues/15210

Comment 11 Meng Bo 2018-01-08 07:29:30 UTC
Tested on v3.6.173.0.94

Delete the containers from the node, there will not have error about the Unexpected command output nsenter: cannot open : No such file or directory

Comment 14 errata-xmlrpc 2018-01-23 17:58:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0113


Note You need to log in before you can comment on or make changes to this bug.