Bug 1507257 - Messages flooded with messages like StopPodSandbox $SHA from runtime service failed: rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod <pod-name> network: CNI failed to retrieve network namespace path: Error: No such container: $SHA
Summary: Messages flooded with messages like StopPodSandbox $SHA from runtime service ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.6.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.7.0
Assignee: Rajat Chopra
QA Contact: Meng Bo
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-28 23:56 UTC by Alexis Solanas
Modified: 2018-03-27 07:44 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Network namespace path is nil when using hostnetwork. Consequence: The teardown in such a case gives out error messages. Fix: Upstream fix - do not let CNI manage containers with hostnetwork=true. Result: No error messages
Clone Of:
Environment:
Last Closed: 2017-11-28 22:20:01 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Origin (Github) 17097 0 None None None 2017-10-31 20:27:06 UTC
Red Hat Knowledge Base (Solution) 3235951 0 None None None 2017-11-08 22:52:41 UTC
Red Hat Product Errata RHSA-2017:3188 0 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update 2017-11-29 02:34:54 UTC

Description Alexis Solanas 2017-10-28 23:56:29 UTC
Description of problem:

 Messages files are getting full with messages:

StopPodSandbox $SHA from runtime service failed: rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod <pod-name> network: CNI failed to retrieve network namespace path: Error: No such container: $SHA

 Pods no longer exist in the nodes, and there's no reference to them in etcd

 Doing a rm -rf /var/lib/docker doesn't solve the problem


Version-Release number of selected component (if applicable):

atomic-openshift-3.6.173.0.21-1.git.0.f95b0e7

How reproducible:

 Unknown

Steps to Reproduce:
1.
2.
3.

Actual results:

 Too many error messages are logged for pods that no longer exist

Expected results:

 "NetworkPlugin cni failed to teardown pod ... " messages are not logged 


Additional info:


 Upstream issue: https://github.com/kubernetes/kubernetes/issues/44307

Comment 1 Rajat Chopra 2017-10-30 23:44:05 UTC
https://github.com/openshift/origin/pull/17097
This should fix the issue, or we need a chain of backports from upstream.

Comment 3 Javier Ramirez 2017-10-31 14:14:30 UTC
(In reply to Rajat Chopra from comment #1)
> https://github.com/openshift/origin/pull/17097
> This should fix the issue, or we need a chain of backports from upstream.

Thanks Rajat, any idea if there is a workaround we could apply now?

Comment 4 Rajat Chopra 2017-11-02 18:12:50 UTC
Javier, unfortunately no workaround there.

Comment 9 Meng Bo 2017-11-03 10:16:43 UTC
Tested on OCP build v3.7.0-0.190.0
Issue has been fixed.

There will be no error messages in the atomic-openshift-node log when deleting the docker container which is created as hostnetwork enabled pod.


With the same steps, the error messages appear in the node log as below on build prior 3.7.0-0.190.0.

Nov 03 03:51:14 qe-bmeng.interna atomic-openshift-node[23194]: W1103 03:51:14.541341   23194 cni.go:258] CNI failed to retrieve network namespace path: Error: No such container: c5e769d5f72ef91c44e50f9eb6a73a74ce92372e5a705a3edf941a38bec60454

Comment 12 errata-xmlrpc 2017-11-28 22:20:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188


Note You need to log in before you can comment on or make changes to this bug.