Bug 1507257 – Messages flooded with messages like StopPodSandbox $SHA from runtime service failed: rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod <pod-name> network: CNI failed to retrieve network namespace path: Error: No such container: $SHA

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1507257 - Messages flooded with messages like StopPodSandbox $SHA from runtime service failed: rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod <pod-name> network: CNI failed to retrieve network namespace path: Error: No such container: $SHA

Summary:	Messages flooded with messages like StopPodSandbox $SHA from runtime service ...

Status:	CLOSED ERRATA

Aliases:	None

Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Networking (Show other bugs)
Sub Component:
Version:	3.6.1
Hardware:	Unspecified Unspecified

Priority	high Severity high
Target Milestone:	---
Target Release:	3.7.0
Assigned To:	Rajat Chopra
QA Contact:	Meng Bo
Docs Contact:

URL:
Whiteboard:
Keywords:

Depends On:
Blocks:
	Show dependency tree / graph

Reported:	2017-10-28 19:56 EDT by Alexis Solanas
Modified:	2018-03-27 03:44 EDT (History)
CC List:	9 users (show)

See Also:
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:	Cause: Network namespace path is nil when using hostnetwork. Consequence: The teardown in such a case gives out error messages. Fix: Upstream fix - do not let CNI manage containers with hostnetwork=true. Result: No error messages
Story Points:	---
Clone Of:
Environment:
Last Closed:	2017-11-28 17:20:01 EST
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Red Hat Knowledge Base (Solution)	3235951	None	None	None	2017-11-08 17:52 EST
Origin (Github)	17097	None	None	None	2017-10-31 16:27 EDT
Red Hat Product Errata	RHSA-2017:3188	normal	SHIPPED_LIVE	Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update	2017-11-28 21:34:54 EST

Groups: None (edit)

Description Alexis Solanas 2017-10-28 19:56:29 EDT

Description of problem:

 Messages files are getting full with messages:

StopPodSandbox $SHA from runtime service failed: rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod <pod-name> network: CNI failed to retrieve network namespace path: Error: No such container: $SHA

 Pods no longer exist in the nodes, and there's no reference to them in etcd

 Doing a rm -rf /var/lib/docker doesn't solve the problem


Version-Release number of selected component (if applicable):

atomic-openshift-3.6.173.0.21-1.git.0.f95b0e7

How reproducible:

 Unknown

Steps to Reproduce:
1.
2.
3.

Actual results:

 Too many error messages are logged for pods that no longer exist

Expected results:

 "NetworkPlugin cni failed to teardown pod ... " messages are not logged 


Additional info:


 Upstream issue: https://github.com/kubernetes/kubernetes/issues/44307

Comment 1 Rajat Chopra 2017-10-30 19:44:05 EDT

https://github.com/openshift/origin/pull/17097
This should fix the issue, or we need a chain of backports from upstream.

Comment 3 Javier Ramirez 2017-10-31 10:14:30 EDT

(In reply to Rajat Chopra from comment #1)
> https://github.com/openshift/origin/pull/17097
> This should fix the issue, or we need a chain of backports from upstream.

Thanks Rajat, any idea if there is a workaround we could apply now?

Comment 4 Rajat Chopra 2017-11-02 14:12:50 EDT

Javier, unfortunately no workaround there.

Comment 9 Meng Bo 2017-11-03 06:16:43 EDT

Tested on OCP build v3.7.0-0.190.0
Issue has been fixed.

There will be no error messages in the atomic-openshift-node log when deleting the docker container which is created as hostnetwork enabled pod.


With the same steps, the error messages appear in the node log as below on build prior 3.7.0-0.190.0.

Nov 03 03:51:14 qe-bmeng.interna atomic-openshift-node[23194]: W1103 03:51:14.541341   23194 cni.go:258] CNI failed to retrieve network namespace path: Error: No such container: c5e769d5f72ef91c44e50f9eb6a73a74ce92372e5a705a3edf941a38bec60454

Comment 12 errata-xmlrpc 2017-11-28 17:20:01 EST

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188

Note You need to log in before you can comment on or make changes to this bug.