Bug 1511576

Summary:	fs.go:359] Stat fs failed. Error: no such file or directory
Product:	OpenShift Container Platform	Reporter:	Eric Paris <eparis>
Component:	Node	Assignee:	Seth Jennings <sjenning>
Status:	CLOSED ERRATA	QA Contact:	DeShuai Ma <dma>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	3.6.1	CC:	aos-bugs, jokerman, mmccomas, wjiang
Target Milestone:	---
Target Release:	3.9.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:	undefined	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-03-28 14:12:08 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Eric Paris 2017-11-09 15:33:02 UTC

I see a bunch of these messages, but they don't tell me what to do. Do they matter? Is there something to fix?

fs.go:359] Stat fs failed. Error: no such file or directory

# rpm -q atomic-openshift-node
atomic-openshift-node-3.6.173.0.5-1.git.0.f30b99e.el7.x86_64

# journalctl --since=-30m | grep 'Stat fs failed.' | wc -l
1573

Comment 1 Seth Jennings 2017-11-09 15:49:12 UTC

Comes from here
https://github.com/google/cadvisor/blob/master/fs/fs.go#L418

Basically the syscall.Statfs() call that getVfsStats() calls returns this error.  Probably in response to a failed/terminated container.

Do you see this recurring over and over for the same path?  Or just one or two times per path?

If just one or two times, then this is log spam and should likely be reduced from Error to V(3).

Comment 2 Eric Paris 2017-11-09 15:52:23 UTC

Someone will need to dig a bit. For example on
starter-us-east-1-node-compute-d9c25
I see that message 1500 times in 30 minutes. And I have no idea what the path is. It isn't in the log...

Comment 3 Tomas Smetana 2017-12-11 10:21:52 UTC

*** Bug 1512714 has been marked as a duplicate of this bug. ***

Comment 4 Tomas Smetana 2017-12-11 10:22:23 UTC

Seems like this was fixed with https://github.com/kubernetes/kubernetes/pull/17883

Comment 5 Seth Jennings 2017-12-11 19:04:43 UTC

Tomas, which commit in cadvisor fixed this?

Comment 6 Seth Jennings 2018-01-19 02:52:16 UTC

Upstream cadvisor:
https://github.com/google/cadvisor/pull/1847

Origin PR:
https://github.com/openshift/origin/pull/18181

Comment 8 weiwei jiang 2018-01-23 09:01:33 UTC

Checked with v3.9.0-0.22.0, and this issue has been fixed.

# journalctl --since=-30m | grep 'Stat fs failed.' | wc -l
0

Comment 11 errata-xmlrpc 2018-03-28 14:12:08 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489