Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1615597

Summary:	Undercloud runs out of inodes as docker takes up all
Product:	Red Hat OpenStack	Reporter:	Pradeep Kilambi <pkilambi>
Component:	python-tripleoclient	Assignee:	RHOS Maint <rhos-maint>
Status:	CLOSED INSUFFICIENT_DATA	QA Contact:	Gurenko Alex <agurenko>
Severity:	urgent	Docs Contact:
Priority:	high
Version:	14.0 (Rocky)	CC:	aschultz, hbrock, jslagle, m.andre, mburns, pkilambi, sbaker
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-09-17 14:58:14 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Pradeep Kilambi 2018-08-13 19:34:28 UTC

On the undercloud node we started seeing these errors:

Could not create lock at /var/run/yum.pid: [Errno 28] No space left on device: '/var/run/yum.pid


digging down looks like docker takes up all the inodes and leaves undercloud starving

[root@undercloud ~]# df -i|grep /run
tmpfs           1537848 1537848        0  100% /run
tmpfs           1537848       9  1537839    1% /run/user/1000

# for i in /run/*; do echo $i; find $i |wc -l; done
<snip>
..

/run/docker
1536761

</snip>

Comment 1 Pradeep Kilambi 2018-08-13 19:40:18 UTC

i see a bunch of these:

# ls /run/docker/libcontainerd/027cf00e5c30c42160e986108cb5aa543e6cc3f508eec72fd9880b7b2bbe5a24/
Display all 49551 possibilities? (y or n)
0000a2685f619905ba2db2d39a5c902a52c15c6f1a99a92d40f892adf66b2f22-stderr
0000a2685f619905ba2db2d39a5c902a52c15c6f1a99a92d40f892adf66b2f22-stdin
0000a2685f619905ba2db2d39a5c902a52c15c6f1a99a92d40f892adf66b2f22-stdout
000520fc0af276793b99980d754a9379220c45f2ea89010a5ba67f9cb8e7fc10-stderr
000520fc0af276793b99980d754a9379220c45f2ea89010a5ba67f9cb8e7fc10-stdin
000520fc0af276793b99980d754a9379220c45f2ea89010a5ba67f9cb8e7fc10-stdout
00053075e466967824c6dfdb0a281628f66f144a45e8701d6b183e55fc69e73c-stderr
00053075e466967824c6dfdb0a281628f66f144a45e8701d6b183e55fc69e73c-stdin
00053075e466967824c6dfdb0a281628f66f144a45e8701d6b183e55fc69e73c-stdout
000ffa7bcc0c531f72ca0cf8c0ae3163f9bce888edb39918adb706d5dbb5823e-stderr
000ffa7bcc0c531f72ca0cf8c0ae3163f9bce888edb39918adb706d5dbb5823e-stdin
000ffa7bcc0c531f72ca0cf8c0ae3163f9bce888edb39918adb706d5dbb5823e-stdout
00117ce9c46fda3666ff88c6d7b4ef0e038977db880e544e1e12a0e77ab0d430-stderr
00117ce9c46fda3666ff88c6d7b4ef0e038977db880e544e1e12a0e77ab0d430-stdin
00117ce9c46fda3666ff88c6d7b4ef0e038977db880e544e1e12a0e77ab0d430-stdout
0017e671266cec303980a24d50dd4e1c065dcf137869301035a3e3d427f13923-stderr
0017e671266cec303980a24d50dd4e1c065dcf137869301035a3e3d427f13923-stdin
....

and many such dirs

Comment 2 Alex Schultz 2018-08-13 21:15:09 UTC

Please provide sosreport or logs

Comment 3 Steve Baker 2018-08-27 21:31:07 UTC

Also, when did this issue occur? During undercloud install? overcloud deploy? later?

Comment 4 Red Hat Bugzilla 2023-09-14 04:33:07 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days