Bug 1615597 - Undercloud runs out of inodes as docker takes up all [NEEDINFO]
Summary: Undercloud runs out of inodes as docker takes up all
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-tripleoclient
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ---
: ---
Assignee: RHOS Maint
QA Contact: Gurenko Alex
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-08-13 19:34 UTC by Pradeep Kilambi
Modified: 2018-09-17 14:58 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-09-17 14:58:14 UTC
Target Upstream Version:
aschultz: needinfo? (pkilambi)


Attachments (Terms of Use)

Description Pradeep Kilambi 2018-08-13 19:34:28 UTC
On the undercloud node we started seeing these errors:

Could not create lock at /var/run/yum.pid: [Errno 28] No space left on device: '/var/run/yum.pid


digging down looks like docker takes up all the inodes and leaves undercloud starving

[root@undercloud ~]# df -i|grep /run
tmpfs           1537848 1537848        0  100% /run
tmpfs           1537848       9  1537839    1% /run/user/1000

# for i in /run/*; do echo $i; find $i |wc -l; done
<snip>
..

/run/docker
1536761

</snip>

Comment 1 Pradeep Kilambi 2018-08-13 19:40:18 UTC
i see a bunch of these:

# ls /run/docker/libcontainerd/027cf00e5c30c42160e986108cb5aa543e6cc3f508eec72fd9880b7b2bbe5a24/
Display all 49551 possibilities? (y or n)
0000a2685f619905ba2db2d39a5c902a52c15c6f1a99a92d40f892adf66b2f22-stderr
0000a2685f619905ba2db2d39a5c902a52c15c6f1a99a92d40f892adf66b2f22-stdin
0000a2685f619905ba2db2d39a5c902a52c15c6f1a99a92d40f892adf66b2f22-stdout
000520fc0af276793b99980d754a9379220c45f2ea89010a5ba67f9cb8e7fc10-stderr
000520fc0af276793b99980d754a9379220c45f2ea89010a5ba67f9cb8e7fc10-stdin
000520fc0af276793b99980d754a9379220c45f2ea89010a5ba67f9cb8e7fc10-stdout
00053075e466967824c6dfdb0a281628f66f144a45e8701d6b183e55fc69e73c-stderr
00053075e466967824c6dfdb0a281628f66f144a45e8701d6b183e55fc69e73c-stdin
00053075e466967824c6dfdb0a281628f66f144a45e8701d6b183e55fc69e73c-stdout
000ffa7bcc0c531f72ca0cf8c0ae3163f9bce888edb39918adb706d5dbb5823e-stderr
000ffa7bcc0c531f72ca0cf8c0ae3163f9bce888edb39918adb706d5dbb5823e-stdin
000ffa7bcc0c531f72ca0cf8c0ae3163f9bce888edb39918adb706d5dbb5823e-stdout
00117ce9c46fda3666ff88c6d7b4ef0e038977db880e544e1e12a0e77ab0d430-stderr
00117ce9c46fda3666ff88c6d7b4ef0e038977db880e544e1e12a0e77ab0d430-stdin
00117ce9c46fda3666ff88c6d7b4ef0e038977db880e544e1e12a0e77ab0d430-stdout
0017e671266cec303980a24d50dd4e1c065dcf137869301035a3e3d427f13923-stderr
0017e671266cec303980a24d50dd4e1c065dcf137869301035a3e3d427f13923-stdin
....

and many such dirs

Comment 2 Alex Schultz 2018-08-13 21:15:09 UTC
Please provide sosreport or logs

Comment 3 Steve Baker 2018-08-27 21:31:07 UTC
Also, when did this issue occur? During undercloud install? overcloud deploy? later?


Note You need to log in before you can comment on or make changes to this bug.