Description of problem: Clusters instances are unable to upgrade because of low disk space on /var. fluentd pods are holding open a large number of deleted files which are exhausting the file system space. ``` fluentd 48345 70036 root 1049r REG 253,1 173 16970432 /var/lib/docker/containers/8bb7037e5b6625468c09172e44a1173a769dc5c03a0dd843a536d1cd054b4c24/8bb7037e5b6625468c09172e44a1173a769dc5c03a0dd843a536d1cd054b4c24-json.log (deleted) fluentd 48345 70036 root 1050r REG 253,1 18904 16954593 /var/lib/docker/containers/a88bef9d524ef54ac51f569ff264e14ef4b51dbf9b08b56bf75956231d7cbd0e/a88bef9d524ef54ac51f569ff264e14ef4b51dbf9b08b56bf75956231d7cbd0e-json.log (deleted) fluentd 48345 70036 root 1051r REG 253,1 76677 8465172 /var/lib/docker/containers/291cf6e5b313e82aec346407f5fc991dc2ec2c0b6eaf7f2baf0d3914598c3c39/291cf6e5b313e82aec346407f5fc991dc2ec2c0b6eaf7f2baf0d3914598c3c39-json.log (deleted) fluentd 48345 70036 root 1052r REG 253,1 18729 25170468 /var/lib/docker/containers/3f6d67171c33bb4e27312834d96ce159f9b00b4ac9b9e44734cb0338789cef7f/3f6d67171c33bb4e27312834d96ce159f9b00b4ac9b9e44734cb0338789cef7f-json.log (deleted) fluentd 48345 70036 root 1053r REG 253,1 583 8411926 /var/lib/docker/containers/54c86154f4e08c4c66527bd969934355220699dd4dc1c96d35bfa5e691358365/54c86154f4e08c4c66527bd969934355220699dd4dc1c96d35bfa5e691358365-json.log (deleted) [root@starter-ca-central-1-node-compute-85bca var]# ps 48345 PID TTY STAT TIME COMMAND 48345 ? Ssl 6:52 /usr/bin/ruby /usr/bin/fluentd --no-supervisor ``` Version-Release number of selected component (if applicable): v3.9.7 How reproducible: Appears systemic Additional info: Deleting the fluentd pod on the node frees the space. ``` [root@starter-ca-central-1-node-compute-85bca ~]# df /var -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/rootvg-var 16G 14G 2.9G 83% /var [root@starter-ca-central-1-master-692e9 ~]# oc delete pod logging-fluentd-x8mkp -n logging pod "logging-fluentd-x8mkp" deleted [root@starter-ca-central-1-node-compute-85bca ~]# docker ps | grep fluentd [root@starter-ca-central-1-node-compute-85bca ~]# df /var -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/rootvg-var 16G 6.0G 11G 38% /var ```
*** This bug has been marked as a duplicate of bug 1560358 ***