Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1561121

Summary: [starter-ca-central-1] fluentd pods slowly consuming all available space on instance /var filesystem
Product: OpenShift Container Platform Reporter: Justin Pierce <jupierce>
Component: LoggingAssignee: Jeff Cantrill <jcantril>
Status: CLOSED DUPLICATE QA Contact: Anping Li <anli>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.9.0CC: aos-bugs, pportant, rmeggins
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-03-28 01:52:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Justin Pierce 2018-03-27 16:18:03 UTC 
Description of problem:
Clusters instances are unable to upgrade because of low disk space on /var. 

fluentd pods are holding open a large number of deleted files which are exhausting the file system space.

```
fluentd    48345  70036     root 1049r      REG              253,1       173   16970432 /var/lib/docker/containers/8bb7037e5b6625468c09172e44a1173a769dc5c03a0dd843a536d1cd054b4c24/8bb7037e5b6625468c09172e44a1173a769dc5c03a0dd843a536d1cd054b4c24-json.log (deleted)
fluentd    48345  70036     root 1050r      REG              253,1     18904   16954593 /var/lib/docker/containers/a88bef9d524ef54ac51f569ff264e14ef4b51dbf9b08b56bf75956231d7cbd0e/a88bef9d524ef54ac51f569ff264e14ef4b51dbf9b08b56bf75956231d7cbd0e-json.log (deleted)
fluentd    48345  70036     root 1051r      REG              253,1     76677    8465172 /var/lib/docker/containers/291cf6e5b313e82aec346407f5fc991dc2ec2c0b6eaf7f2baf0d3914598c3c39/291cf6e5b313e82aec346407f5fc991dc2ec2c0b6eaf7f2baf0d3914598c3c39-json.log (deleted)
fluentd    48345  70036     root 1052r      REG              253,1     18729   25170468 /var/lib/docker/containers/3f6d67171c33bb4e27312834d96ce159f9b00b4ac9b9e44734cb0338789cef7f/3f6d67171c33bb4e27312834d96ce159f9b00b4ac9b9e44734cb0338789cef7f-json.log (deleted)
fluentd    48345  70036     root 1053r      REG              253,1       583    8411926 /var/lib/docker/containers/54c86154f4e08c4c66527bd969934355220699dd4dc1c96d35bfa5e691358365/54c86154f4e08c4c66527bd969934355220699dd4dc1c96d35bfa5e691358365-json.log (deleted)

[root@starter-ca-central-1-node-compute-85bca var]# ps 48345
   PID TTY      STAT   TIME COMMAND
 48345 ?        Ssl    6:52 /usr/bin/ruby /usr/bin/fluentd --no-supervisor
```



Version-Release number of selected component (if applicable):
v3.9.7

How reproducible:
Appears systemic


Additional info:
Deleting the fluentd pod on the node frees the space.

```
[root@starter-ca-central-1-node-compute-85bca ~]# df /var -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-var   16G   14G  2.9G  83% /var

[root@starter-ca-central-1-master-692e9 ~]# oc delete pod logging-fluentd-x8mkp -n logging
pod "logging-fluentd-x8mkp" deleted

[root@starter-ca-central-1-node-compute-85bca ~]# docker ps | grep fluentd
[root@starter-ca-central-1-node-compute-85bca ~]# df /var -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-var   16G  6.0G   11G  38% /var
```
Comment 1 Peter Portante 2018-03-28 01:52:37 UTC 

*** This bug has been marked as a duplicate of bug 1560358 ***