Bug 1561121 - [starter-ca-central-1] fluentd pods slowly consuming all available space on instance /var filesystem
Summary: [starter-ca-central-1] fluentd pods slowly consuming all available space on i...
Keywords:
Status: CLOSED DUPLICATE of bug 1560358
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Jeff Cantrill
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-03-27 16:18 UTC by Justin Pierce
Modified: 2018-03-28 01:52 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-03-28 01:52:37 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Justin Pierce 2018-03-27 16:18:03 UTC
Description of problem:
Clusters instances are unable to upgrade because of low disk space on /var. 

fluentd pods are holding open a large number of deleted files which are exhausting the file system space.

```
fluentd    48345  70036     root 1049r      REG              253,1       173   16970432 /var/lib/docker/containers/8bb7037e5b6625468c09172e44a1173a769dc5c03a0dd843a536d1cd054b4c24/8bb7037e5b6625468c09172e44a1173a769dc5c03a0dd843a536d1cd054b4c24-json.log (deleted)
fluentd    48345  70036     root 1050r      REG              253,1     18904   16954593 /var/lib/docker/containers/a88bef9d524ef54ac51f569ff264e14ef4b51dbf9b08b56bf75956231d7cbd0e/a88bef9d524ef54ac51f569ff264e14ef4b51dbf9b08b56bf75956231d7cbd0e-json.log (deleted)
fluentd    48345  70036     root 1051r      REG              253,1     76677    8465172 /var/lib/docker/containers/291cf6e5b313e82aec346407f5fc991dc2ec2c0b6eaf7f2baf0d3914598c3c39/291cf6e5b313e82aec346407f5fc991dc2ec2c0b6eaf7f2baf0d3914598c3c39-json.log (deleted)
fluentd    48345  70036     root 1052r      REG              253,1     18729   25170468 /var/lib/docker/containers/3f6d67171c33bb4e27312834d96ce159f9b00b4ac9b9e44734cb0338789cef7f/3f6d67171c33bb4e27312834d96ce159f9b00b4ac9b9e44734cb0338789cef7f-json.log (deleted)
fluentd    48345  70036     root 1053r      REG              253,1       583    8411926 /var/lib/docker/containers/54c86154f4e08c4c66527bd969934355220699dd4dc1c96d35bfa5e691358365/54c86154f4e08c4c66527bd969934355220699dd4dc1c96d35bfa5e691358365-json.log (deleted)

[root@starter-ca-central-1-node-compute-85bca var]# ps 48345
   PID TTY      STAT   TIME COMMAND
 48345 ?        Ssl    6:52 /usr/bin/ruby /usr/bin/fluentd --no-supervisor
```



Version-Release number of selected component (if applicable):
v3.9.7

How reproducible:
Appears systemic


Additional info:
Deleting the fluentd pod on the node frees the space.

```
[root@starter-ca-central-1-node-compute-85bca ~]# df /var -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-var   16G   14G  2.9G  83% /var

[root@starter-ca-central-1-master-692e9 ~]# oc delete pod logging-fluentd-x8mkp -n logging
pod "logging-fluentd-x8mkp" deleted

[root@starter-ca-central-1-node-compute-85bca ~]# docker ps | grep fluentd
[root@starter-ca-central-1-node-compute-85bca ~]# df /var -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-var   16G  6.0G   11G  38% /var
```

Comment 1 Peter Portante 2018-03-28 01:52:37 UTC

*** This bug has been marked as a duplicate of bug 1560358 ***


Note You need to log in before you can comment on or make changes to this bug.