Bug 1399761 - Logging fluentD daemon set does not set Memory limit for the pods
Summary: Logging fluentD daemon set does not set Memory limit for the pods
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.3.1
Hardware: Unspecified
OS: Linux
unspecified
high
Target Milestone: ---
: ---
Assignee: ewolinet
QA Contact: Xia Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-29 16:48 UTC by Boris Kurktchiev
Modified: 2020-03-11 15:26 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Feature: Fluentd memory limit set by default to 512m Reason: Fluentd will run unbound by memory and can grow to upwards of 3G on a node Result: Fluentd by default will be bound to 512m as part of the template change in the deployer
Clone Of:
Environment:
Last Closed: 2017-02-16 21:03:48 UTC
Target Upstream Version:

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Description Boris Kurktchiev 2016-11-29 16:48:51 UTC 
Description of problem:
The logging deployer does not set a memory limit in the DaemonSet for the fluentD pods, which means the pods can consume as much memory as they like.

Version-Release number of selected component (if applicable):
3.3.1.5

How reproducible:
Deploy Logging and let fluentD run on a busy system and watch its memory usage grow

Additional info:
Spoke with jcantril in #openshift-dev about this and basically it seems to be no real technical reason fluentd needs to have no memory limit. The DaemonSet that gets created by the logging-deployer sets a CPU limit but does not set one for memory.

The problem with not setting one is that fluentd can basically end up consuming the entirety of a node's RAM (unlikely but possible). I currently have pods that are sitting at 3GB utilization.

The daemonset needs to set a "sane" (whatever that may be) Memory limit in the spec as otherwise unpleasantries can happen.
Comment 4 Xia Zhao 2016-12-01 08:09:31 UTC 
Verified with the latest deployer pod on ops registry:
openshift3/logging-deployer        3.4.0               5b8c3c9eb40d   

For fluentd daemonset:

        name: fluentd-elasticsearch
        resources:
          limits:
            cpu: 100m
            memory: 512Mi

For each fluentd pods:

    name: fluentd-elasticsearch
    resources:
      limits:
        cpu: 100m
        memory: 512Mi
      requests:
        cpu: 100m
        memory: 512Mi

# openshift version
openshift v3.4.0.32+d349492
kubernetes v1.4.0+776c994
etcd 3.1.0-rc.0
Comment 5 Troy Dawson 2017-02-16 21:03:48 UTC 
This bug was fixed with the latest OCP 3.4.0 that is already released.
Note You need to log in before you can comment on or make changes to this bug.