Created attachment 1407281 [details]
Configuration and performance result details.

Description of problem:

Fluentd cannot reach the same throughput message rate of reading and forwarding pod logs to Elasticsearch when the pod runtime is CRI-O as compared to Docker.

In OCP 3.9 fluentd performance testing, performance data (see attached document) indicated that fluentd was using considerably more CPU to process logs when the pod runtime was CRI-O.   The I/O pattern of how CRI-O writes/batches logs to disk appears to be a primary cause.   

Example:  Sending 1Kb messages at an aggregate rate of 500 messages/second across 100 pods on a single node.   For a docker runtime, fluentd CPU is 60% of 1 core.  For a cri-o runtime, it is maxing out the core at 100%.  fluentd as configured for use with OCP can only use 1 core.

With a docker runtime, we normally see maximum throughput around 750 - 1000 1K messages/second in our standard AWS m4.xlarge compute node configuration.   With a cri-o runtime the cap is somewhere below 500 1K messages/second

Version-Release number of selected component (if applicable): 3.9.4


How reproducible: Always


Steps to Reproduce:
1.  Install a cluster on AWS with 1 master, 3 infra and 3 compute nodes
2.  Deploy logging 3.9.4 with a  3 node elasticsearch cluster
3.  Use the SVT logtest tool (https://github.com/openshift/svt/blob/master/openshift_scalability/content/logtest/ocp_logtest-README.md) with the configuration below to load a cluster with 300 projects and pods (100 pods/compute node) logging at an aggregated rate of 500 1K messages/second/node
4.  Monitor CPU usage for fluentd

Actual results:

fluentd uses 100% of a single core - the max it can use - on a cri-o pod run time.  It uses ~ 60% CPU on a docker pod runtime.  See the attached document.

Expected results:

fluentd performance on a cri-o runtime compareable to a docker runtime

Additional info:

log test configuration 

- 300 projects and pods (100/node)
- 300 messages/minute/pod = 500 messages/second aggregate on the node.
- each pod sends 6000 1K messages = ~20 minutes of run time.   Can do longer if desired.

projects:
  - num: 300
    basename: logtest
    ifexists: delete
    tuning: default
    templates:
      - num: 1
        file: ./content/logtest/logtest-rc.json
        parameters:
         - REPLICAS: "1"
         - INITIAL_FLAGS: "--num-lines 6000 --line-length 1024 --word-length 9 --rate 300 --fixed-line\n"

tuningsets:
  - name: default
    pods:
      stepping:
        stepsize: 5
        pause: 0 min
      rate_limit:
        delay: 0 ms