Description of problem:

When performing the following procedure a certain number of times:
 - Create namespace
 - Create Pod
 - make pod to log a line
 - Remove namespace
Fluent collects all the documents generated by the pods, but they are indexed to the wrong index UID. It's supposed that each document should be appended to the index corresponding to its namespace, having the format project.{project_name}.{project_uuid}.YYYY.MM.DD, instead, some documents are attached to the index of other namespaces


Version-Release number of selected component (if applicable):
OCP/EFK 3.7

How reproducible:
100%

Steps to Reproduce:

- Using the following script:

####################################################
#!/usr/bin/bash

NS="pelle-foo"

for ((x=1; x<=20; x++)); do
    oc create namespace $NS &>/dev/null
    while ! oc get namespace $NS &>/dev/null; do sleep 1; done

    printf "%s\n" $(oc get namespace $NS -o jsonpath={.metadata.uid})

    oc create -f - <<EOF &>/dev/null
apiVersion: v1
kind: Pod
metadata:
  name: bar
  namespace: $NS
spec:
  containers:
  - image: gcr.io/google_containers/busybox:latest
    name: bar
    command: [ "/bin/sh", "-c", "echo foo-bar $x" ]
  restartPolicy: Never
EOF

    while ! oc get pod bar -n $NS &>/dev/null; do sleep 1; done
    sleep 5s
    oc delete namespace $NS &>/dev/null
    while oc get namespace $NS &>/dev/null; do sleep 1; done
done
####################################################

- 20 namespaces are created, which is the expected:

[quicklab@master-0 ~]$ sh estest.sh
d9b7e87f-42fb-11e8-a037-fa163e2c1fb0
e177c93b-42fb-11e8-a037-fa163e2c1fb0
e94bdbad-42fb-11e8-a037-fa163e2c1fb0
f10fc61e-42fb-11e8-a037-fa163e2c1fb0
f8c938fc-42fb-11e8-a037-fa163e2c1fb0
008eeca8-42fc-11e8-a037-fa163e2c1fb0
0860eb9c-42fc-11e8-a037-fa163e2c1fb0
10492553-42fc-11e8-a037-fa163e2c1fb0
183d06ee-42fc-11e8-a037-fa163e2c1fb0
2008d011-42fc-11e8-a037-fa163e2c1fb0
27c98988-42fc-11e8-a037-fa163e2c1fb0
2f938059-42fc-11e8-a037-fa163e2c1fb0
37580849-42fc-11e8-a037-fa163e2c1fb0
3f1edd67-42fc-11e8-a037-fa163e2c1fb0
46dacf7e-42fc-11e8-a037-fa163e2c1fb0
4e9f8525-42fc-11e8-a037-fa163e2c1fb0
56abf58f-42fc-11e8-a037-fa163e2c1fb0
5eadb433-42fc-11e8-a037-fa163e2c1fb0
66647614-42fc-11e8-a037-fa163e2c1fb0
6e29181d-42fc-11e8-a037-fa163e2c1fb0



- And 20 documents created out of each pod (per each namespace), again, which is the expected:

# for i in $(oc rsh -c elasticsearch $es_pod es_util --query=_cat/indices?h=index | grep pelle-foo); do o=$(oc rsh -c elasticsearch $es_pod  es_util --query=$i/_search?sort=@timestamp:desc); jq '.' <<< "$o" | tee -a es_search.txt; done
# grep foo-bar es_search.txt 
          "message": "foo-bar 20",
          "message": "foo-bar 10",
          "message": "foo-bar 5",
          "message": "foo-bar 19",
          "message": "foo-bar 18",
          "message": "foo-bar 17",
          "message": "foo-bar 16",
          "message": "foo-bar 15",
          "message": "foo-bar 13",
          "message": "foo-bar 11",
          "message": "foo-bar 9",
          "message": "foo-bar 8",
          "message": "foo-bar 7",
          "message": "foo-bar 14",
          "message": "foo-bar 12",
          "message": "foo-bar 6",
          "message": "foo-bar 1",


But, Only 3 indices were created, instead of having 20 indices each one with the document generated by its pod:

# oc rsh -c elasticsearch $es_pod es_util  --query=_cat/indices?h=index,docs.count | grep pelle
project.pelle-foo.53a8d3e3-42f7-11e8-a037-fa163e2c1fb0.2018.04.18                 3 
project.pelle-foo.5b71ac2c-42f7-11e8-a037-fa163e2c1fb0.2018.04.18                13 
project.pelle-foo.6b04cc03-42f7-11e8-a037-fa163e2c1fb0.2018.04.18                 4 

Actual results:

Documents appended to the wrong index

Expected results:

One index per namespace should be created and it should contain the document generated only by the pod running within the namespace.

Additional info: