Bug 1924657

Created attachment 1754675 [details]
OK Screenshot

Can you confirm the es cluster pods are running? Is it a transient error?

This looks like a permanent Error.

Please note this is OpenShift on OpenStack with Kuryr.

~~~
[stack@osp16amd ocp-test1]$ oc get pods -n openshift-logging -o wide
NAME                                            READY   STATUS      RESTARTS   AGE    IP               NODE                       NOMINATED NODE   READINESS GATES
cluster-logging-operator-9d8548544-qgxfh        1/1     Running     0          32h    10.128.119.191   ocp-phnb2-worker-0-j7mc6   <none>           <none>
elasticsearch-cdm-d8zk1c7i-1-cd879774-6ttwh     2/2     Running     0          55m    10.128.119.138   ocp-phnb2-worker-1-9wpjw   <none>           <none>
elasticsearch-delete-app-1612386000-566c4       0/1     Completed   0          3m1s   10.128.118.156   ocp-phnb2-worker-1-9wpjw   <none>           <none>
elasticsearch-delete-audit-1612386000-h9wxh     0/1     Completed   0          3m1s   10.128.118.178   ocp-phnb2-worker-1-9wpjw   <none>           <none>
elasticsearch-delete-infra-1612386000-x87nr     0/1     Completed   0          3m1s   10.128.118.220   ocp-phnb2-worker-1-9wpjw   <none>           <none>
elasticsearch-rollover-app-1612386000-5pn7w     0/1     Completed   0          3m1s   10.128.118.141   ocp-phnb2-worker-1-9wpjw   <none>           <none>
elasticsearch-rollover-audit-1612386000-w2tgc   0/1     Completed   0          3m1s   10.128.119.117   ocp-phnb2-worker-1-9wpjw   <none>           <none>
elasticsearch-rollover-infra-1612386000-zmhrh   0/1     Completed   0          3m1s   10.128.118.190   ocp-phnb2-worker-1-9wpjw   <none>           <none>
fluentd-7k725                                   1/1     Running     0          55m    10.128.118.85    ocp-phnb2-worker-0-vtqmd   <none>           <none>
fluentd-8nkmp                                   1/1     Running     0          55m    10.128.118.6     ocp-phnb2-master-1         <none>           <none>
fluentd-9p5tl                                   1/1     Running     0          55m    10.128.119.43    ocp-phnb2-master-0         <none>           <none>
fluentd-bpzb9                                   1/1     Running     0          55m    10.128.118.47    ocp-phnb2-worker-0-j7mc6   <none>           <none>
fluentd-j6hbd                                   1/1     Running     0          55m    10.128.119.244   ocp-phnb2-worker-1-9wpjw   <none>           <none>
fluentd-trx8t                                   1/1     Running     0          55m    10.128.118.127   ocp-phnb2-master-2         <none>           <none>
kibana-7594bc85dc-jvb8g                         2/2     Running     0          55m    10.128.118.113   ocp-phnb2-worker-0-vtqmd   <none>           <none>
~~~

I see another very strange thing when a "ClusterLogging" resource is deployed:

~~~
[stack@osp16amd ocp-test1]$ oc get clusterloggings.logging.openshift.io instance -o json | jq -r .spec
{
  "collection": {
    "logs": {
      "fluentd": {},
      "type": "fluentd"
    }
  },
  "curation": {
    "curator": {
      "schedule": "30 3 * * *"
    },
    "type": "curator"
  },
  "logStore": {
    "elasticsearch": {
      "nodeCount": 1,
      "redundancyPolicy": "ZeroRedundancy",
      "resources": {
        "limits": {
          "memory": "2Gi"
        }
      },
      "storage": {
        "size": "30G",
        "storageClassName": "standard"
      }
    },
    "type": "elasticsearch"
  },
  "managementState": "Managed",
  "visualization": {
    "kibana": {
      "replicas": 1
    },
    "type": "kibana"
  }
}
~~~

The scraping of the master fluentd pods is not working. It seems on the master nodes, the fluentd is not listening on 24231, on the worker nodes it is:

Number of Metrics by pod:
~~~
[stack@osp16amd ocp-test1]$ oc get pods -n openshift-logging -l component=fluentd -o name | cut -d/ -f2| xargs -r -n1 -I {} oc rsh {} bash -c 'echo -n {}: && http_proxy="" https_proxy="" HTTP_PROXY="" HTTPS_PROXY="" curl -s -k  https://$POD_IP:24231/metrics | grep ^fluentd | wc -l' 
fluentd-7k725:151
fluentd-8nkmp:0
fluentd-9p5tl:0
fluentd-bpzb9:151
fluentd-j6hbd:151
fluentd-trx8t:0
~~~

Fluentd listening ?
~~~
[stack@osp16amd ocp-test1]$ oc get pods -n openshift-logging -l component=fluentd -o name | cut -d/ -f2| xargs -r -n1 -I {} oc rsh {} bash -c 'echo -n {}:; ss -lt | grep -q 24231 && echo -n LISTENING; echo '
fluentd-7k725:LISTENING
fluentd-8nkmp:
fluentd-9p5tl:
fluentd-bpzb9:LISTENING
fluentd-j6hbd:LISTENING
fluentd-trx8t:
~~~

Matching node names:
~~~
[stack@osp16amd ocp-test1]$ oc get pods -n openshift-logging -l component=fluentd -o name | cut -d/ -f2| xargs -r -n1 -I {} oc rsh {} bash -c 'echo -n {}:; echo $NODE_NAME '
fluentd-7k725:ocp-phnb2-worker-0-vtqmd
fluentd-8nkmp:ocp-phnb2-master-1
fluentd-9p5tl:ocp-phnb2-master-0
fluentd-bpzb9:ocp-phnb2-worker-0-j7mc6
fluentd-j6hbd:ocp-phnb2-worker-1-9wpjw
fluentd-trx8t:ocp-phnb2-master-2
~~~

But maybe this should be the input for another Bug Report ?

Testing with elasticsearch-operator.4.6.0-202103010126.p0, the fix is not in the CSV:

$ oc get csv elasticsearch-operator.4.6.0-202103010126.p0 -oyaml
                image: registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:4659ddeebccecddbcadc322266a4cfa63f0752a0f1eae723a2df777efe4e395a
                imagePullPolicy: IfNotPresent
                name: elasticsearch-operator
                ports:
                - containerPort: 60000
                  name: metrics
                  protocol: TCP
                resources: {}
              nodeSelector:
                kubernetes.io/os: linux
              serviceAccountName: elasticsearch-operator

$ oc get pod elasticsearch-operator-5dc78b5596-88wlz -oyaml
    image: registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:4659ddeebccecddbcadc322266a4cfa63f0752a0f1eae723a2df777efe4e395a
    imagePullPolicy: IfNotPresent
    name: elasticsearch-operator
    ports:
    - containerPort: 60000
      name: metrics
      protocol: TCP
    resources: {}

Verified with elasticsearch-operator.4.6.0-202103202154.p0

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.23 extras update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0954