Bug 1832652 - The kibana pod doesn't use the node selector configurations.
Summary: The kibana pod doesn't use the node selector configurations.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 4.5
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.5.0
Assignee: Periklis Tsirakidis
QA Contact: Anping Li
URL:
Whiteboard:
Depends On: 1832662
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-05-07 02:17 UTC by Qiaoling Tang
Modified: 2020-07-13 17:35 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Kibana CRD is a managed resource by elasticsearch-operator in 4.5. The cluster-logging-operator only creates a custom resource for kibana and not the kibana pods anymore. Consequence: Missed to passed node selectors from ClusterLogging CR to Kibana CR. Fix: Pass node selectors to Kibana CR Result: A Kibana CR with custom node selectors is reconciled by the elasticsearch-operator to a pod spec with customized resources for the kibana container and for the kibana proxy container.
Clone Of:
Environment:
Last Closed: 2020-07-13 17:35:44 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:35:57 UTC

Description Qiaoling Tang 2020-05-07 02:17:48 UTC
Description of problem:
Deploy clusterlogging, set nodeSelector for kibana in the clusterlogging instance, then check the logging pods, there doesn't have nodeSelector in the kibana instance, deployment and pods.

$ oc get clusterlogging instance -oyaml
apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  creationTimestamp: "2020-05-07T02:01:30Z"
  generation: 1
  name: instance
  namespace: openshift-logging
  resourceVersion: "65950"
  selfLink: /apis/logging.openshift.io/v1/namespaces/openshift-logging/clusterloggings/instance
  uid: de323da1-5cb4-468e-8526-1f26ef8a7ac4
spec:
  collection:
    logs:
      fluentd:
        nodeSelector:
          logging: test
      type: fluentd
  logStore:
    elasticsearch:
      nodeCount: 3
      nodeSelector:
        logging: test
      redundancyPolicy: SingleRedundancy
      resources:
        requests:
          memory: 2Gi
      storage:
        size: 20Gi
        storageClassName: standard
    retentionPolicy:
      application:
        maxAge: 1d
      audit:
        maxAge: 1w
      infra:
        maxAge: 7d
    type: elasticsearch
  managementState: Managed
  visualization:
    kibana:
      nodeSelector:
        logging: test
      replicas: 1
    type: kibana

No nodes have the label `logging=test`, but the kibana pod could be scheduled to the worker nodes.

$ oc get pod
NAME                                            READY   STATUS    RESTARTS   AGE
cluster-logging-operator-75774d56b6-jpr22       1/1     Running   0          76m
elasticsearch-cdm-3d1zpsdr-1-5c57b569b8-v777x   0/2     Pending   0          4m26s
elasticsearch-cdm-3d1zpsdr-2-66fd465db9-jr5rv   0/2     Pending   0          3m25s
elasticsearch-cdm-3d1zpsdr-3-69b56bbf7d-64cmk   0/2     Pending   0          2m24s
kibana-779ff5c9d5-9tgcj                         2/2     Running   0          4m26s


$ oc get kibana instance -oyaml
apiVersion: logging.openshift.io/v1
kind: Kibana
metadata:
  creationTimestamp: "2020-05-07T02:01:30Z"
  generation: 1
  name: instance
  namespace: openshift-logging
  ownerReferences:
  - apiVersion: logging.openshift.io/v1
    controller: true
    kind: ClusterLogging
    name: instance
    uid: de323da1-5cb4-468e-8526-1f26ef8a7ac4
  resourceVersion: "65585"
  selfLink: /apis/logging.openshift.io/v1/namespaces/openshift-logging/kibanas/instance
  uid: 91a54360-4f08-41d0-ab80-56bf74e7cc3d
spec:
  image: ""
  managementState: Managed
  proxy:
    image: ""
    resources: null
  replicas: 1
  resources:
    limits:
      memory: 736Mi
    requests:
      cpu: 100m
      memory: 736Mi
status:
- deployment: kibana
  pods:
    failed: []
    notReady: []
    ready:
    - kibana-779ff5c9d5-9tgcj
  replicaSets:
  - kibana-779ff5c9d5
  replicas: 1

$ oc get deploy kibana -oyaml |grep -A 3 nodeSelector
            f:nodeSelector:
              .: {}
              f:kubernetes.io/os: {}
            f:restartPolicy: {}
--
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: Always
      schedulerName: default-scheduler

Version-Release number of selected component (if applicable):
Logging images are from 4.5.0-0.ci-2020-05-06-225918	
Manifests are copied from the master branch
Cluster version: 4.5.0-0.nightly-2020-05-06-003431

How reproducible:
Always

Steps to Reproduce:
1.deploy clusterlogging with:
apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  name: "instance"
  namespace: "openshift-logging"
spec:
  managementState: "Managed"
  logStore:
    type: "elasticsearch"
    retentionPolicy: 
      application:
        maxAge: 1d
      infra:
        maxAge: 7d
      audit:
        maxAge: 1w
    elasticsearch:
      nodeCount: 3
      nodeSelector:
        logging: test
      redundancyPolicy: "SingleRedundancy"
      resources:
        requests:
          memory: "2Gi"
      storage:
        storageClassName: "standard"
        size: "20Gi"
  visualization:
    type: "kibana"
    kibana:
      nodeSelector:
        logging: test
      replicas: 1
  collection:
    logs:
      type: "fluentd"
      fluentd:
        nodeSelector:
          logging: test
note: no nodes have label `logging=test` in the cluster
2. check pods status
3.

Actual results:
Kibana doesn't use the node selector in the clusterlogging instance.

Expected results:
The Kibana should use the node selector configurations in the clusterlogging instance.

Additional info:

Comment 1 Periklis Tsirakidis 2020-05-08 17:38:44 UTC
PR in: https://bugzilla.redhat.com/show_bug.cgi?id=1832662

Comment 2 Periklis Tsirakidis 2020-05-14 06:44:11 UTC
Move manually to MODIFIED because same fix as https://bugzilla.redhat.com/show_bug.cgi?id=1832662

Comment 6 Qiaoling Tang 2020-05-15 07:14:48 UTC
Verified with images from 4.5.0-0.ci-2020-05-14-224329.

  spec:
    managementState: Managed
    nodeSelector:
      logging: test
    proxy:
      resources:
        limits:
          memory: 256Mi
        requests:
          cpu: 100m
          memory: 256Mi
    replicas: 1
    resources:
      limits:
        memory: 736Mi
      requests:
        cpu: 100m
        memory: 736Mi
  status:
  - clusterCondition:
      kibana-7cbdff4bfd-mz8kb:
      - lastTransitionTime: "2020-05-15T07:13:06Z"
        message: '0/6 nodes are available: 6 node(s) didn''t match node selector.'
        reason: Unschedulable
        status: "True"
        type: Unschedulable


$ oc get deploy kibana -oyaml |grep -A 3 nodeSelector
            f:nodeSelector:
              .: {}
              f:kubernetes.io/os: {}
              f:logging: {}
--
      nodeSelector:
        kubernetes.io/os: linux
        logging: test
      restartPolicy: Always

Comment 7 errata-xmlrpc 2020-07-13 17:35:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.