Bug 1882982 - CLO is repeating error message: `clusterlogging-controller error updating status`
Summary: CLO is repeating error message: `clusterlogging-controller error updating st...
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 4.6
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: ---
: 4.7.0
Assignee: Jeff Cantrill
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-27 07:51 UTC by Qiaoling Tang
Modified: 2020-10-12 17:19 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-12 17:19:50 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Qiaoling Tang 2020-09-27 07:51:50 UTC
Description of problem:
Deploy logging, then check the CLO logs, it keeps repeating some error messages:
time="2020-09-27T07:22:35Z" level=error msg="clusterlogging-controller error updating status: Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again"
{"level":"error","ts":1601191355.1462588,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"clusterlogging-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90"}
time="2020-09-27T07:23:08Z" level=error msg="clusterlogging-controller error updating status: Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again"
{"level":"error","ts":1601191388.7426267,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"clusterlogging-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90"}

$ oc get cl instance -oyaml
apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  creationTimestamp: "2020-09-27T06:56:26Z"
  generation: 1
  name: instance
  namespace: openshift-logging
  resourceVersion: "275945"
  selfLink: /apis/logging.openshift.io/v1/namespaces/openshift-logging/clusterloggings/instance
  uid: 9dbd7b5f-b9ea-489d-ad35-1bf01d698ac1
spec:
  collection:
    logs:
      fluentd: {}
      type: fluentd
  curation:
    curator:
      schedule: '*/10 * * * *'
    type: curator
  logStore:
    elasticsearch:
      nodeCount: 3
      redundancyPolicy: SingleRedundancy
      resources:
        requests:
          memory: 2Gi
    type: elasticsearch
  managementState: Managed
  visualization:
    kibana:
      replicas: 1
    type: kibana
status:
  clusterConditions:
  - lastTransitionTime: "2020-09-27T06:57:04Z"
    status: "False"
    type: CollectorDeadEnd
  collection:
    logs:
      fluentdStatus:
        daemonSet: fluentd
        nodes:
          fluentd-2lprk: juzhao-share2-r6v5v-compute-0
          fluentd-c6mh4: juzhao-share2-r6v5v-control-plane-2
          fluentd-c9lzb: juzhao-share2-r6v5v-compute-2
          fluentd-h2gsv: juzhao-share2-r6v5v-rhel-2
          fluentd-hggd9: juzhao-share2-r6v5v-compute-1
          fluentd-mmwx8: juzhao-share2-r6v5v-control-plane-0
          fluentd-sv7hj: juzhao-share2-r6v5v-rhel-0
          fluentd-txgch: juzhao-share2-r6v5v-control-plane-1
          fluentd-z9pff: juzhao-share2-r6v5v-rhel-1
        pods:
          failed: []
          notReady: []
          ready:
          - fluentd-2lprk
          - fluentd-c6mh4
          - fluentd-c9lzb
          - fluentd-h2gsv
          - fluentd-hggd9
          - fluentd-mmwx8
          - fluentd-sv7hj
          - fluentd-txgch
          - fluentd-z9pff
  curation:
    curatorStatus:
    - clusterCondition:
        curator-1601192400-92lkp:
        - lastTransitionTime: "2020-09-27T07:40:03Z"
          reason: Completed
          status: "True"
          type: ContainerTerminated
      cronJobs: curator
      schedules: '*/10 * * * *'
      suspended: false
  logStore:
    elasticsearchStatus:
    - cluster:
        activePrimaryShards: 12
        activeShards: 24
        initializingShards: 0
        numDataNodes: 3
        numNodes: 3
        pendingTasks: 0
        relocatingShards: 0
        status: green
        unassignedShards: 0
      clusterName: elasticsearch
      nodeConditions:
        elasticsearch-cdm-ppb0i0fs-1: []
        elasticsearch-cdm-ppb0i0fs-2: []
        elasticsearch-cdm-ppb0i0fs-3: []
      nodeCount: 3
      pods:
        client:
          failed: []
          notReady: []
          ready:
          - elasticsearch-cdm-ppb0i0fs-1-5dfdf8fbf7-mm92h
          - elasticsearch-cdm-ppb0i0fs-2-5d54cfd649-d2rwr
          - elasticsearch-cdm-ppb0i0fs-3-754666fffc-59dcl
        data:
          failed: []
          notReady: []
          ready:
          - elasticsearch-cdm-ppb0i0fs-1-5dfdf8fbf7-mm92h
          - elasticsearch-cdm-ppb0i0fs-2-5d54cfd649-d2rwr
          - elasticsearch-cdm-ppb0i0fs-3-754666fffc-59dcl
        master:
          failed: []
          notReady: []
          ready:
          - elasticsearch-cdm-ppb0i0fs-1-5dfdf8fbf7-mm92h
          - elasticsearch-cdm-ppb0i0fs-2-5d54cfd649-d2rwr
          - elasticsearch-cdm-ppb0i0fs-3-754666fffc-59dcl
      shardAllocationEnabled: all
  visualization:
    kibanaStatus:
    - deployment: kibana
      pods:
        failed: []
        notReady: []
        ready:
        - kibana-76b76456ff-zfg2w
      replicaSets:
      - kibana-76b76456ff
      replicas: 1


Version-Release number of selected component (if applicable):
clusterlogging.4.6.0-202009240527.p0
elasticsearch-operator.4.6.0-202009232303.p0


How reproducible:
Always

Steps to Reproduce:
1. deploy logging
2. check CLO pod log
3.

Actual results:


Expected results:
Don't have these logs in the CLO.

Additional info:

Comment 1 Vimal Kumar 2020-09-28 15:11:44 UTC
This error is returned by k8s , and CLO will be invoked again after 30 secs. Does this log statement appear repeatedly?

As per [1] this is a benign error and can be ignored, and perhaps it should be a warning.


[1] https://github.com/kubernetes/kubernetes/issues/28149#issuecomment-229469157

Comment 2 Jeff Cantrill 2020-09-28 16:30:47 UTC
Setting to low as this is a non-blocking 4.6 issue

Comment 3 Qiaoling Tang 2020-09-29 05:44:11 UTC
(In reply to Vimal Kumar from comment #1)
> This error is returned by k8s , and CLO will be invoked again after 30 secs.
> Does this log statement appear repeatedly?
> 

Seems different cluster has different behavior.

This log statement appeared repeatedly when I reported this bz, but I checked the logs again in a cluster I launched today, it stopped appearing after running for 10 minutes.

Comment 4 Jeff Cantrill 2020-10-01 12:42:06 UTC
Moving to 4.7 to satisfy CF requirements for 4.6

Comment 5 Jeff Cantrill 2020-10-12 17:19:50 UTC
Closing per https://bugzilla.redhat.com/show_bug.cgi?id=1882982#c3  Please reopen if needed


Note You need to log in before you can comment on or make changes to this bug.