Description of problem: Deploy logging, then check the CLO logs, it keeps repeating some error messages: time="2020-09-27T07:22:35Z" level=error msg="clusterlogging-controller error updating status: Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again" {"level":"error","ts":1601191355.1462588,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"clusterlogging-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90"} time="2020-09-27T07:23:08Z" level=error msg="clusterlogging-controller error updating status: Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again" {"level":"error","ts":1601191388.7426267,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"clusterlogging-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on clusterloggings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90"} $ oc get cl instance -oyaml apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: creationTimestamp: "2020-09-27T06:56:26Z" generation: 1 name: instance namespace: openshift-logging resourceVersion: "275945" selfLink: /apis/logging.openshift.io/v1/namespaces/openshift-logging/clusterloggings/instance uid: 9dbd7b5f-b9ea-489d-ad35-1bf01d698ac1 spec: collection: logs: fluentd: {} type: fluentd curation: curator: schedule: '*/10 * * * *' type: curator logStore: elasticsearch: nodeCount: 3 redundancyPolicy: SingleRedundancy resources: requests: memory: 2Gi type: elasticsearch managementState: Managed visualization: kibana: replicas: 1 type: kibana status: clusterConditions: - lastTransitionTime: "2020-09-27T06:57:04Z" status: "False" type: CollectorDeadEnd collection: logs: fluentdStatus: daemonSet: fluentd nodes: fluentd-2lprk: juzhao-share2-r6v5v-compute-0 fluentd-c6mh4: juzhao-share2-r6v5v-control-plane-2 fluentd-c9lzb: juzhao-share2-r6v5v-compute-2 fluentd-h2gsv: juzhao-share2-r6v5v-rhel-2 fluentd-hggd9: juzhao-share2-r6v5v-compute-1 fluentd-mmwx8: juzhao-share2-r6v5v-control-plane-0 fluentd-sv7hj: juzhao-share2-r6v5v-rhel-0 fluentd-txgch: juzhao-share2-r6v5v-control-plane-1 fluentd-z9pff: juzhao-share2-r6v5v-rhel-1 pods: failed: [] notReady: [] ready: - fluentd-2lprk - fluentd-c6mh4 - fluentd-c9lzb - fluentd-h2gsv - fluentd-hggd9 - fluentd-mmwx8 - fluentd-sv7hj - fluentd-txgch - fluentd-z9pff curation: curatorStatus: - clusterCondition: curator-1601192400-92lkp: - lastTransitionTime: "2020-09-27T07:40:03Z" reason: Completed status: "True" type: ContainerTerminated cronJobs: curator schedules: '*/10 * * * *' suspended: false logStore: elasticsearchStatus: - cluster: activePrimaryShards: 12 activeShards: 24 initializingShards: 0 numDataNodes: 3 numNodes: 3 pendingTasks: 0 relocatingShards: 0 status: green unassignedShards: 0 clusterName: elasticsearch nodeConditions: elasticsearch-cdm-ppb0i0fs-1: [] elasticsearch-cdm-ppb0i0fs-2: [] elasticsearch-cdm-ppb0i0fs-3: [] nodeCount: 3 pods: client: failed: [] notReady: [] ready: - elasticsearch-cdm-ppb0i0fs-1-5dfdf8fbf7-mm92h - elasticsearch-cdm-ppb0i0fs-2-5d54cfd649-d2rwr - elasticsearch-cdm-ppb0i0fs-3-754666fffc-59dcl data: failed: [] notReady: [] ready: - elasticsearch-cdm-ppb0i0fs-1-5dfdf8fbf7-mm92h - elasticsearch-cdm-ppb0i0fs-2-5d54cfd649-d2rwr - elasticsearch-cdm-ppb0i0fs-3-754666fffc-59dcl master: failed: [] notReady: [] ready: - elasticsearch-cdm-ppb0i0fs-1-5dfdf8fbf7-mm92h - elasticsearch-cdm-ppb0i0fs-2-5d54cfd649-d2rwr - elasticsearch-cdm-ppb0i0fs-3-754666fffc-59dcl shardAllocationEnabled: all visualization: kibanaStatus: - deployment: kibana pods: failed: [] notReady: [] ready: - kibana-76b76456ff-zfg2w replicaSets: - kibana-76b76456ff replicas: 1 Version-Release number of selected component (if applicable): clusterlogging.4.6.0-202009240527.p0 elasticsearch-operator.4.6.0-202009232303.p0 How reproducible: Always Steps to Reproduce: 1. deploy logging 2. check CLO pod log 3. Actual results: Expected results: Don't have these logs in the CLO. Additional info:
This error is returned by k8s , and CLO will be invoked again after 30 secs. Does this log statement appear repeatedly? As per [1] this is a benign error and can be ignored, and perhaps it should be a warning. [1] https://github.com/kubernetes/kubernetes/issues/28149#issuecomment-229469157
Setting to low as this is a non-blocking 4.6 issue
(In reply to Vimal Kumar from comment #1) > This error is returned by k8s , and CLO will be invoked again after 30 secs. > Does this log statement appear repeatedly? > Seems different cluster has different behavior. This log statement appeared repeatedly when I reported this bz, but I checked the logs again in a cluster I launched today, it stopped appearing after running for 10 minutes.
Moving to 4.7 to satisfy CF requirements for 4.6
Closing per https://bugzilla.redhat.com/show_bug.cgi?id=1882982#c3 Please reopen if needed