Bug 1781479
| Summary: | The `key` and `ops-key` in secret/curator turn to empty after running for several hours with forwarding logs to external logstore enabled. | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Qiaoling Tang <qitang> |
| Component: | Logging | Assignee: | Jeff Cantrill <jcantril> |
| Status: | CLOSED ERRATA | QA Contact: | Anping Li <anli> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.4 | CC: | aos-bugs |
| Target Milestone: | --- | ||
| Target Release: | 4.4.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-05-04 11:19:30 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Please retest with the lastest 4.4 images. There has been significant work to managing certificates in the operator which may have resolve this. I'm not able to reproduce this issue with ose-cluster-logging-operator-v4.4.0-202002022216, move this bug to VERIFIED. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581 |
Description of problem: The `key` and `ops-key` in the secret/curator turn to empty after running for 4h20m with forwarding logs to external logstore enabled, other secrets were not changed. The CLO was repeating 'level=info msg="Updating status of Curator"'. All the error messages in the CLO pod are in the below: $ oc logs cluster-logging-operator-58dccd9db-kvwjm |grep error time="2019-12-09T03:01:01Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T03:02:21Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" {"level":"error","ts":1575860714.0120184,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"logforwarding-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on logforwardings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88"} time="2019-12-09T03:05:17Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T03:10:15Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" {"level":"error","ts":1575861276.8150146,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"logforwarding-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on logforwardings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88"} time="2019-12-09T03:16:01Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T03:21:41Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T03:26:45Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T03:34:25Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T03:42:41Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T03:42:44Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T03:42:45Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T03:48:26Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T03:56:20Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T03:58:16Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T03:58:17Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T03:58:18Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T04:06:45Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T04:06:48Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T04:06:49Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T04:15:31Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T04:21:25Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T04:24:23Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T04:29:41Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T04:29:41Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T04:39:13Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T04:39:14Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T04:39:15Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T04:44:29Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T04:44:30Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T04:44:31Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T04:50:50Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T04:50:51Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T04:58:03Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T04:58:04Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T05:12:19Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T05:21:04Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T05:26:19Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T05:32:30Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T05:32:33Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T05:39:35Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T05:39:38Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T05:39:39Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T05:53:39Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T05:53:40Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T05:53:41Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T06:00:14Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T06:00:15Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T06:06:48Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T06:06:49Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" {"level":"error","ts":1575871655.6114109,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"logforwarding-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on logforwardings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88"} time="2019-12-09T06:07:38Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T06:14:55Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T06:14:56Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T06:14:57Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T06:19:51Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T06:20:44Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" {"level":"error","ts":1575872525.808817,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"logforwarding-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on logforwardings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88"} time="2019-12-09T06:29:02Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" {"level":"error","ts":1575872996.0121827,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"logforwarding-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on logforwardings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88"} time="2019-12-09T06:29:59Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T06:35:01Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" {"level":"error","ts":1575873372.409862,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"logforwarding-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on logforwardings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88"} time="2019-12-09T06:37:42Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T06:37:44Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" {"level":"error","ts":1575873501.62436,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"logforwarding-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on logforwardings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88"} {"level":"error","ts":1575873754.6148715,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"logforwarding-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on logforwardings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88"} time="2019-12-09T06:42:37Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T06:46:16Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T06:52:08Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T06:52:11Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" {"level":"error","ts":1575874591.614392,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"logforwarding-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on logforwardings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88"} time="2019-12-09T06:56:34Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T06:59:28Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T06:59:29Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T06:59:32Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T06:59:33Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T07:03:23Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T07:04:38Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T07:06:30Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T07:13:10Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T07:13:11Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T07:19:02Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T07:19:58Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T07:19:59Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T07:22:01Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T07:22:04Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T07:22:19Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T07:26:47Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T07:27:58Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T07:35:24Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T07:35:26Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T07:37:09Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T07:45:17Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T07:45:19Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" {"level":"error","ts":1575877685.0138645,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"logforwarding-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on logforwardings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88"} {"level":"error","ts":1575878618.6158159,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"logforwarding-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on logforwardings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88"} time="2019-12-09T08:03:41Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T08:10:20Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T08:18:51Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T08:18:52Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T08:24:04Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" time="2019-12-09T08:33:00Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again" Version-Release number of selected component (if applicable): ose-cluster-logging-operator-v4.4.0-201912080435 How reproducible: Not sure, only saw this issue twice. Steps to Reproduce: 1.deploy clusterlogging 2.enable logforwarding to forward logs to external elasticsearch as unsecure 3.several hours later, check the curator status, the curator pod was in Error status Actual results: Expected results: Additional info: