Bug 1781479 - The `key` and `ops-key` in secret/curator turn to empty after running for several hours with forwarding logs to external logstore enabled.
Summary: The `key` and `ops-key` in secret/curator turn to empty after running for se...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 4.4
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.4.0
Assignee: Jeff Cantrill
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-12-10 06:10 UTC by Qiaoling Tang
Modified: 2020-05-04 11:20 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-04 11:19:30 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:0581 0 None None None 2020-05-04 11:20:10 UTC

Description Qiaoling Tang 2019-12-10 06:10:13 UTC
Description of problem:

The `key` and `ops-key` in the secret/curator turn to empty after running for 4h20m with forwarding logs to external logstore enabled, other secrets were not changed. The CLO was repeating 'level=info msg="Updating status of Curator"'. All the error messages in the CLO pod are in the below:

$ oc logs cluster-logging-operator-58dccd9db-kvwjm |grep error
time="2019-12-09T03:01:01Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T03:02:21Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
{"level":"error","ts":1575860714.0120184,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"logforwarding-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on logforwardings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
time="2019-12-09T03:05:17Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T03:10:15Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
{"level":"error","ts":1575861276.8150146,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"logforwarding-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on logforwardings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
time="2019-12-09T03:16:01Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T03:21:41Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T03:26:45Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T03:34:25Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T03:42:41Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T03:42:44Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T03:42:45Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T03:48:26Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T03:56:20Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T03:58:16Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T03:58:17Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T03:58:18Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T04:06:45Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T04:06:48Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T04:06:49Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T04:15:31Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T04:21:25Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T04:24:23Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T04:29:41Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T04:29:41Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T04:39:13Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T04:39:14Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T04:39:15Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T04:44:29Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T04:44:30Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T04:44:31Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T04:50:50Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T04:50:51Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T04:58:03Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T04:58:04Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T05:12:19Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T05:21:04Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T05:26:19Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T05:32:30Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T05:32:33Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T05:39:35Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T05:39:38Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T05:39:39Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T05:53:39Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T05:53:40Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T05:53:41Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T06:00:14Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T06:00:15Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T06:06:48Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T06:06:49Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
{"level":"error","ts":1575871655.6114109,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"logforwarding-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on logforwardings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
time="2019-12-09T06:07:38Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T06:14:55Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T06:14:56Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T06:14:57Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T06:19:51Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T06:20:44Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
{"level":"error","ts":1575872525.808817,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"logforwarding-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on logforwardings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
time="2019-12-09T06:29:02Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
{"level":"error","ts":1575872996.0121827,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"logforwarding-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on logforwardings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
time="2019-12-09T06:29:59Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T06:35:01Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
{"level":"error","ts":1575873372.409862,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"logforwarding-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on logforwardings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
time="2019-12-09T06:37:42Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T06:37:44Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
{"level":"error","ts":1575873501.62436,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"logforwarding-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on logforwardings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1575873754.6148715,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"logforwarding-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on logforwardings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
time="2019-12-09T06:42:37Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T06:46:16Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T06:52:08Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T06:52:11Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
{"level":"error","ts":1575874591.614392,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"logforwarding-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on logforwardings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
time="2019-12-09T06:56:34Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T06:59:28Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T06:59:29Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T06:59:32Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T06:59:33Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T07:03:23Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T07:04:38Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T07:06:30Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T07:13:10Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T07:13:11Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T07:19:02Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T07:19:58Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T07:19:59Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T07:22:01Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T07:22:04Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T07:22:19Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T07:26:47Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T07:27:58Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T07:35:24Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T07:35:26Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T07:37:09Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T07:45:17Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T07:45:19Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
{"level":"error","ts":1575877685.0138645,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"logforwarding-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on logforwardings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1575878618.6158159,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"logforwarding-controller","request":"openshift-logging/instance","error":"Operation cannot be fulfilled on logforwardings.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
time="2019-12-09T08:03:41Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T08:10:20Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T08:18:51Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T08:18:52Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T08:24:04Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-12-09T08:33:00Z" level=error msg="Error updating &TypeMeta{Kind:Secret,APIVersion:v1,}: Operation cannot be fulfilled on secrets \"kibana-proxy\": the object has been modified; please apply your changes to the latest version and try again"

Version-Release number of selected component (if applicable):
ose-cluster-logging-operator-v4.4.0-201912080435

How reproducible:
Not sure, only saw this issue twice.

Steps to Reproduce:
1.deploy clusterlogging
2.enable logforwarding to forward logs to external elasticsearch as unsecure
3.several hours later, check the curator status, the curator pod was in Error status

Actual results:


Expected results:


Additional info:

Comment 3 Jeff Cantrill 2020-01-29 21:54:12 UTC
Please retest with the lastest 4.4 images.  There has been significant work to managing certificates in the operator which may have resolve this.

Comment 4 Qiaoling Tang 2020-02-05 00:14:32 UTC
I'm not able to reproduce this issue with ose-cluster-logging-operator-v4.4.0-202002022216, move this bug to VERIFIED.

Comment 6 errata-xmlrpc 2020-05-04 11:19:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.