Bug 1817097 - Failed to create or get service for metrics: services is forbidden: cannot set blockOwnerDeletion in cluster-logging-operator
Summary: Failed to create or get service for metrics: services is forbidden: cannot se...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 4.3.z
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.5.0
Assignee: Periklis Tsirakidis
QA Contact: Qiaoling Tang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-03-25 14:59 UTC by Oscar Casal Sanchez
Modified: 2020-07-13 17:24 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-07-13 17:23:43 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift cluster-logging-operator pull 492 None closed Bug 1817097: Grant perm to set owner ref to metrics Service resource 2020-12-02 00:42:35 UTC
Red Hat Knowledge Base (Solution) 5085891 None None None 2020-05-18 13:12:46 UTC
Red Hat Product Errata RHBA-2020:2409 None None None 2020-07-13 17:23:59 UTC

Description Oscar Casal Sanchez 2020-03-25 14:59:25 UTC
[Description of problem]

Deploying cluster logging following the documentation [1] step by step, in the cluster-logging-operator pod and the elasticsearch-operator pod is possible to read the next error:

Elasticsearch-operator:

~~~
$ oc -n openshift-operators-redhat logs elasticsearch-operator-59d8d6b6cc-d7zxh |grep blockOwnerDeletion 
{"level":"info","ts":1585135974.742253,"logger":"cmd","msg":"failed to create or get service for metrics: services \"elasticsearch-operator\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>"}
~~~

Cluster-logging-operator:

~~~
$ oc -n openshift-logging logs cluster-logging-operator-74886fbcf9-jtgjb |grep blockOwnerDeletion
{"level":"info","ts":1585134774.0195408,"logger":"cmd","msg":"failed to create or get service for metrics: services \"cluster-logging-operator\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>"}
~~~

[Version-Release number of selected component (if applicable)]

$ oc version
Client Version: 4.3.5
Server Version: 4.3.5
Kubernetes Version: v1.16.2

[How reproducible]
Always

[Steps to Reproduce]
Follow the Deploying cluster logging documentation step by step [1]

[Actual results]
Elasticsearch-operator and cluster-logging-operator give the error:


Elasticsearch-operator:

~~~
$ oc -n openshift-operators-redhat logs elasticsearch-operator-59d8d6b6cc-d7zxh |grep blockOwnerDeletion 
{"level":"info","ts":1585135974.742253,"logger":"cmd","msg":"failed to create or get service for metrics: services \"elasticsearch-operator\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>"}
~~~

Cluster-logging-operator:

~~~
$ oc -n openshift-logging logs cluster-logging-operator-74886fbcf9-jtgjb |grep blockOwnerDeletion
{"level":"info","ts":1585134774.0195408,"logger":"cmd","msg":"failed to create or get service for metrics: services \"cluster-logging-operator\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>"}
~~~


[Expected results]

Not error showed.



[1] https://docs.openshift.com/container-platform/4.3/logging/cluster-logging-deploying.html

Comment 5 Qiaoling Tang 2020-04-24 06:05:45 UTC
There still have these error logs in the CLO and EO pod.

Testing images are from 4.5.0-0.ci-2020-04-24-012839

$ oc logs -n openshift-operators-redhat elasticsearch-operator-b44d4f876-s68kl
{"level":"info","ts":1587707915.5165293,"logger":"cmd","msg":"Go Version: go1.13.8"}
{"level":"info","ts":1587707915.5165737,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1587707915.516579,"logger":"cmd","msg":"Version of operator-sdk: v0.8.2"}
{"level":"info","ts":1587707915.516989,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1587707915.742507,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1587707915.7529628,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1587707916.0702565,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1587707916.0712004,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"kibana-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1587707916.071423,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"elasticsearch-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1587707916.0716872,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"proxyconfig-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1587707916.0718582,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"kibanasecret-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1587707916.0721056,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"trustedcabundle-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1587707916.3842432,"logger":"cmd","msg":"failed to create or get service for metrics: services \"elasticsearch-operator\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>"}
{"level":"info","ts":1587707916.3842745,"logger":"cmd","msg":"Starting the Cmd."}

$ oc logs cluster-logging-operator-6fff647b89-pbg4h
{"level":"info","ts":1587707938.0083652,"logger":"cmd","msg":"Go Version: go1.12.16"}
{"level":"info","ts":1587707938.0083916,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1587707938.0084045,"logger":"cmd","msg":"Version of operator-sdk: v0.8.2"}
{"level":"info","ts":1587707938.008979,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1587707938.1408584,"logger":"leader","msg":"Found existing lock","LockOwner":"cluster-logging-operator-7fb9b7c876-njjjp"}
{"level":"info","ts":1587707938.1547015,"logger":"leader","msg":"Not the leader. Waiting."}
{"level":"info","ts":1587707939.3150458,"logger":"leader","msg":"Not the leader. Waiting."}
{"level":"info","ts":1587707941.4651976,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1587707941.5699837,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1587707941.5705988,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"clusterlogging-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1587707941.5707474,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"logforwarding-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1587707941.570857,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"collector-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1587707941.57104,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"proxyconfig-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1587707941.571199,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"trustedcabundle-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1587707941.813846,"logger":"cmd","msg":"failed to create or get service for metrics: services \"cluster-logging-operator\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>"}
{"level":"info","ts":1587707941.8138752,"logger":"cmd","msg":"Starting the Cmd."}

Comment 6 Qiaoling Tang 2020-04-24 06:30:48 UTC
(In reply to Qiaoling Tang from comment #5)
> There still have these error logs in the CLO and EO pod.

I forgot to update the manifests when I testing this issue. The issue in CLO is fixed, but it still could be reproduced in EO.

> Testing images are from 4.5.0-0.ci-2020-04-24-012839
> 
> $ oc logs -n openshift-operators-redhat
> elasticsearch-operator-b44d4f876-s68kl
> {"level":"info","ts":1587707915.5165293,"logger":"cmd","msg":"Go Version:
> go1.13.8"}
> {"level":"info","ts":1587707915.5165737,"logger":"cmd","msg":"Go OS/Arch:
> linux/amd64"}
> {"level":"info","ts":1587707915.516579,"logger":"cmd","msg":"Version of
> operator-sdk: v0.8.2"}
> {"level":"info","ts":1587707915.516989,"logger":"leader","msg":"Trying to
> become the leader."}
> {"level":"info","ts":1587707915.742507,"logger":"leader","msg":"No
> pre-existing lock was found."}
> {"level":"info","ts":1587707915.7529628,"logger":"leader","msg":"Became the
> leader."}
> {"level":"info","ts":1587707916.0702565,"logger":"cmd","msg":"Registering
> Components."}
> {"level":"info","ts":1587707916.0712004,"logger":"kubebuilder.controller",
> "msg":"Starting EventSource","controller":"kibana-controller","source":"kind
> source: /, Kind="}
> {"level":"info","ts":1587707916.071423,"logger":"kubebuilder.controller",
> "msg":"Starting
> EventSource","controller":"elasticsearch-controller","source":"kind source:
> /, Kind="}
> {"level":"info","ts":1587707916.0716872,"logger":"kubebuilder.controller",
> "msg":"Starting
> EventSource","controller":"proxyconfig-controller","source":"kind source: /,
> Kind="}
> {"level":"info","ts":1587707916.0718582,"logger":"kubebuilder.controller",
> "msg":"Starting
> EventSource","controller":"kibanasecret-controller","source":"kind source:
> /, Kind="}
> {"level":"info","ts":1587707916.0721056,"logger":"kubebuilder.controller",
> "msg":"Starting
> EventSource","controller":"trustedcabundle-controller","source":"kind
> source: /, Kind="}
> {"level":"info","ts":1587707916.3842432,"logger":"cmd","msg":"failed to
> create or get service for metrics: services \"elasticsearch-operator\" is
> forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a
> resource you can't set finalizers on: , <nil>"}
> {"level":"info","ts":1587707916.3842745,"logger":"cmd","msg":"Starting the
> Cmd."}
>

Comment 7 Periklis Tsirakidis 2020-04-24 08:26:58 UTC
@Qiaoling Tang 

TBH, I had to split the issue for EO in a separate BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1827063

This makes backporting and BZ-depenendecies easier for us. Please reconsider to put this BUG to VERIFIED for CLO.

Comment 8 Qiaoling Tang 2020-04-24 08:52:09 UTC
Sorry, I didn't noticed that.

Per comment 6 and comment 7, move this bug to VERIFIED.

Comment 10 errata-xmlrpc 2020-07-13 17:23:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.