Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2107001

Summary: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden
Product: OpenShift Container Platform Reporter: Germano Veit Michel <gveitmic>
Component: NetworkingAssignee: Ben Nemec <bnemec>
Networking sub component: kubernetes-nmstate-operator QA Contact: Aleksandra Malykhin <amalykhi>
Status: CLOSED DUPLICATE Docs Contact:
Severity: medium    
Priority: unspecified CC: phoracek, pibanezr
Version: 4.10   
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-07-18 01:22:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Germano Veit Michel 2022-07-14 05:47:11 UTC 
Description of problem:

Permission denied when the operator tries to get a list of Nodes in the cluster.

I0714 05:09:29.356331 1 request.go:665] Waited for 1.04917251s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/imageregistry.operator.openshift.io/v1?timeout=32s
{"level":"info","ts":"2022-07-14T05:09:30.759Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2022-07-14T05:09:30.759Z","logger":"controller.nmstate","msg":"Starting EventSource","reconciler group":"nmstate.io","reconciler kind":"NMState","source":"kind source: *v1.NMState"}
{"level":"info","ts":"2022-07-14T05:09:30.759Z","logger":"controller.nmstate","msg":"Starting Controller","reconciler group":"nmstate.io","reconciler kind":"NMState"}
{"level":"info","ts":"2022-07-14T05:09:30.860Z","logger":"controller.nmstate","msg":"Starting workers","reconciler group":"nmstate.io","reconciler kind":"NMState","worker count":1}
2022/07/14 05:27:22 reconciling (apiextensions.k8s.io/v1, Kind=CustomResourceDefinition) /nodenetworkconfigurationenactments.nmstate.io
2022/07/14 05:27:22 does not exist, creating (apiextensions.k8s.io/v1, Kind=CustomResourceDefinition) /nodenetworkconfigurationenactments.nmstate.io
2022/07/14 05:27:22 successfully created (apiextensions.k8s.io/v1, Kind=CustomResourceDefinition) /nodenetworkconfigurationenactments.nmstate.io
2022/07/14 05:27:22 reconciling (apiextensions.k8s.io/v1, Kind=CustomResourceDefinition) /nodenetworkconfigurationpolicies.nmstate.io
2022/07/14 05:27:22 does not exist, creating (apiextensions.k8s.io/v1, Kind=CustomResourceDefinition) /nodenetworkconfigurationpolicies.nmstate.io
2022/07/14 05:27:22 successfully created (apiextensions.k8s.io/v1, Kind=CustomResourceDefinition) /nodenetworkconfigurationpolicies.nmstate.io
2022/07/14 05:27:22 reconciling (apiextensions.k8s.io/v1, Kind=CustomResourceDefinition) /nodenetworkstates.nmstate.io
2022/07/14 05:27:22 does not exist, creating (apiextensions.k8s.io/v1, Kind=CustomResourceDefinition) /nodenetworkstates.nmstate.io
2022/07/14 05:27:22 successfully created (apiextensions.k8s.io/v1, Kind=CustomResourceDefinition) /nodenetworkstates.nmstate.io
2022/07/14 05:27:22 reconciling (/v1, Kind=Namespace) /openshift-nmstate
2022/07/14 05:27:22 update was successful
2022/07/14 05:27:22 reconciling (rbac.authorization.k8s.io/v1, Kind=Role) openshift-nmstate/nmstate-handler
2022/07/14 05:27:22 does not exist, creating (rbac.authorization.k8s.io/v1, Kind=Role) openshift-nmstate/nmstate-handler
2022/07/14 05:27:22 successfully created (rbac.authorization.k8s.io/v1, Kind=Role) openshift-nmstate/nmstate-handler
2022/07/14 05:27:22 reconciling (rbac.authorization.k8s.io/v1, Kind=ClusterRole) openshift-nmstate/nmstate-handler
2022/07/14 05:27:22 does not exist, creating (rbac.authorization.k8s.io/v1, Kind=ClusterRole) openshift-nmstate/nmstate-handler
2022/07/14 05:27:22 successfully created (rbac.authorization.k8s.io/v1, Kind=ClusterRole) openshift-nmstate/nmstate-handler
2022/07/14 05:27:22 reconciling (rbac.authorization.k8s.io/v1, Kind=RoleBinding) openshift-nmstate/nmstate-handler
2022/07/14 05:27:23 does not exist, creating (rbac.authorization.k8s.io/v1, Kind=RoleBinding) openshift-nmstate/nmstate-handler
2022/07/14 05:27:23 successfully created (rbac.authorization.k8s.io/v1, Kind=RoleBinding) openshift-nmstate/nmstate-handler
2022/07/14 05:27:23 reconciling (rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding) openshift-nmstate/nmstate-handler
2022/07/14 05:27:23 does not exist, creating (rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding) openshift-nmstate/nmstate-handler
2022/07/14 05:27:23 successfully created (rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding) openshift-nmstate/nmstate-handler
2022/07/14 05:27:23 reconciling (/v1, Kind=ServiceAccount) openshift-nmstate/nmstate-handler
2022/07/14 05:27:23 update was successful
E0714 05:27:23.120634 1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:openshift-nmstate:nmstate-operator" cannot list resource "nodes" in API group "" at the cluster scope
E0714 05:27:24.663696 1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:openshift-nmstate:nmstate-operator" cannot list resource "nodes" in API group "" at the cluster scope
E0714 05:27:27.287152 1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:openshift-nmstate:nmstate-operator" cannot list resource "nodes" in API group "" at the cluster scope
E0714 05:27:30.630141 1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:openshift-nmstate:nmstate-operator" cannot list resource "nodes" in API group "" at the cluster scope
E0714 05:27:37.638912 1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:openshift-nmstate:nmstate-operator" cannot list resource "nodes" in API group "" at the cluster scope
E0714 05:27:52.861439 1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:openshift-nmstate:nmstate-operator" cannot list resource "nodes" in API group "" at the cluster scope
E0714 05:28:24.411581 1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:openshift-nmstate:nmstate-operator" cannot list resource "nodes" in API group "" at the cluster scope
E0714 05:29:22.017030 1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:openshift-nmstate:nmstate-operator" cannot list resource "nodes" in API group "" at the cluster scope

Version-Release number of selected component (if applicable):
OCP 4.10.9 (also reproduced on 4.10.20)
CNV 4.10.2
kubernetes-nmstate-operator.4.10.0-202207041436

If I manually add "Nodes" on the Role of nmstate operator, all works.

How reproducible:
Always

Steps to Reproduce:
1. Install OCP fresh
2. Install NMstate Operator
   Install OCP Virtualization Operator and create Hyperconverged (will create NMstate)
3. See nmstate operator pod logs

Actual results:
* No nmstate handler pods
* nmstate operator cannot get list of nodes

Expected results:
* nmstate working
Comment 2 Aleksandra Malykhin 2022-07-17 09:51:02 UTC 
The duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=2104938
Comment 3 Germano Veit Michel 2022-07-18 01:22:52 UTC 
(In reply to Aleksandra Malykhin from comment #2)
> The duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=2104938

It is indeed, hopefully setting as dup will migrate the KCS and customer case.

Thank you.

*** This bug has been marked as a duplicate of bug 2104938 ***
Comment 4 Aleksandra Malykhin 2022-07-18 05:51:16 UTC 
FYI
Verified  with Kubernetes NMState Operator   4.10.0-202207140916

All pods are started, no "*v1.Node: nodes is forbidden" errors in the log
$ oc get pods
NAME                                    READY   STATUS    RESTARTS   AGE
nmstate-cert-manager-8587fbb95d-wlvlw   1/1     Running   0          3m18s
nmstate-handler-476q8                   1/1     Running   0          3m18s
nmstate-handler-4xngk                   1/1     Running   0          3m18s
nmstate-handler-8xh6q                   1/1     Running   0          3m18s
nmstate-handler-dt667                   1/1     Running   0          3m18s
nmstate-handler-ghqmz                   1/1     Running   0          3m18s
nmstate-operator-5fd746bd5-msv8j        1/1     Running   0          3m41s
nmstate-webhook-8c4fdfcdb-qbdl8         1/1     Running   0          3m18s
nmstate-webhook-8c4fdfcdb-sdr7v         1/1     Running   0          3m18s