Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2104938

Summary: The kubernetes nmstate pods didn't start after instance creation
Product: OpenShift Container Platform Reporter: Aleksandra Malykhin <amalykhi>
Component: NetworkingAssignee: Christoph Stäbler <cstabler>
Networking sub component: kubernetes-nmstate-operator QA Contact: Aleksandra Malykhin <amalykhi>
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: high CC: cstabler, gveitmic, jgato, pablo.iranzo, sasha
Version: 4.10Keywords: Regression
Target Milestone: ---   
Target Release: 4.10.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2106634 (view as bug list) Environment:
Last Closed: 2022-07-20 07:46:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2106634    
Bug Blocks: 2067081, 2092034, 2103586    

Description Aleksandra Malykhin 2022-07-07 14:07:37 UTC 
Description of problem:
The kubernates nmstate pods didn't start after instance creation 

Version-Release number of selected component (if applicable):
Cluster version is 4.10.22

How reproducible:


Steps to Reproduce:
1.  install nmstate operator via CLI or UI
2.  check csv is in Succeeded state
[root@sealusa25 ~]# oc get csv
NAME                                              DISPLAY                       VERSION               REPLACES   PHASE
kubernetes-nmstate-operator.4.10.0-202207041436   Kubernetes NMState Operator   4.10.0-202207041436              Succeeded

3.  create nmstate instance and wait all pods start running

Actual results:
the pods are not started
[root@sealusa25 ~]# oc get pods
NAME                                READY   STATUS    RESTARTS   AGE
nmstate-operator-6bff67d4fb-d55hg   1/1     Running   0          9m44s


Expected results:
All required pods are started


Additional info:
[root@sealusa25 ~]# oc logs nmstate-operator-6bff67d4fb-d55hg

E0707 14:08:07.854061       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:openshift-nmstate:nmstate-operator" cannot list resource "nodes" in API group "" at the cluster scope
Comment 1 Aleksandra Malykhin 2022-07-07 14:14:43 UTC 
must gather http://rhos-compute-node-10.lab.eng.rdu2.redhat.com/logs/BZ_2104938.tar.gz
Comment 2 Aleksandra Malykhin 2022-07-10 10:02:54 UTC 
the logs in the comment #1 are from sno cluster

the following logs from regular cluster:
Cluster version is 4.10.0-0.nightly-2022-06-08-150219
kubernetes-nmstate-operator.4.10.0-202207041436



ClusterID: 85f891fe-7c47-4be4-a03c-40f4c98791d3
ClusterVersion: Stable at "4.10.0-0.nightly-2022-06-08-150219"
ClusterOperators:
	All healthy and stable

http://rhos-compute-node-10.lab.eng.rdu2.redhat.com/logs/BZ_2104938_2_must-gather.tar.gz
Comment 3 Aleksandra Malykhin 2022-07-10 10:03:46 UTC 
The pods started successfully with the previous knmstate build:
 Kubernetes NMState Operator   4.10.0-202206270836
Comment 4 Ben Nemec 2022-07-13 17:03:01 UTC 
*** Bug 2106840 has been marked as a duplicate of this bug. ***
Comment 7 Germano Veit Michel 2022-07-18 01:22:52 UTC 
*** Bug 2107001 has been marked as a duplicate of this bug. ***
Comment 8 Aleksandra Malykhin 2022-07-18 05:50:54 UTC 
Verified  with Kubernetes NMState Operator   4.10.0-202207140916

All pods are started, no "*v1.Node: nodes is forbidden" errors in the log
$ oc get pods
NAME                                    READY   STATUS    RESTARTS   AGE
nmstate-cert-manager-8587fbb95d-wlvlw   1/1     Running   0          3m18s
nmstate-handler-476q8                   1/1     Running   0          3m18s
nmstate-handler-4xngk                   1/1     Running   0          3m18s
nmstate-handler-8xh6q                   1/1     Running   0          3m18s
nmstate-handler-dt667                   1/1     Running   0          3m18s
nmstate-handler-ghqmz                   1/1     Running   0          3m18s
nmstate-operator-5fd746bd5-msv8j        1/1     Running   0          3m41s
nmstate-webhook-8c4fdfcdb-qbdl8         1/1     Running   0          3m18s
nmstate-webhook-8c4fdfcdb-sdr7v         1/1     Running   0          3m18s
Comment 10 errata-xmlrpc 2022-07-20 07:46:12 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.10.23 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:5568
Comment 11 Jose Gato 2022-07-27 10:16:49 UTC 
Hi there,

this new version also fix this bug:

https://bugzilla.redhat.com/show_bug.cgi?id=2102682

similar, but needing other perm.