Bug 1832343 - [hco deployment] nmstate-handler- pods are in crashloop state after deployment
Summary: [hco deployment] nmstate-handler- pods are in crashloop state after deployment
Keywords:
Status: CLOSED DUPLICATE of bug 1822079
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Networking
Version: 2.4.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 2.4.0
Assignee: Quique Llorente
QA Contact: Meni Yakove
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-05-06 14:53 UTC by Tareq Alayan
Modified: 2020-05-13 10:27 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-13 10:25:37 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Tareq Alayan 2020-05-06 14:53:11 UTC
Description of problem:
After a succesful deplyment of cnv the nmstate-handler pods goes into crashloop state

Version-Release number of selected component (if applicable):
egistry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-hyperconverged-cluster-operator@sha256:1ee979117474bfebd77d7184d57f1c99e259701b3942ecd8201fecd1bd78d47c

registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-kubernetes-nmstate-handler-rhel8@sha256:363c08c91acc99ea2386b9ca01e4d2fad580e58af8dff9e9108d68c652260c57

Server Version: 4.5.0-0.nightly-2020-05-04-113741
Kubernetes Version: v1.18.0-rc.1

How reproducible:
always

Steps to Reproduce:
1. deploy 2.4 CNV
2. check pods oc get pods -nopenshift-cnv
3. oc describe hyperconverged kubevirt-hyperconverged -nopenshift-cnv

Actual results:
 nmstate-handler-fc45f                                 0/1     CrashLoopBackOff   43         5h2m
nmstate-handler-n8bhh                                 0/1     CrashLoopBackOff   43         5h2m

in the logs:
I0506 14:04:19.046314       1 certificate_manager.go:381] Rotating certificates
E0506 14:04:19.065537       1 certificate_manager.go:400] Failed while requesting a signed certificate from the master: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io "csr-hkmlc" is forbidden: user not permitted to approve requests with signerName "kubernetes.io/legacy-unknown"
I0506 14:04:21.100698       1 certificate_manager.go:381] Rotating certificates
E0506 14:04:21.111802       1 certificate_manager.go:400] Failed while requesting a signed certificate from the master: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io "csr-8pjq5" is forbidden: user not permitted to approve requests with signerName "kubernetes.io/legacy-unknown"
I0506 14:04:25.150769       1 certificate_manager.go:381] Rotating certificates
E0506 14:04:25.163283       1 certificate_manager.go:400] Failed while requesting a signed certificate from the master: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io "csr-hxgcg" is forbidden: user not permitted to approve requests with signerName "kubernetes.io/legacy-unknown"
I0506 14:04:33.404221       1 certificate_manager.go:381] Rotating certificates
E0506 14:04:33.415650       1 certificate_manager.go:400] Failed while requesting a signed certificate from the master: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io "csr-6dsxw" is forbidden: user not permitted to approve requests with signerName "kubernetes.io/legacy-unknown"
I0506 14:04:50.240282       1 certificate_manager.go:381] Rotating certificates
E0506 14:04:50.250590       1 certificate_manager.go:400] Failed while requesting a signed certificate from the master: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io "csr-qwxz2" is forbidden: user not permitted to approve requests with signerName "kubernetes.io/legacy-unknown"
E0506 14:04:50.250764       1 certificate_manager.go:290] Reached backoff limit, still unable to rotate certs: timed out waiting for the condition



{"level":"info","ts":1588773848.0096767,"logger":"cmd","msg":"Could not generate and serve custom resource metrics","error":"discovering resource information failed for NMstate in nmstate.io/v1alpha1: &{%!w(string=apiVersion nmstate.io/v1alpha1 and kind NMstate not found available in Kubernetes cluster)}"}
{"level":"info","ts":1588773850.9825263,"logger":"cmd","msg":"Could not create metrics Service","error":"failed to initialize service object for metrics: &{{{%!w(string=) %!w(string=)} {%!w(string=) %!w(string=) %!w(string=) %!w(*int64=<nil>)} %!w(string=Failure) %!w(string=networkaddonsconfigs.networkaddonsoperator.network.kubevirt.io \"cluster\" is forbidden: User \"system:serviceaccount:openshift-cnv:nmstate-handler\" cannot get resource \"networkaddonsconfigs\" in API group \"networkaddonsoperator.network.kubevirt.io\" at the cluster scope) %!w(v1.StatusReason=Forbidden) %!w(*v1.StatusDetails=&{cluster networkaddonsoperator.network.kubevirt.io networkaddonsconfigs  [] 0}) %!w(int32=403)}}"}


oc get  hyperconverged kubevirt-hyperconverged -nopenshift-cnv -oyaml

lastHeartbeatTime: "2020-05-06T14:50:09Z"
    lastTransitionTime: "2020-05-06T14:50:08Z"
    message: 'NetworkAddonsConfig is progressing: DaemonSet "openshift-cnv/nmstate-handler"
      is not available (awaiting 1 nodes)'
    reason: NetworkAddonsConfigProgressing
    status: "True"
    type: Progressing



Expected results:


Additional info:

Comment 1 Petr Horáček 2020-05-13 10:25:37 UTC

*** This bug has been marked as a duplicate of bug 1822079 ***


Note You need to log in before you can comment on or make changes to this bug.