Created attachment 1699083 [details] openshift-cnv kubemacpool-mac-controller-manager-f9f978598-6cszk.log Description of problem: $ oc get pod -n openshift-cnv kubemacpool-mac-controller-manager-f9f978598-6cszk 0/1 Running 27 2d8h kubemacpool-mac-controller-manager-f9f978598-wpl6c 0/1 Running 27 2d8h $ oc get -n openshift-cnv hyperconverged -o=yaml - lastHeartbeatTime: "2020-06-29T07:23:29Z" lastTransitionTime: "2020-06-26T23:11:50Z" message: 'NetworkAddonsConfig is progressing: Deployment "openshift-cnv/kubemacpool-mac-controller-manager" is not available (awaiting 2 nodes)' reason: NetworkAddonsConfigProgressing status: "False" type: Upgradeable $ oc logs -n openshift-cnv kubemacpool-mac-controller-manager-f9f978598-6cszk E0629 07:09:51.014082 1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key {"level":"error","ts":1593414591.015211,"logger":"manager","msg":"unable to run the manager","error":"failed watting for ready TLS key/cert: timed out waiting for the condition","errorVerbose":"timed out waiting for the condition\nfailed watting for ready TLS key/cert\ngithub.com/qinqon/kube-admission-webhook/pkg/webhook/server.(*Server).Start\n\t/go/src/github.com/k8snetworkplumbingwg/kubemacpool/vendor/github.com/qinqon/kube-admission-webhook/pkg/webhook/server/server.go:159\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startNonLeaderElectionRunnables.func1\n\t/go/src/github.com/k8snetworkplumbingwg/kubemacpool/vendor/sigs.k8s.io/controller-runtime/pkg/manager/internal.go:492\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1357","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/k8snetworkplumbingwg/kubemacpool/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/k8snetworkplumbingwg/kubemacpool/pkg/manager.(*KubeMacPoolManager).Run\n\t/go/src/github.com/k8snetworkplumbingwg/kubemacpool/pkg/manager/manager.go:125\nmain.main\n\t/go/src/github.com/k8snetworkplumbingwg/kubemacpool/cmd/manager/main.go:88\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:203"} {"level":"info","ts":1593414591.0157564,"logger":"manager","msg":"Setting up Manager"} I0629 07:09:52.066415 1 request.go:621] Throttling request took 1.043446451s, request: GET:https://172.30.0.1:443/apis/imageregistry.operator.openshift.io/v1?timeout=32s {"level":"info","ts":1593414593.7263834,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"} {"level":"info","ts":1593414593.731542,"logger":"manager","msg":"Setting up controllers"} {"level":"info","ts":1593414593.7317173,"logger":"manager","msg":"Setting up webhooks"} {"level":"info","ts":1593414593.731812,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/readyz"} {"level":"info","ts":1593414593.7318296,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-pods"} {"level":"info","ts":1593414593.7318873,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-virtualmachines"} I0629 07:09:53.732028 1 leaderelection.go:242] attempting to acquire leader lease openshift-cnv/kubemacpool-election... {"level":"info","ts":1593414593.732924,"logger":"webhook/server","msg":"Starting nodenetworkconfigurationpolicy webhook server"} E0629 07:09:53.733103 1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key {"level":"info","ts":1593414593.7329278,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"} Version-Release number of selected component (if applicable): OCP-4.6 HCO-v2.3.0-411 How reproducible: 100 Steps to Reproduce: 1. Deploy CNV 2. 3. Actual results: kubemacpool-mac-controller-manager failing to start Expected results: CNV deployed successfuly Additional info:
We did not see this issue at least for a month . Moving it to Verified then.
I have the same issue with a CNV 2.4.1 deployment on OCP 4.5.11. Maybe it is other issue but this is the only BZ that I found that mention this : # oc logs kubemacpool-mac-controller-manager-6f9c447bbd-f92v4 {"level":"info","ts":1601553984.2422936,"logger":"PoolManager.vmWaitingCleanupLook","msg":"starting cleanup loop for waiting mac addresses"} E1001 12:06:24.242414 1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key {"level":"info","ts":1601553984.2425172,"logger":"webhook/server/certificate/manager","msg":"Starting cert manager","webhookType":"Mutating","webhookName":"kubemacpool-mutator"} {"level":"info","ts":1601553984.2426136,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"virtualmachine-controller","source":"kind source: /, Kind="} {"level":"info","ts":1601553984.2426832,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"pod-controller","source":"kind source: /, Kind="} {"level":"info","ts":1601553984.3438911,"logger":"webhook/server/certificate/manager","msg":"Certificate expiration is 2021-10-01 11:37:58 +0000 UTC, rotation deadline is 2021-08-03 02:56:48.485187432 +0000 UTC"," webhookType":"Mutating","webhookName":"kubemacpool-mutator"} {"level":"info","ts":1601553984.3439732,"logger":"webhook/server/certificate/manager","msg":"Cert rotation times {now: 2020-10-01 12:06:24.34395619 +0000 UTC m=+3.883341798, deadline: 2021-08-03 02:56:48.485187432 +0000 UTC, elapsedToRotate: 7334h50m24.141231242s}","webhookType":"Mutating","webhookName":"kubemacpool-mutator"} {"level":"info","ts":1601553984.3439853,"logger":"webhook/server/certificate/manager","msg":"Waiting 7334h50m24.141231242s for next certificate rotation","webhookType":"Mutating","webhookName":"kubemacpool-mutator "} {"level":"info","ts":1601553984.3445072,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"virtualmachine-controller"} {"level":"info","ts":1601553985.046094,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"pod-controller"} {"level":"info","ts":1601553985.0463066,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"pod-controller","worker count":1} {"level":"info","ts":1601553985.0463428,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"virtualmachine-controller","worker count":1} {"level":"info","ts":1601553987.320697,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}} E1001 12:06:29.243244 1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key {"level":"info","ts":1601553990.3445244,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}} {"level":"info","ts":1601553993.3220909,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}} E1001 12:06:34.243064 1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key {"level":"info","ts":1601553996.3196132,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}} E1001 12:06:39.243008 1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key {"level":"info","ts":1601553999.3202014,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}} {"level":"info","ts":1601554002.3195734,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}} E1001 12:06:44.243188 1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key {"level":"info","ts":1601554005.3212478,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}} {"level":"info","ts":1601554008.343275,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}} E1001 12:06:49.243524 1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key {"level":"info","ts":1601554011.3204112,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}} E1001 12:06:54.243405 1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key {"level":"info","ts":1601554014.2537966,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}} {"level":"info","ts":1601554017.2499819,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}} E1001 12:06:59.243111 1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key {"level":"info","ts":1601554020.2695057,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}} {"level":"info","ts":1601554023.2550614,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}} E1001 12:07:04.243096 1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key {"level":"info","ts":1601554026.2540143,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}} E1001 12:07:09.243475 1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key {"level":"info","ts":1601554029.2505765,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}} {"level":"info","ts":1601554032.2484906,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}} E1001 12:07:14.243187 1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key {"level":"info","ts":1601554035.2493591,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}} {"level":"info","ts":1601554038.2496822,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}} E1001 12:07:19.243056 1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key {"level":"info","ts":1601554041.2504363,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}} E1001 12:07:24.242971 1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key {"level":"info","ts":1601554044.2556305,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Virtualization 2.5.0 Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:5127
Hi, It seems I'm facing this issue again during the deployment of HCO-v2.5.5 in an OCP cluster version 4.6.18. $ oc get -n openshift-cnv hyperconverged -o=yaml - lastHeartbeatTime: "2021-05-04T14:14:47Z" lastTransitionTime: "2021-05-04T11:19:39Z" message: 'NetworkAddonsConfig is progressing: Deployment "openshift-cnv/kubemacpool-mac-controller-manager" is not available (awaiting 1 nodes)' reason: NetworkAddonsConfigProgressing status: "True" type: Progressing $ oc get pods -n openshift-cnv kubemacpool-mac-controller-manager-594bbc7f7d-t8m2w 0/1 CrashLoopBackOff 25 $ oc logs kubemacpool-mac-controller-manager-594bbc7f7d-t8m2w -n openshift-cnv {"level":"info","ts":1620137601.0900297,"logger":"webhook/server","msg":"Starting nodenetworkconfigurationpolicy webhook server"} {"level":"info","ts":1620137601.090019,"logger":"PoolManager.vmWaitingCleanupLook","msg":"starting cleanup loop for waiting mac addresses"} {"level":"info","ts":1620137601.0900602,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"} {"level":"info","ts":1620137601.090115,"logger":"controller","msg":"Starting EventSource","controller":"pod-controller","source":"kind source: /, Kind="} {"level":"info","ts":1620137601.0901716,"logger":"controller","msg":"Starting EventSource","controller":"certificate-controller","source":"kind source: /, Kind="} {"level":"info","ts":1620137601.0901961,"logger":"controller","msg":"Starting EventSource","controller":"virtualmachine-controller","source":"kind source: /, Kind="} W0504 14:13:21.092327 1 warnings.go:67] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration W0504 14:13:21.095121 1 warnings.go:67] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration E0504 14:13:21.190565 1 server.go:133] failed verifying /etc/webhook/certs/tls.crt//etc/webhook/certs/tls.key: failed parsing PEM TLS key: data does not contain a valid RSA or ECDSA private key {"level":"info","ts":1620137601.1909199,"logger":"controller","msg":"Starting Controller","controller":"virtualmachine-controller"} {"level":"info","ts":1620137604.1799245,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}} {"level":"info","ts":1620137605.092462,"logger":"controller","msg":"Starting Controller","controller":"pod-controller"} E0504 14:13:26.479856 1 server.go:133] failed verifying /etc/webhook/certs/tls.crt//etc/webhook/certs/tls.key: failed parsing PEM TLS key: data does not contain a valid RSA or ECDSA private key {"level":"info","ts":1620137607.0918505,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}} {"level":"info","ts":1620137610.0924299,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}} E0504 14:13:31.190980 1 server.go:133] failed verifying /etc/webhook/certs/tls.crt//etc/webhook/certs/tls.key: failed parsing PEM TLS key: data does not contain a valid RSA or ECDSA private key
reopening BZ due to a support case opened: https://access.redhat.com/support/cases/#/case/02932882 currently waiting for more info on the environment such as Kubemacpool logs, secret yaml, etc.
Created attachment 1779734 [details] The kubemacpool-mac-controller-manager log related to comment 6
Created attachment 1779735 [details] The secret of kubemacpool realted to comment 6
Thanks all for debugging this. However, the issue you are seeing is not related to the original bug of this ticket. Therefore I'm closing this in favor of this new BZ I created. Please continue the discussion there: https://bugzilla.redhat.com/show_bug.cgi?id=1958108 We are investigating possible fixes.