Bug 1851829

Summary:

[CNV-2.5] kubemacpool-mac-controller-manager failing to start due invalid private key

Product:

Container Native Virtualization (CNV)

Reporter:

Lukas Bednar <lbednar>

Component:

Networking

Assignee:

Ram Lavi <ralavi>

Status:

CLOSED ERRATA

QA Contact:

Meni Yakove <myakove>

Severity:

high

Docs Contact:

Priority:

high

Version:

2.5.0

CC:

cnv-qe-bugs, jsaucier, kgershon, lbednar, maugarci, ncredi, phoracek, vhernand

Target Milestone:

---

Keywords:

AutomationBlocker, Regression, Reopened

Target Release:

2.5.0

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

At some point in 2.5 this was fixed and never turned to QE

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2021-05-07 08:25:15 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
openshift-cnv kubemacpool-mac-controller-manager-f9f978598-6cszk.log	none
The kubemacpool-mac-controller-manager log related to comment 6	none
The secret of kubemacpool realted to comment 6	none

Description Lukas Bednar 2020-06-29 07:28:46 UTC

Created attachment 1699083 [details]
openshift-cnv kubemacpool-mac-controller-manager-f9f978598-6cszk.log

Description of problem:


$ oc get pod -n openshift-cnv
kubemacpool-mac-controller-manager-f9f978598-6cszk   0/1     Running   27         2d8h
kubemacpool-mac-controller-manager-f9f978598-wpl6c   0/1     Running   27         2d8h


$ oc get -n openshift-cnv hyperconverged -o=yaml
    - lastHeartbeatTime: "2020-06-29T07:23:29Z"
      lastTransitionTime: "2020-06-26T23:11:50Z"
      message: 'NetworkAddonsConfig is progressing: Deployment "openshift-cnv/kubemacpool-mac-controller-manager"
        is not available (awaiting 2 nodes)'
      reason: NetworkAddonsConfigProgressing
      status: "False"
      type: Upgradeable


$ oc logs -n openshift-cnv kubemacpool-mac-controller-manager-f9f978598-6cszk
E0629 07:09:51.014082       1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key
{"level":"error","ts":1593414591.015211,"logger":"manager","msg":"unable to run the manager","error":"failed watting for ready TLS key/cert: timed out waiting for the condition","errorVerbose":"timed out waiting for the condition\nfailed watting for ready TLS key/cert\ngithub.com/qinqon/kube-admission-webhook/pkg/webhook/server.(*Server).Start\n\t/go/src/github.com/k8snetworkplumbingwg/kubemacpool/vendor/github.com/qinqon/kube-admission-webhook/pkg/webhook/server/server.go:159\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startNonLeaderElectionRunnables.func1\n\t/go/src/github.com/k8snetworkplumbingwg/kubemacpool/vendor/sigs.k8s.io/controller-runtime/pkg/manager/internal.go:492\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1357","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/k8snetworkplumbingwg/kubemacpool/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/k8snetworkplumbingwg/kubemacpool/pkg/manager.(*KubeMacPoolManager).Run\n\t/go/src/github.com/k8snetworkplumbingwg/kubemacpool/pkg/manager/manager.go:125\nmain.main\n\t/go/src/github.com/k8snetworkplumbingwg/kubemacpool/cmd/manager/main.go:88\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:203"}
{"level":"info","ts":1593414591.0157564,"logger":"manager","msg":"Setting up Manager"}
I0629 07:09:52.066415       1 request.go:621] Throttling request took 1.043446451s, request: GET:https://172.30.0.1:443/apis/imageregistry.operator.openshift.io/v1?timeout=32s
{"level":"info","ts":1593414593.7263834,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1593414593.731542,"logger":"manager","msg":"Setting up controllers"}
{"level":"info","ts":1593414593.7317173,"logger":"manager","msg":"Setting up webhooks"}
{"level":"info","ts":1593414593.731812,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/readyz"}
{"level":"info","ts":1593414593.7318296,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-pods"}
{"level":"info","ts":1593414593.7318873,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-virtualmachines"}
I0629 07:09:53.732028       1 leaderelection.go:242] attempting to acquire leader lease  openshift-cnv/kubemacpool-election...
{"level":"info","ts":1593414593.732924,"logger":"webhook/server","msg":"Starting nodenetworkconfigurationpolicy webhook server"}
E0629 07:09:53.733103       1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key
{"level":"info","ts":1593414593.7329278,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}



Version-Release number of selected component (if applicable):
OCP-4.6
HCO-v2.3.0-411


How reproducible: 100


Steps to Reproduce:
1. Deploy CNV
2.
3.

Actual results: kubemacpool-mac-controller-manager failing to start


Expected results: CNV deployed successfuly 


Additional info:

Comment 1 Lukas Bednar 2020-08-17 08:53:07 UTC

We did not see this issue at least for a month . Moving it to Verified then.

Comment 2 Jean-Francois Saucier 2020-10-01 12:09:11 UTC

I have the same issue with a CNV 2.4.1 deployment on OCP 4.5.11. Maybe it is other issue but this is the only BZ that I found that mention this :

# oc logs kubemacpool-mac-controller-manager-6f9c447bbd-f92v4
{"level":"info","ts":1601553984.2422936,"logger":"PoolManager.vmWaitingCleanupLook","msg":"starting cleanup loop for waiting mac addresses"}
E1001 12:06:24.242414       1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key
{"level":"info","ts":1601553984.2425172,"logger":"webhook/server/certificate/manager","msg":"Starting cert manager","webhookType":"Mutating","webhookName":"kubemacpool-mutator"}
{"level":"info","ts":1601553984.2426136,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"virtualmachine-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1601553984.2426832,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"pod-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1601553984.3438911,"logger":"webhook/server/certificate/manager","msg":"Certificate expiration is 2021-10-01 11:37:58 +0000 UTC, rotation deadline is 2021-08-03 02:56:48.485187432 +0000 UTC","
webhookType":"Mutating","webhookName":"kubemacpool-mutator"}
{"level":"info","ts":1601553984.3439732,"logger":"webhook/server/certificate/manager","msg":"Cert rotation times {now: 2020-10-01 12:06:24.34395619 +0000 UTC m=+3.883341798, deadline: 2021-08-03 02:56:48.485187432
 +0000 UTC, elapsedToRotate: 7334h50m24.141231242s}","webhookType":"Mutating","webhookName":"kubemacpool-mutator"}
{"level":"info","ts":1601553984.3439853,"logger":"webhook/server/certificate/manager","msg":"Waiting 7334h50m24.141231242s for next certificate rotation","webhookType":"Mutating","webhookName":"kubemacpool-mutator
"}
{"level":"info","ts":1601553984.3445072,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"virtualmachine-controller"}
{"level":"info","ts":1601553985.046094,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"pod-controller"}
{"level":"info","ts":1601553985.0463066,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"pod-controller","worker count":1}
{"level":"info","ts":1601553985.0463428,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"virtualmachine-controller","worker count":1}
{"level":"info","ts":1601553987.320697,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}
E1001 12:06:29.243244       1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key
{"level":"info","ts":1601553990.3445244,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}
{"level":"info","ts":1601553993.3220909,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}
E1001 12:06:34.243064       1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key
{"level":"info","ts":1601553996.3196132,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}
E1001 12:06:39.243008       1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key
{"level":"info","ts":1601553999.3202014,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}
{"level":"info","ts":1601554002.3195734,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}
E1001 12:06:44.243188       1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key
{"level":"info","ts":1601554005.3212478,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}
{"level":"info","ts":1601554008.343275,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}
E1001 12:06:49.243524       1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key
{"level":"info","ts":1601554011.3204112,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}
E1001 12:06:54.243405       1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key
{"level":"info","ts":1601554014.2537966,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}
{"level":"info","ts":1601554017.2499819,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}
E1001 12:06:59.243111       1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key
{"level":"info","ts":1601554020.2695057,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}
{"level":"info","ts":1601554023.2550614,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}
E1001 12:07:04.243096       1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key
{"level":"info","ts":1601554026.2540143,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}
E1001 12:07:09.243475       1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key
{"level":"info","ts":1601554029.2505765,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}
{"level":"info","ts":1601554032.2484906,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}
E1001 12:07:14.243187       1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key
{"level":"info","ts":1601554035.2493591,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}
{"level":"info","ts":1601554038.2496822,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}
E1001 12:07:19.243056       1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key
{"level":"info","ts":1601554041.2504363,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}
E1001 12:07:24.242971       1 server.go:147] failed parsing TLS key: data does not contain a valid RSA or ECDSA private key
{"level":"info","ts":1601554044.2556305,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}

Comment 5 errata-xmlrpc 2020-11-17 13:24:21 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Virtualization 2.5.0 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:5127

Comment 6 maugarci 2021-05-04 14:25:07 UTC

Hi, 

It seems I'm facing this issue again during the deployment of HCO-v2.5.5 in an OCP cluster version 4.6.18.


$ oc get -n openshift-cnv hyperconverged -o=yaml
 
     - lastHeartbeatTime: "2021-05-04T14:14:47Z"
      lastTransitionTime: "2021-05-04T11:19:39Z"
      message: 'NetworkAddonsConfig is progressing: Deployment "openshift-cnv/kubemacpool-mac-controller-manager" is not available (awaiting 1 nodes)'
      reason: NetworkAddonsConfigProgressing
      status: "True"
      type: Progressing

$ oc get pods -n openshift-cnv
kubemacpool-mac-controller-manager-594bbc7f7d-t8m2w   0/1     CrashLoopBackOff   25


$ oc logs kubemacpool-mac-controller-manager-594bbc7f7d-t8m2w -n openshift-cnv

{"level":"info","ts":1620137601.0900297,"logger":"webhook/server","msg":"Starting nodenetworkconfigurationpolicy webhook server"}
{"level":"info","ts":1620137601.090019,"logger":"PoolManager.vmWaitingCleanupLook","msg":"starting cleanup loop for waiting mac addresses"}
{"level":"info","ts":1620137601.0900602,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1620137601.090115,"logger":"controller","msg":"Starting EventSource","controller":"pod-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1620137601.0901716,"logger":"controller","msg":"Starting EventSource","controller":"certificate-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1620137601.0901961,"logger":"controller","msg":"Starting EventSource","controller":"virtualmachine-controller","source":"kind source: /, Kind="}
W0504 14:13:21.092327       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
W0504 14:13:21.095121       1 warnings.go:67] admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
E0504 14:13:21.190565       1 server.go:133] failed verifying /etc/webhook/certs/tls.crt//etc/webhook/certs/tls.key: failed parsing PEM TLS key: data does not contain a valid RSA or ECDSA private key
{"level":"info","ts":1620137601.1909199,"logger":"controller","msg":"Starting Controller","controller":"virtualmachine-controller"}
{"level":"info","ts":1620137604.1799245,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}
{"level":"info","ts":1620137605.092462,"logger":"controller","msg":"Starting Controller","controller":"pod-controller"}
E0504 14:13:26.479856       1 server.go:133] failed verifying /etc/webhook/certs/tls.crt//etc/webhook/certs/tls.key: failed parsing PEM TLS key: data does not contain a valid RSA or ECDSA private key
{"level":"info","ts":1620137607.0918505,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}
{"level":"info","ts":1620137610.0924299,"logger":"PoolManager.vmWaitingCleanupLook","msg":"the configMap is empty","configMapName":"kubemacpool-vm-configmap","macPoolMap":{}}
E0504 14:13:31.190980       1 server.go:133] failed verifying /etc/webhook/certs/tls.crt//etc/webhook/certs/tls.key: failed parsing PEM TLS key: data does not contain a valid RSA or ECDSA private key

Comment 7 Ram Lavi 2021-05-05 08:59:30 UTC

reopening BZ due to a support case opened: https://access.redhat.com/support/cases/#/case/02932882
currently waiting for more info on the environment such as Kubemacpool logs, secret yaml, etc.

Comment 11 maugarci 2021-05-05 10:47:50 UTC

Created attachment 1779734 [details]
The kubemacpool-mac-controller-manager log related to comment 6

Comment 12 maugarci 2021-05-05 10:49:23 UTC

Created attachment 1779735 [details]
The secret of kubemacpool realted to comment 6

Comment 15 Petr Horáček 2021-05-07 08:25:15 UTC

Thanks all for debugging this. However, the issue you are seeing is not related to the original bug of this ticket. Therefore I'm closing this in favor of this new BZ I created. Please continue the discussion there: https://bugzilla.redhat.com/show_bug.cgi?id=1958108

We are investigating possible fixes.