Description of problem: Seeing a recurrence of BZ 1963730 Version-Release number of selected component (if applicable): 4.8.24 vSphere IPI How reproducible: Every installation Steps to Reproduce: 1. Install 4.8.24 OCP Actual results: One kube-apiserver pod goes into CLBO with the error 2022-02-11T19:52:21.969264100Z I0211 19:52:21.969203 20 dynamic_serving_content.go:111] Loaded a new cert/key pair for "sni-serving-cert::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.crt::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.key" 2022-02-11T19:52:21.969457707Z Error: failed to load SNI cert and key: tls: failed to find any PEM data in key input 2022-02-11T19:52:21.971842077Z I0211 19:52:21.971779 1 main.go:198] Termination finished with exit code 1 Checking keys show they exist and look fine, data is not malformed. Eventually cluster becomes unusable, customer re-installs, issue occurs again on different node. Expected results: A normally running cluster. Additional info: This is almost identical to BZ 1963730 which was fixed in 4.8.5.
Sosreport and must-gather are too large to upload here, you can find them in the case: https://access.redhat.com/support/cases/#/case/03148397
Uploading kube-apiserver pod logs and pod yaml. pod was missing cert-syncer logs. revision label is 14. [cshepher@supportshell-1 sosreport-openshift48release-jh7ht-master-2-03148397-2022-03-03-hidsxlf]$ ls -lZ etc/kubernetes/static-pod-resources/kube-apiserver-pod-14 drwxrwxrwx. yank yank system_u:object_r:nfs_t:s0 configmaps -rw-rw-rw-. yank yank system_u:object_r:nfs_t:s0 kube-apiserver-pod.yaml drwxrwxrwx. yank yank system_u:object_r:nfs_t:s0 secrets
A user provided serving certificate key was provided in a malformed PEM container (missing "-----END PRIVATE KEY-----").