1971624 – [release-4.9] kube-apiserver failed to load SNI cert and key

Bug 1971624 - [release-4.9] kube-apiserver failed to load SNI cert and key

Summary: [release-4.9] kube-apiserver failed to load SNI cert and key

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	kube-apiserver
Sub Component:
Version:	4.8
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	4.9.0
Assignee:	Lukasz Szaszkiewicz
QA Contact:	Ke Wang
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1963730 2108622
TreeView+	depends on / blocked

Reported:	2021-06-14 13:12 UTC by Lukasz Szaszkiewicz
Modified:	2022-07-19 14:11 UTC (History)
CC List:	11 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:	Writes Kube API server certificates in an atomic way. It prevents races between multiple processes which might have left some certificates empty. That in turn could prevent the server from running.
Clone Of:	1963730
Environment:
Last Closed:	2021-10-18 17:33:59 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift cluster-kube-apiserver-operator pull 1145	0	None	open	kube-apiserver failed to load SNI cert and key	2021-06-14 13:14:37 UTC
Red Hat Product Errata	RHSA-2021:3759	0	None	None	None	2021-10-18 17:34:23 UTC

Comment 2 Ke Wang 2021-07-12 07:33:13 UTC

1. Did a quick search from CI tests in the past 14 days [1], there is no any related error logs except finding the bugs that already exists。 

[1] https://search.ci.openshift.org/?search=load+SNI+cert+and+key&maxAge=168h&context=1&type=all&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job


2. I ran one script to force kube-apiserver rollout every 30 minutes， each rollout results in secret manifest files write operation, 
$ cat test.sh
for i in {1..40};do
    oc patch kubeapiserver/cluster --type=json -p '[ {"op": "replace", "path": "/spec/forceRedeploymentReason", "value": "roll-'"$( date --rfc-3339=ns )"'"} ]'
    sleep 1800
done 
masters=$(oc get no -l node-role.kubernetes.io/master | sed '1d' | awk '{print $1}')
for node in $masters; do echo $node;oc debug no/$node -- chroot /host bash -c "grep -nr 'Writing secret manifest.*internal-loadbalancer-serving-certkey' /var/log/";done > write-secret.txt
for node in $masters; do echo $node;oc debug no/$node -- chroot /host bash -c "grep -nr 'failed to load SNI cert and key' /var/log/";done > error.txt

$ grep 'fail' error.txt
no results found.

3. And I checked the 'Writing secret manifest' operation in another cluster, observed the installer pod and the cert-syncer container did 'Writing secret manifest' in order without any race.

kewang-0749dr-pz8ml-control-plane-0
/var/log/pods/openshift-kube-apiserver_installer-8-kewang-0749dr-pz8ml-control-plane-0_3d7e6f8d-b277-4c00-9d66-2f01f69855b6/installer/0.log:174:2021-07-08T03:43:24.600750659+00:00 stderr F I0708 03:43:24.600741       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/openshift-kube-apiserver_installer-8-kewang-0749dr-pz8ml-control-plane-0_3d7e6f8d-b277-4c00-9d66-2f01f69855b6/installer/0.log:175:2021-07-08T03:43:24.600851300+00:00 stderr F I0708 03:43:24.600834       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/openshift-kube-apiserver_kube-apiserver-kewang-0749dr-pz8ml-control-plane-0_606af606-f54f-4963-b64a-1c49ff103e9e/kube-apiserver-cert-syncer/0.log:168:2021-07-08T09:41:55.024178015+00:00 stderr F I0708 09:41:55.023980       1 certsync_controller.go:266] Writing secret manifest "/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/openshift-kube-apiserver_kube-apiserver-kewang-0749dr-pz8ml-control-plane-0_606af606-f54f-4963-b64a-1c49ff103e9e/kube-apiserver-cert-syncer/0.log:169:2021-07-08T09:41:55.025151263+00:00 stderr F I0708 09:41:55.025095       1 certsync_controller.go:266] Writing secret manifest "/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/openshift-kube-apiserver_installer-11-kewang-0749dr-pz8ml-control-plane-0_7fa8827c-5536-48e8-a617-b59229eff5da/installer/0.log:174:2021-07-08T09:27:30.781865464+00:00 stderr F I0708 09:27:30.781853       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/openshift-kube-apiserver_installer-11-kewang-0749dr-pz8ml-control-plane-0_7fa8827c-5536-48e8-a617-b59229eff5da/installer/0.log:175:2021-07-08T09:27:30.781987259+00:00 stderr F I0708 09:27:30.781971       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/openshift-kube-apiserver_installer-6-kewang-0749dr-pz8ml-control-plane-0_0e6f0077-a7a8-4361-a844-2c67b3234c9d/installer/0.log:174:2021-07-07T09:54:05.298973839+00:00 stderr F I0707 09:54:05.298961       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/openshift-kube-apiserver_installer-6-kewang-0749dr-pz8ml-control-plane-0_0e6f0077-a7a8-4361-a844-2c67b3234c9d/installer/0.log:175:2021-07-07T09:54:05.299079432+00:00 stderr F I0707 09:54:05.299056       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/openshift-kube-apiserver_installer-7-kewang-0749dr-pz8ml-control-plane-0_1e96102d-3900-4454-bb73-2417c471bc50/installer/0.log:174:2021-07-07T10:01:48.080707020+00:00 stderr F I0707 10:01:48.080697       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/openshift-kube-apiserver_installer-7-kewang-0749dr-pz8ml-control-plane-0_1e96102d-3900-4454-bb73-2417c471bc50/installer/0.log:175:2021-07-07T10:01:48.080800353+00:00 stderr F I0707 10:01:48.080784       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/openshift-kube-apiserver_installer-10-kewang-0749dr-pz8ml-control-plane-0_64d5a7b0-bca5-4a1b-a530-03d4e045866d/installer/0.log:174:2021-07-08T04:42:49.609205334+00:00 stderr F I0708 04:42:49.609190       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/openshift-kube-apiserver_installer-10-kewang-0749dr-pz8ml-control-plane-0_64d5a7b0-bca5-4a1b-a530-03d4e045866d/installer/0.log:175:2021-07-08T04:42:49.609426616+00:00 stderr F I0708 04:42:49.609400       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/openshift-kube-apiserver_installer-9-kewang-0749dr-pz8ml-control-plane-0_114979d9-96ae-4439-9a21-75ba211ad261/installer/0.log:174:2021-07-08T04:39:47.812652298+00:00 stderr F I0708 04:39:47.812641       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/openshift-kube-apiserver_installer-9-kewang-0749dr-pz8ml-control-plane-0_114979d9-96ae-4439-9a21-75ba211ad261/installer/0.log:175:2021-07-08T04:39:47.812714614+00:00 stderr F I0708 04:39:47.812698       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
kewang-0749dr-pz8ml-control-plane-1
/var/log/pods/openshift-kube-apiserver_installer-3-kewang-0749dr-pz8ml-control-plane-1_9fbb9797-f73d-49dc-b7eb-37dbc318ed48/installer/0.log:171:2021-07-07T09:47:49.354424552+00:00 stderr F I0707 09:47:49.293587       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/openshift-kube-apiserver_installer-3-kewang-0749dr-pz8ml-control-plane-1_9fbb9797-f73d-49dc-b7eb-37dbc318ed48/installer/0.log:172:2021-07-07T09:47:49.354424552+00:00 stderr F I0707 09:47:49.293748       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/openshift-kube-apiserver_installer-6-kewang-0749dr-pz8ml-control-plane-1_bd4c8eba-576e-4549-8dca-e187fa762857/installer/0.log:174:2021-07-07T09:52:30.787390626+00:00 stderr F I0707 09:52:30.782860       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/openshift-kube-apiserver_installer-6-kewang-0749dr-pz8ml-control-plane-1_bd4c8eba-576e-4549-8dca-e187fa762857/installer/0.log:175:2021-07-07T09:52:30.787390626+00:00 stderr F I0707 09:52:30.782934       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/openshift-kube-apiserver_kube-apiserver-kewang-0749dr-pz8ml-control-plane-1_17db517c-6964-4c4d-ad01-8729168b9c27/kube-apiserver-cert-syncer/0.log:109:2021-07-08T09:41:55.016807311+00:00 stderr F I0708 09:41:55.016691       1 certsync_controller.go:266] Writing secret manifest "/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/openshift-kube-apiserver_kube-apiserver-kewang-0749dr-pz8ml-control-plane-1_17db517c-6964-4c4d-ad01-8729168b9c27/kube-apiserver-cert-syncer/0.log:110:2021-07-08T09:41:55.016867021+00:00 stderr F I0708 09:41:55.016851       1 certsync_controller.go:266] Writing secret manifest "/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/openshift-kube-apiserver_installer-4-kewang-0749dr-pz8ml-control-plane-1_3aba821a-fc2f-4ae8-bc42-2397baa4cc61/installer/0.log:172:2021-07-07T09:50:23.489363190+00:00 stderr F I0707 09:50:23.489340       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/openshift-kube-apiserver_installer-4-kewang-0749dr-pz8ml-control-plane-1_3aba821a-fc2f-4ae8-bc42-2397baa4cc61/installer/0.log:173:2021-07-07T09:50:23.489432999+00:00 stderr F I0707 09:50:23.489421       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/openshift-kube-apiserver_installer-5-kewang-0749dr-pz8ml-control-plane-1_5430e281-b69d-4d43-8f07-822d53508d05/installer/0.log:172:2021-07-07T09:51:07.691842193+00:00 stderr F I0707 09:51:07.691834       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/openshift-kube-apiserver_installer-5-kewang-0749dr-pz8ml-control-plane-1_5430e281-b69d-4d43-8f07-822d53508d05/installer/0.log:173:2021-07-07T09:51:07.691954584+00:00 stderr F I0707 09:51:07.691942       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/openshift-kube-apiserver_installer-7-kewang-0749dr-pz8ml-control-plane-1_fde8f01e-06ca-4589-a818-eb9a32c05cf3/installer/0.log:174:2021-07-07T10:05:31.286574144+00:00 stderr F I0707 10:05:31.286329       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/openshift-kube-apiserver_installer-7-kewang-0749dr-pz8ml-control-plane-1_fde8f01e-06ca-4589-a818-eb9a32c05cf3/installer/0.log:175:2021-07-07T10:05:31.286574144+00:00 stderr F I0707 10:05:31.286444       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/openshift-kube-apiserver_installer-8-kewang-0749dr-pz8ml-control-plane-1_d382c542-fd17-4750-9ca2-63f371bba17d/installer/0.log:174:2021-07-08T03:47:20.180506742+00:00 stderr F I0708 03:47:20.180499       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/openshift-kube-apiserver_installer-8-kewang-0749dr-pz8ml-control-plane-1_d382c542-fd17-4750-9ca2-63f371bba17d/installer/0.log:175:2021-07-08T03:47:20.180581787+00:00 stderr F I0708 03:47:20.180568       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/openshift-kube-apiserver_installer-10-kewang-0749dr-pz8ml-control-plane-1_323dcdae-c089-4492-bd4d-06a413f94633/installer/0.log:174:2021-07-08T04:44:22.976171432+00:00 stderr F I0708 04:44:22.976160       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/openshift-kube-apiserver_installer-10-kewang-0749dr-pz8ml-control-plane-1_323dcdae-c089-4492-bd4d-06a413f94633/installer/0.log:175:2021-07-08T04:44:22.976254085+00:00 stderr F I0708 04:44:22.976242       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/default_kewang-0749dr-pz8ml-control-plane-1-debug_05ac2b22-2df3-4898-982d-d0304c67151a/container-00/0.log:1:2021-07-08T10:57:48.403050220+00:00 stdout F /var/log/pods/openshift-kube-apiserver_installer-3-kewang-0749dr-pz8ml-control-plane-1_9fbb9797-f73d-49dc-b7eb-37dbc318ed48/installer/0.log:171:2021-07-07T09:47:49.354424552+00:00 stderr F I0707 09:47:49.293587       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/default_kewang-0749dr-pz8ml-control-plane-1-debug_05ac2b22-2df3-4898-982d-d0304c67151a/container-00/0.log:2:2021-07-08T10:57:48.403050220+00:00 stdout F /var/log/pods/openshift-kube-apiserver_installer-3-kewang-0749dr-pz8ml-control-plane-1_9fbb9797-f73d-49dc-b7eb-37dbc318ed48/installer/0.log:172:2021-07-07T09:47:49.354424552+00:00 stderr F I0707 09:47:49.293748       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/default_kewang-0749dr-pz8ml-control-plane-1-debug_05ac2b22-2df3-4898-982d-d0304c67151a/container-00/0.log:3:2021-07-08T10:57:48.403050220+00:00 stdout F /var/log/pods/openshift-kube-apiserver_installer-6-kewang-0749dr-pz8ml-control-plane-1_bd4c8eba-576e-4549-8dca-e187fa762857/installer/0.log:174:2021-07-07T09:52:30.787390626+00:00 stderr F I0707 09:52:30.782860       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/default_kewang-0749dr-pz8ml-control-plane-1-debug_05ac2b22-2df3-4898-982d-d0304c67151a/container-00/0.log:4:2021-07-08T10:57:48.403050220+00:00 stdout F /var/log/pods/openshift-kube-apiserver_installer-6-kewang-0749dr-pz8ml-control-plane-1_bd4c8eba-576e-4549-8dca-e187fa762857/installer/0.log:175:2021-07-07T09:52:30.787390626+00:00 stderr F I0707 09:52:30.782934       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/default_kewang-0749dr-pz8ml-control-plane-1-debug_05ac2b22-2df3-4898-982d-d0304c67151a/container-00/0.log:5:2021-07-08T10:57:48.403050220+00:00 stdout F /var/log/pods/openshift-kube-apiserver_kube-apiserver-kewang-0749dr-pz8ml-control-plane-1_17db517c-6964-4c4d-ad01-8729168b9c27/kube-apiserver-cert-syncer/0.log:109:2021-07-08T09:41:55.016807311+00:00 stderr F I0708 09:41:55.016691       1 certsync_controller.go:266] Writing secret manifest "/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/default_kewang-0749dr-pz8ml-control-plane-1-debug_05ac2b22-2df3-4898-982d-d0304c67151a/container-00/0.log:6:2021-07-08T10:57:48.403050220+00:00 stdout F /var/log/pods/openshift-kube-apiserver_kube-apiserver-kewang-0749dr-pz8ml-control-plane-1_17db517c-6964-4c4d-ad01-8729168b9c27/kube-apiserver-cert-syncer/0.log:110:2021-07-08T09:41:55.016867021+00:00 stderr F I0708 09:41:55.016851       1 certsync_controller.go:266] Writing secret manifest "/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/default_kewang-0749dr-pz8ml-control-plane-1-debug_05ac2b22-2df3-4898-982d-d0304c67151a/container-00/0.log:7:2021-07-08T10:57:48.403050220+00:00 stdout F /var/log/pods/openshift-kube-apiserver_installer-4-kewang-0749dr-pz8ml-control-plane-1_3aba821a-fc2f-4ae8-bc42-2397baa4cc61/installer/0.log:172:2021-07-07T09:50:23.489363190+00:00 stderr F I0707 09:50:23.489340       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/default_kewang-0749dr-pz8ml-control-plane-1-debug_05ac2b22-2df3-4898-982d-d0304c67151a/container-00/0.log:8:2021-07-08T10:57:48.403050220+00:00 stdout F /var/log/pods/openshift-kube-apiserver_installer-4-kewang-0749dr-pz8ml-control-plane-1_3aba821a-fc2f-4ae8-bc42-2397baa4cc61/installer/0.log:173:2021-07-07T09:50:23.489432999+00:00 stderr F I0707 09:50:23.489421       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/default_kewang-0749dr-pz8ml-control-plane-1-debug_05ac2b22-2df3-4898-982d-d0304c67151a/container-00/0.log:9:2021-07-08T10:57:48.403050220+00:00 stdout F /var/log/pods/openshift-kube-apiserver_installer-5-kewang-0749dr-pz8ml-control-plane-1_5430e281-b69d-4d43-8f07-822d53508d05/installer/0.log:172:2021-07-07T09:51:07.691842193+00:00 stderr F I0707 09:51:07.691834       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/default_kewang-0749dr-pz8ml-control-plane-1-debug_05ac2b22-2df3-4898-982d-d0304c67151a/container-00/0.log:10:2021-07-08T10:57:48.403050220+00:00 stdout F /var/log/pods/openshift-kube-apiserver_installer-5-kewang-0749dr-pz8ml-control-plane-1_5430e281-b69d-4d43-8f07-822d53508d05/installer/0.log:173:2021-07-07T09:51:07.691954584+00:00 stderr F I0707 09:51:07.691942       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/openshift-kube-apiserver_installer-11-kewang-0749dr-pz8ml-control-plane-1_b0e7bbad-3990-47bf-80a0-4726d5c8b043/installer/0.log:174:2021-07-08T09:31:36.686470015+00:00 stderr F I0708 09:31:36.686463       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/openshift-kube-apiserver_installer-11-kewang-0749dr-pz8ml-control-plane-1_b0e7bbad-3990-47bf-80a0-4726d5c8b043/installer/0.log:175:2021-07-08T09:31:36.686563099+00:00 stderr F I0708 09:31:36.686545       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
kewang-0749dr-pz8ml-control-plane-2
/var/log/pods/openshift-kube-apiserver_installer-4-kewang-0749dr-pz8ml-control-plane-2_4ff76b0c-f1d4-483c-9545-01c391273038/installer/0.log:172:2021-07-07T09:48:56.339768117+00:00 stderr F I0707 09:48:56.339749       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/openshift-kube-apiserver_installer-4-kewang-0749dr-pz8ml-control-plane-2_4ff76b0c-f1d4-483c-9545-01c391273038/installer/0.log:173:2021-07-07T09:48:56.339857815+00:00 stderr F I0707 09:48:56.339832       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/openshift-kube-apiserver_installer-11-kewang-0749dr-pz8ml-control-plane-2_c7e0c35b-5a6c-45d3-986d-9de0cc5bbf43/installer/0.log:174:2021-07-08T09:36:04.040115406+00:00 stderr F I0708 09:36:04.040105       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/openshift-kube-apiserver_installer-11-kewang-0749dr-pz8ml-control-plane-2_c7e0c35b-5a6c-45d3-986d-9de0cc5bbf43/installer/0.log:175:2021-07-08T09:36:04.040212179+00:00 stderr F I0708 09:36:04.040192       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/openshift-kube-apiserver_installer-7-kewang-0749dr-pz8ml-control-plane-2_f6c9f2c3-0a40-41d4-abf8-6f4b47a2dd33/installer/0.log:174:2021-07-07T09:57:51.340507481+00:00 stderr F I0707 09:57:51.340499       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/openshift-kube-apiserver_installer-7-kewang-0749dr-pz8ml-control-plane-2_f6c9f2c3-0a40-41d4-abf8-6f4b47a2dd33/installer/0.log:175:2021-07-07T09:57:51.340647530+00:00 stderr F I0707 09:57:51.340623       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/openshift-kube-apiserver_installer-10-kewang-0749dr-pz8ml-control-plane-2_f7d8a6ca-22e0-4638-97ed-a48e6e8a2ce9/installer/0.log:174:2021-07-08T04:48:04.838947093+00:00 stderr F I0708 04:48:04.838439       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/openshift-kube-apiserver_installer-10-kewang-0749dr-pz8ml-control-plane-2_f7d8a6ca-22e0-4638-97ed-a48e6e8a2ce9/installer/0.log:175:2021-07-08T04:48:04.838947093+00:00 stderr F I0708 04:48:04.838556       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/openshift-kube-apiserver_installer-8-kewang-0749dr-pz8ml-control-plane-2_0b325bb1-cb97-4aee-82a3-5b08b012c1cd/installer/0.log:174:2021-07-08T03:51:06.368347574+00:00 stderr F I0708 03:51:06.367398       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
/var/log/pods/openshift-kube-apiserver_installer-8-kewang-0749dr-pz8ml-control-plane-2_0b325bb1-cb97-4aee-82a3-5b08b012c1cd/installer/0.log:175:2021-07-08T03:51:06.368347574+00:00 stderr F I0708 03:51:06.367567       1 cmd.go:413] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/openshift-kube-apiserver_kube-apiserver-kewang-0749dr-pz8ml-control-plane-2_b6aee9ac-d7ac-4161-9e0a-10cfc6fb97cc/kube-apiserver-cert-syncer/0.log:65:2021-07-08T09:41:55.021045485+00:00 stderr F I0708 09:41:55.021023       1 certsync_controller.go:266] Writing secret manifest "/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
/var/log/pods/openshift-kube-apiserver_kube-apiserver-kewang-0749dr-pz8ml-control-plane-2_b6aee9ac-d7ac-4161-9e0a-10cfc6fb97cc/kube-apiserver-cert-syncer/0.log:66:2021-07-08T09:41:55.021260400+00:00 stderr F I0708 09:41:55.021214       1 certsync_controller.go:266] Writing secret manifest "/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...

4.I have observed all of our e2e and upgrade CI tests recently and no reproductions have been found, the conditions that cause this bug are really rare. The easiest way to prevent race conditions due to "read-modify-write" instructions is by ensuring that such operations are atomic,  an atomic way to write file was already used in this bug's PR. 

Based on the four points above， I think the bug was fxied with the PR https://github.com/openshift/cluster-kube-apiserver-operator/pull/1145.

Comment 5 errata-xmlrpc 2021-10-18 17:33:59 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759

Note You need to log in before you can comment on or make changes to this bug.