Hide Forgot
Description of problem: Occasionally, the pod-identity-webhook pod will start before the pod-identity-webhook secret has been created/updated. In a recent ci run: pod-identity-webhook pod log: 2021-11-16T19:55:53.291344432Z E1116 19:55:53.291318 1 store.go:60] Error fetching secret: secrets "pod-identity-webhook" not found ... continues to start anyway pod-identity-webhook secret: creationTimestamp: "2021-11-16T19:56:16Z" I've noticed this much more often in single-node CI jobs.
Verified with 4.10.0-0.nightly-2021-11-18-225109 1. Check pod-identity-webhook pod log, no "Error fetching secret" W1119 02:44:16.958796 1 client_config.go:615] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I1119 02:44:16.961134 1 main.go:191] Creating server I1119 02:44:16.961299 1 main.go:211] Listening on :9999 for metrics and healthz I1119 02:44:16.961493 1 main.go:205] Listening on :6443 2. Check creationTimestamp of secret pod-identity-webhook is before that of "Creating server" in log oc get secret pod-identity-webhook -n openshift-cloud-credential-operator -o yaml creationTimestamp: "2021-11-19T02:42:36Z" 3. Check the volume for secret pod-identity-webhook in pod yaml oc get pod pod-identity-webhook-55bf4bbbfd-ntztx -n openshift-cloud-credential-operator -o yaml volumes: - name: webhook-certs secret: defaultMode: 420 secretName: pod-identity-webhook Reproduced with 4.10.0-0.nightly-2021-11-15-034648 1. Check pod-identity-webhook pod log, no "Error fetching secret" W1119 01:10:16.225018 1 client_config.go:615] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I1119 01:10:16.237606 1 store.go:63] Fetched secret: openshift-cloud-credential-operator/pod-identity-webhook I1119 01:10:16.237741 1 main.go:191] Creating server I1119 01:10:16.238255 1 main.go:211] Listening on :9999 for metrics and healthz I1119 01:10:16.238517 1 main.go:205] Listening on :6443 E1119 01:10:16.240030 1 certificate_manager.go:437] Failed while requesting a signed certificate from the master: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:serviceaccount:openshift-cloud-credential-operator:pod-identity-webhook" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope E1119 01:10:18.362758 1 certificate_manager.go:437] Failed while requesting a signed certificate from the master: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:serviceaccount:openshift-cloud-credential-operator:pod-identity-webhook" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope E1119 01:10:22.631129 1 certificate_manager.go:437] Failed while requesting a signed certificate from the master: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:serviceaccount:openshift-cloud-credential-operator:pod-identity-webhook" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope 2. Check creationTimestamp of secret pod-identity-webhook is before that of "Creating server" in log oc get secret pod-identity-webhook -n openshift-cloud-credential-operator -o yaml creationTimestamp: "2021-11-19T01:10:11Z" 3. Check the volume for secret pod-identity-webhook in pod yaml oc get pod pod-identity-webhook-6db9ff4f5b-khzds -n openshift-cloud-credential-operator -o yaml volumes: - emptyDir: {} name: webhook-certs
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056