Bug 2024613
| Summary: | pod-identity-webhook starts without tls | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Luis Sanchez <sanchezl> |
| Component: | Cloud Credential Operator | Assignee: | Luis Sanchez <sanchezl> |
| Status: | CLOSED ERRATA | QA Contact: | Jianping SHu <jshu> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.10 | CC: | jshu, lwan |
| Target Milestone: | --- | ||
| Target Release: | 4.10.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-03-10 16:29:22 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 2024751 | ||
Verified with 4.10.0-0.nightly-2021-11-18-225109
1. Check pod-identity-webhook pod log, no "Error fetching secret"
W1119 02:44:16.958796 1 client_config.go:615] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I1119 02:44:16.961134 1 main.go:191] Creating server
I1119 02:44:16.961299 1 main.go:211] Listening on :9999 for metrics and healthz
I1119 02:44:16.961493 1 main.go:205] Listening on :6443
2. Check creationTimestamp of secret pod-identity-webhook is before that of "Creating server" in log
oc get secret pod-identity-webhook -n openshift-cloud-credential-operator -o yaml
creationTimestamp: "2021-11-19T02:42:36Z"
3. Check the volume for secret pod-identity-webhook in pod yaml
oc get pod pod-identity-webhook-55bf4bbbfd-ntztx -n openshift-cloud-credential-operator -o yaml
volumes:
- name: webhook-certs
secret:
defaultMode: 420
secretName: pod-identity-webhook
Reproduced with 4.10.0-0.nightly-2021-11-15-034648
1. Check pod-identity-webhook pod log, no "Error fetching secret"
W1119 01:10:16.225018 1 client_config.go:615] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I1119 01:10:16.237606 1 store.go:63] Fetched secret: openshift-cloud-credential-operator/pod-identity-webhook
I1119 01:10:16.237741 1 main.go:191] Creating server
I1119 01:10:16.238255 1 main.go:211] Listening on :9999 for metrics and healthz
I1119 01:10:16.238517 1 main.go:205] Listening on :6443
E1119 01:10:16.240030 1 certificate_manager.go:437] Failed while requesting a signed certificate from the master: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:serviceaccount:openshift-cloud-credential-operator:pod-identity-webhook" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope
E1119 01:10:18.362758 1 certificate_manager.go:437] Failed while requesting a signed certificate from the master: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:serviceaccount:openshift-cloud-credential-operator:pod-identity-webhook" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope
E1119 01:10:22.631129 1 certificate_manager.go:437] Failed while requesting a signed certificate from the master: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:serviceaccount:openshift-cloud-credential-operator:pod-identity-webhook" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope
2. Check creationTimestamp of secret pod-identity-webhook is before that of "Creating server" in log
oc get secret pod-identity-webhook -n openshift-cloud-credential-operator -o yaml
creationTimestamp: "2021-11-19T01:10:11Z"
3. Check the volume for secret pod-identity-webhook in pod yaml
oc get pod pod-identity-webhook-6db9ff4f5b-khzds -n openshift-cloud-credential-operator -o yaml
volumes:
- emptyDir: {}
name: webhook-certs
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056 |
Description of problem: Occasionally, the pod-identity-webhook pod will start before the pod-identity-webhook secret has been created/updated. In a recent ci run: pod-identity-webhook pod log: 2021-11-16T19:55:53.291344432Z E1116 19:55:53.291318 1 store.go:60] Error fetching secret: secrets "pod-identity-webhook" not found ... continues to start anyway pod-identity-webhook secret: creationTimestamp: "2021-11-16T19:56:16Z" I've noticed this much more often in single-node CI jobs.