Bug 2024613 - pod-identity-webhook starts without tls
Summary: pod-identity-webhook starts without tls
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Credential Operator
Version: 4.10
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.10.0
Assignee: Luis Sanchez
QA Contact: Jianping SHu
URL:
Whiteboard:
Depends On:
Blocks: 2024751
TreeView+ depends on / blocked
 
Reported: 2021-11-18 13:47 UTC by Luis Sanchez
Modified: 2022-03-10 16:29 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-10 16:29:22 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cloud-credential-operator pull 421 0 None open Bug 2024613: pod-identity-webhook starts without tls 2021-11-18 17:58:37 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:29:38 UTC

Description Luis Sanchez 2021-11-18 13:47:07 UTC
Description of problem:

Occasionally, the pod-identity-webhook pod will start before the pod-identity-webhook secret has been created/updated.

In a recent ci run:

pod-identity-webhook pod log:

2021-11-16T19:55:53.291344432Z E1116 19:55:53.291318       1 store.go:60] Error fetching secret: secrets "pod-identity-webhook" not found
... continues to start anyway

pod-identity-webhook secret:
    creationTimestamp: "2021-11-16T19:56:16Z"

I've noticed this much more often in single-node CI jobs.

Comment 3 Jianping SHu 2021-11-19 04:12:57 UTC
Verified with 4.10.0-0.nightly-2021-11-18-225109
1. Check pod-identity-webhook pod log, no "Error fetching secret"  
W1119 02:44:16.958796       1 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1119 02:44:16.961134       1 main.go:191] Creating server
I1119 02:44:16.961299       1 main.go:211] Listening on :9999 for metrics and healthz
I1119 02:44:16.961493       1 main.go:205] Listening on :6443

2. Check creationTimestamp of secret pod-identity-webhook is before that of "Creating server" in log  
oc get secret pod-identity-webhook -n openshift-cloud-credential-operator -o yaml
  creationTimestamp: "2021-11-19T02:42:36Z"

3. Check the volume for secret pod-identity-webhook in pod yaml  
oc get pod pod-identity-webhook-55bf4bbbfd-ntztx -n openshift-cloud-credential-operator -o yaml
  volumes:
  - name: webhook-certs
    secret:
      defaultMode: 420
      secretName: pod-identity-webhook



Reproduced with 4.10.0-0.nightly-2021-11-15-034648
1. Check pod-identity-webhook pod log, no "Error fetching secret"  
W1119 01:10:16.225018       1 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1119 01:10:16.237606       1 store.go:63] Fetched secret: openshift-cloud-credential-operator/pod-identity-webhook
I1119 01:10:16.237741       1 main.go:191] Creating server
I1119 01:10:16.238255       1 main.go:211] Listening on :9999 for metrics and healthz
I1119 01:10:16.238517       1 main.go:205] Listening on :6443
E1119 01:10:16.240030       1 certificate_manager.go:437] Failed while requesting a signed certificate from the master: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:serviceaccount:openshift-cloud-credential-operator:pod-identity-webhook" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope
E1119 01:10:18.362758       1 certificate_manager.go:437] Failed while requesting a signed certificate from the master: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:serviceaccount:openshift-cloud-credential-operator:pod-identity-webhook" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope
E1119 01:10:22.631129       1 certificate_manager.go:437] Failed while requesting a signed certificate from the master: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:serviceaccount:openshift-cloud-credential-operator:pod-identity-webhook" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope

2. Check creationTimestamp of secret pod-identity-webhook is before that of "Creating server" in log  
oc get secret pod-identity-webhook -n openshift-cloud-credential-operator -o yaml
  creationTimestamp: "2021-11-19T01:10:11Z"

3. Check the volume for secret pod-identity-webhook in pod yaml  
oc get pod pod-identity-webhook-6db9ff4f5b-khzds -n openshift-cloud-credential-operator -o yaml
  volumes:
  - emptyDir: {}
    name: webhook-certs

Comment 6 errata-xmlrpc 2022-03-10 16:29:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.