A serving cert supplied by the service ca operator appears to be used to secure the healthz/metrics endpoint of the kube scheduler. If the serving cert is regenerated (i.e. when the service CA is rotated), it does not appear that the healthz/metrics endpoint will be refreshed or the scheduler restarted. This could result in a broken healthz/metrics endpoint. The 'Refresh Strategies' section of the linked compatibility doc catalogs potential strategies for responding to changes in key material supplied by the service CA operator. Note that CA rotation can be manually triggered in any 4.x release by removing the signing secret. Automated rotation is likely to be introduced in a future z-stream release. References: Enhancement for automated service CA rotation: https://github.com/openshift/enhancements/blob/master/enhancements/automated-service-ca-rotation.md Operator compatibility with service ca rotation: https://docs.google.com/document/d/1NB2wUf9e8XScfVM6jFBl8VuLYG6-3uV63eUpqmYE8Ts/edit
KS is correctly picking up the new certs. The flow to follow is: When starting the operator we pass a set of resources that are maintained using revision controller from library-go: https://github.com/openshift/cluster-kube-scheduler-operator/blob/master/pkg/operator/starter.go#L106 those secrets are defined in https://github.com/openshift/cluster-kube-scheduler-operator/blob/master/pkg/operator/starter.go#L157-L170 one of them is serving-cert which is managed by the service-serving-cert-signer controller. The code responsible for updating pod with the new cert lives in: https://github.com/openshift/cluster-kube-scheduler-operator/blob/master/pkg/operator/target_config_reconciler_v410_00.go#L126-L131 I've manually performed the test but an automatic one which would be verifying metrics endpoint is needed. Sally can you add an end-to-end test for ks-o which will be checking one of the scheduler metrics, scheduler_scheduling_duration_seconds_sum for example. This will: 1. Verify the metrics are properly served by the ks. 2. Verify the metrics are served even when the cert is rotated. Marun will be working on a separate test suite that forces rotation and we need a test proving it's working as expected. The test should be as follows: 1. check the current values of scheduler_scheduling_duration_seconds_sum (or other of your choosing) 2. schedule some test application, a pod, a simple deployment 3. check values of scheduler_scheduling_duration_seconds_sum (the same as in 1) and compare - they should differ. You may want to sync with Mike about which metric to pick other than scheduler_scheduling_duration_seconds_sum. If in doubt check Mike's latest e2e here: https://github.com/openshift/cluster-kube-controller-manager-operator/pull/311
My apologies, I was previously tracing the wrong path. SecureServingWithLoopback ensures the use of NewDynamicServingContentFromFiles. https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/server/options/serving.go#L229 https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/server/options/serving_with_loopback.go#L44 https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-scheduler/app/options/options.go#L188 https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-scheduler/app/server.go#L138
This will be fixed in 4.4, moving accordingly.
Created attachment 1657375 [details] Before pod Scheduling
Created attachment 1657376 [details] After pod scheduling
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581