Description of problem:
In testing Knative against prerelease builds of 3.11, I've discovered that mutating and validating admission webhooks can't be enabled by an admissionConfig stanza in kube-apiserver's master-config.yaml such as:
In OCP and OKD 3.10, enabling them like this was enough to get both working. In 3.11, after enabling them and restarting the kube-apiserver, the following log entries are observed:
I1003 20:46:52.920061 75 plugins.go:84] Registered admission plugin "ValidatingAdmissionWebhook"
I1003 20:46:52.920068 75 plugins.go:84] Registered admission plugin "MutatingAdmissionWebhook"
I1003 20:46:52.922508 75 server.go:62] `kube-apiserver [--admission-control-config-file=/tmp/kubeapiserver-admission-config.yaml110003270 --allow-privileged=true --anonymous-auth=false --authorization-mode=RBAC --authorization-mode=Node --bind-address=0.0.0.0 --client-ca-file=/etc/origin/master/ca.crt --cors-allowed-origins=//127\.0\.0\.1(:|$) --cors-allowed-origins=//127\.0\.0\.1:8943$ --cors-allowed-origins=//localhost(:|$) --enable-admission-plugins=ExternalIPRanger --enable-admission-plugins=MutatingAdmissionWebhook --enable-admission-plugins=ValidatingAdmissionWebhook --enable-admission-plugins=openshift.io/ImagePolicy --enable-admission-plugins=openshift.io/RestrictedEndpointsAdmission --enable-logs-handler=false --enable-swagger-ui=true --endpoint-reconciler-type=lease --etcd-cafile=/etc/origin/master/ca.crt --etcd-certfile=/etc/origin/master/master.etcd-client.crt --etcd-keyfile=/etc/origin/master/master.etcd-client.key --etcd-prefix=openshift.io --etcd-servers=https://127.0.0.1:4001 --insecure-port=0 --kubelet-certificate-authority=/etc/origin/master/ca.crt --kubelet-client-certificate=/etc/origin/master/master.kubelet-client.crt --kubelet-client-key=/etc/origin/master/master.kubelet-client.key --kubelet-https=true --kubelet-preferred-address-types=Hostname --kubelet-preferred-address-types=InternalIP --kubelet-preferred-address-types=ExternalIP --kubelet-read-only-port=0 --kubernetes-service-node-port=0 --max-mutating-requests-inflight=600 --max-requests-inflight=1200 --min-request-timeout=3600 --proxy-client-cert-file=/etc/origin/master/openshift-aggregator.crt --proxy-client-key-file=/etc/origin/master/openshift-aggregator.key --requestheader-allowed-names=system:openshift-aggregator --requestheader-client-ca-file=/etc/origin/master/frontproxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=8943 --service-cluster-ip-range=172.30.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --storage-media-type=application/vnd.kubernetes.protobuf --tls-cert-file=/etc/origin/master/master.server.crt --tls-min-version= --tls-private-key-file=/etc/origin/master/master.server.key]`
I1003 20:46:52.923256 75 flags.go:27] FLAG: --enable-admission-plugins="[ExternalIPRanger,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,openshift.io/ImagePolicy,openshift.io/RestrictedEndpointsAdmission]"
I1003 20:46:53.689281 75 register.go:151] Admission plugin MutatingAdmissionWebhook is not enabled. It will not be started.
I1003 20:46:53.689289 75 register.go:151] Admission plugin ValidatingAdmissionWebhook is not enabled. It will not be started.
As those entries indicate, the changes in my master-config.yaml are being read, the plugins registered, and added to a list of enabled admission plugins when starting the kube-apiserver. But, when the kube-apiserver runs, it thinks the webhook plugins are not enabled and doesn't start them.
This is supported by creating a ValidatingWebhookConfiguration and verifying that it does not get called. In my tests, I just create a dummy ValidatingWebhookConfiguration that references a non-existent Service with a failurePolicy of Fail so that I get immediate notification if the webhook is called or not by the failure message I get when creating an object covered by the webhook's configuration rules.
If you repeat the test by editing the ValidatingAdmissionWebhook section of the master-config.yaml to the value below and restart the kube-apiserver, you'll see that the ValidatingAdmissionWebhook does get called:
It should be possible to enable the admission webhooks as in the first example, without the full configuration. I suspect this bug actually impacts any admission plugin in the list of DefaultOffPlugins at https://github.com/openshift/origin/blob/release-3.11/pkg/cmd/server/origin/admission/register.go#L106-L135 that support being enabled without custom configuration. AlwaysPullImages would be another one to test to see if it actually works in 3.11 when you simple enable it.
IMHO i feel the plugins needed to be supplemented with https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#authenticate-apiservers to make them activated. But I dont find the apiVersion in OpenShift, guess we have an equivalent?
(In reply to Kamesh Sampath from comment #1)
> IMHO i feel the plugins needed to be supplemented with
> controllers/#authenticate-apiservers to make them activated. But I dont find
> the apiVersion in OpenShift, guess we have an equivalent?
You'll see at the end of my comment above that is actually the workaround I use to get these enabled. However, that workaround should not be required to enable the admission webhooks.
As I suspected, it seems other plugins in the DefaultOffPlugins list can't be enabled as documented either. I tested locally with AlwaysPullImages plugin and it did not rewrite the ImagePullPolicy on a newly created pod from IfNotPresent to Always.
Talking with OpenShift teams, I heard "the fixes required to enable by default were explicitly kicked from 3.11. We didn't have them ready in time and they were too large... It was a large change that swept every admission plugin"
Still TBD on if this will get in 3.11.z. I think it's one of many things competing for time to rebase.
Just for clarity, fixing this bug does not require enabling the admission webhooks by default. This bug is that you can't enable the admission webhooks via the normal, documented method of supplying a DefaultAdmissionConfig config stanza with disabled:false. Instead, you have to supply custom WebhookAdmission config pointing to a fake kubeConfigFile to emulate the desired behavior of just enabling them.
See the references to DefaultAdmissionConfig here - https://docs.openshift.com/container-platform/3.10/architecture/additional_concepts/admission_controllers.html#admission-controllers-general-admission-rules
In 3.11, that will not work to enable the plugins. The documented example will not enable the AlwaysPullImages admission plugin, for example. The reason it won't work is due to this bug.
Closing this as WONTFIX for 3.11. Mutating and validating admission webhooks were not fully supported in 3.x, and the changeset to address this significant.