Description of problem:

In testing Knative against prerelease builds of 3.11, I've discovered that mutating and validating admission webhooks can't be enabled by an admissionConfig stanza in kube-apiserver's master-config.yaml such as:

admissionConfig:
  pluginConfig:
    MutatingAdmissionWebhook:
      configuration:
        apiVersion: v1
        disable: false
        kind: DefaultAdmissionConfig
    ValidatingAdmissionWebhook:
      configuration:
        apiVersion: v1
        disable: false
        kind: DefaultAdmissionConfig

In OCP and OKD 3.10, enabling them like this was enough to get both working. In 3.11, after enabling them and restarting the kube-apiserver, the following log entries are observed:

I1003 20:46:52.920061 75 plugins.go:84] Registered admission plugin "ValidatingAdmissionWebhook"
I1003 20:46:52.920068 75 plugins.go:84] Registered admission plugin "MutatingAdmissionWebhook"
I1003 20:46:52.922508 75 server.go:62] `kube-apiserver [--admission-control-config-file=/tmp/kubeapiserver-admission-config.yaml110003270 --allow-privileged=true --anonymous-auth=false --authorization-mode=RBAC --authorization-mode=Node --bind-address=0.0.0.0 --client-ca-file=/etc/origin/master/ca.crt --cors-allowed-origins=//127\.0\.0\.1(:|$) --cors-allowed-origins=//127\.0\.0\.1:8943$ --cors-allowed-origins=//localhost(:|$) --enable-admission-plugins=ExternalIPRanger --enable-admission-plugins=MutatingAdmissionWebhook --enable-admission-plugins=ValidatingAdmissionWebhook --enable-admission-plugins=openshift.io/ImagePolicy --enable-admission-plugins=openshift.io/RestrictedEndpointsAdmission --enable-logs-handler=false --enable-swagger-ui=true --endpoint-reconciler-type=lease --etcd-cafile=/etc/origin/master/ca.crt --etcd-certfile=/etc/origin/master/master.etcd-client.crt --etcd-keyfile=/etc/origin/master/master.etcd-client.key --etcd-prefix=openshift.io --etcd-servers=https://127.0.0.1:4001 --insecure-port=0 --kubelet-certificate-authority=/etc/origin/master/ca.crt --kubelet-client-certificate=/etc/origin/master/master.kubelet-client.crt --kubelet-client-key=/etc/origin/master/master.kubelet-client.key --kubelet-https=true --kubelet-preferred-address-types=Hostname --kubelet-preferred-address-types=InternalIP --kubelet-preferred-address-types=ExternalIP --kubelet-read-only-port=0 --kubernetes-service-node-port=0 --max-mutating-requests-inflight=600 --max-requests-inflight=1200 --min-request-timeout=3600 --proxy-client-cert-file=/etc/origin/master/openshift-aggregator.crt --proxy-client-key-file=/etc/origin/master/openshift-aggregator.key --requestheader-allowed-names=system:openshift-aggregator --requestheader-client-ca-file=/etc/origin/master/frontproxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=8943 --service-cluster-ip-range=172.30.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --storage-media-type=application/vnd.kubernetes.protobuf --tls-cert-file=/etc/origin/master/master.server.crt --tls-min-version= --tls-private-key-file=/etc/origin/master/master.server.key]`
I1003 20:46:52.923256 75 flags.go:27] FLAG: --enable-admission-plugins="[ExternalIPRanger,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,openshift.io/ImagePolicy,openshift.io/RestrictedEndpointsAdmission]"
I1003 20:46:53.689281 75 register.go:151] Admission plugin MutatingAdmissionWebhook is not enabled. It will not be started.
I1003 20:46:53.689289 75 register.go:151] Admission plugin ValidatingAdmissionWebhook is not enabled. It will not be started.



As those entries indicate, the changes in my master-config.yaml are being read, the plugins registered, and added to a list of enabled admission plugins when starting the kube-apiserver. But, when the kube-apiserver runs, it thinks the webhook plugins are not enabled and doesn't start them.

This is supported by creating a ValidatingWebhookConfiguration and verifying that it does not get called. In my tests, I just create a dummy ValidatingWebhookConfiguration that references a non-existent Service with a failurePolicy of Fail so that I get immediate notification if the webhook is called or not by the failure message I get when creating an object covered by the webhook's configuration rules.


If you repeat the test by editing the ValidatingAdmissionWebhook section of the master-config.yaml to the value below and restart the kube-apiserver, you'll see  that the ValidatingAdmissionWebhook does get called:

    ValidatingAdmissionWebhook:
      configuration:
        apiVersion: apiserver.config.k8s.io/v1alpha1
        kubeConfigFile: /dev/null
        kind: WebhookAdmission


It should be possible to enable the admission webhooks as in the first example, without the full configuration. I suspect this bug actually impacts any admission plugin in the list of DefaultOffPlugins at https://github.com/openshift/origin/blob/release-3.11/pkg/cmd/server/origin/admission/register.go#L106-L135 that support being enabled without custom configuration. AlwaysPullImages would be another one to test to see if it actually works in 3.11 when you simple enable it.