Bug 1635918 - Mutating and Validating Admission Webhooks Won't Enable Without Custom Config
Summary: Mutating and Validating Admission Webhooks Won't Enable Without Custom Config
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.11.z
Assignee: David Eads
QA Contact: Xingxing Xia
Depends On:
TreeView+ depends on / blocked
Reported: 2018-10-03 23:25 UTC by Ben Browning
Modified: 2020-01-07 19:51 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2020-01-07 19:51:32 UTC
Target Upstream Version:

Attachments (Terms of Use)

Description Ben Browning 2018-10-03 23:25:12 UTC
Description of problem:

In testing Knative against prerelease builds of 3.11, I've discovered that mutating and validating admission webhooks can't be enabled by an admissionConfig stanza in kube-apiserver's master-config.yaml such as:

        apiVersion: v1
        disable: false
        kind: DefaultAdmissionConfig
        apiVersion: v1
        disable: false
        kind: DefaultAdmissionConfig

In OCP and OKD 3.10, enabling them like this was enough to get both working. In 3.11, after enabling them and restarting the kube-apiserver, the following log entries are observed:

I1003 20:46:52.920061 75 plugins.go:84] Registered admission plugin "ValidatingAdmissionWebhook"
I1003 20:46:52.920068 75 plugins.go:84] Registered admission plugin "MutatingAdmissionWebhook"
I1003 20:46:52.922508 75 server.go:62] `kube-apiserver [--admission-control-config-file=/tmp/kubeapiserver-admission-config.yaml110003270 --allow-privileged=true --anonymous-auth=false --authorization-mode=RBAC --authorization-mode=Node --bind-address= --client-ca-file=/etc/origin/master/ca.crt --cors-allowed-origins=//127\.0\.0\.1(:|$) --cors-allowed-origins=//127\.0\.0\.1:8943$ --cors-allowed-origins=//localhost(:|$) --enable-admission-plugins=ExternalIPRanger --enable-admission-plugins=MutatingAdmissionWebhook --enable-admission-plugins=ValidatingAdmissionWebhook --enable-admission-plugins=openshift.io/ImagePolicy --enable-admission-plugins=openshift.io/RestrictedEndpointsAdmission --enable-logs-handler=false --enable-swagger-ui=true --endpoint-reconciler-type=lease --etcd-cafile=/etc/origin/master/ca.crt --etcd-certfile=/etc/origin/master/master.etcd-client.crt --etcd-keyfile=/etc/origin/master/master.etcd-client.key --etcd-prefix=openshift.io --etcd-servers= --insecure-port=0 --kubelet-certificate-authority=/etc/origin/master/ca.crt --kubelet-client-certificate=/etc/origin/master/master.kubelet-client.crt --kubelet-client-key=/etc/origin/master/master.kubelet-client.key --kubelet-https=true --kubelet-preferred-address-types=Hostname --kubelet-preferred-address-types=InternalIP --kubelet-preferred-address-types=ExternalIP --kubelet-read-only-port=0 --kubernetes-service-node-port=0 --max-mutating-requests-inflight=600 --max-requests-inflight=1200 --min-request-timeout=3600 --proxy-client-cert-file=/etc/origin/master/openshift-aggregator.crt --proxy-client-key-file=/etc/origin/master/openshift-aggregator.key --requestheader-allowed-names=system:openshift-aggregator --requestheader-client-ca-file=/etc/origin/master/frontproxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=8943 --service-cluster-ip-range= --service-node-port-range=30000-32767 --storage-backend=etcd3 --storage-media-type=application/vnd.kubernetes.protobuf --tls-cert-file=/etc/origin/master/master.server.crt --tls-min-version= --tls-private-key-file=/etc/origin/master/master.server.key]`
I1003 20:46:52.923256 75 flags.go:27] FLAG: --enable-admission-plugins="[ExternalIPRanger,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,openshift.io/ImagePolicy,openshift.io/RestrictedEndpointsAdmission]"
I1003 20:46:53.689281 75 register.go:151] Admission plugin MutatingAdmissionWebhook is not enabled. It will not be started.
I1003 20:46:53.689289 75 register.go:151] Admission plugin ValidatingAdmissionWebhook is not enabled. It will not be started.

As those entries indicate, the changes in my master-config.yaml are being read, the plugins registered, and added to a list of enabled admission plugins when starting the kube-apiserver. But, when the kube-apiserver runs, it thinks the webhook plugins are not enabled and doesn't start them.

This is supported by creating a ValidatingWebhookConfiguration and verifying that it does not get called. In my tests, I just create a dummy ValidatingWebhookConfiguration that references a non-existent Service with a failurePolicy of Fail so that I get immediate notification if the webhook is called or not by the failure message I get when creating an object covered by the webhook's configuration rules.

If you repeat the test by editing the ValidatingAdmissionWebhook section of the master-config.yaml to the value below and restart the kube-apiserver, you'll see  that the ValidatingAdmissionWebhook does get called:

        apiVersion: apiserver.config.k8s.io/v1alpha1
        kubeConfigFile: /dev/null
        kind: WebhookAdmission

It should be possible to enable the admission webhooks as in the first example, without the full configuration. I suspect this bug actually impacts any admission plugin in the list of DefaultOffPlugins at https://github.com/openshift/origin/blob/release-3.11/pkg/cmd/server/origin/admission/register.go#L106-L135 that support being enabled without custom configuration. AlwaysPullImages would be another one to test to see if it actually works in 3.11 when you simple enable it.

Comment 1 Kamesh Sampath 2018-10-04 05:33:48 UTC
IMHO i feel the plugins needed to be supplemented with https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#authenticate-apiservers to make them activated. But I dont find the apiVersion in OpenShift, guess we have an equivalent?

Comment 2 Ben Browning 2018-10-04 11:29:25 UTC
(In reply to Kamesh Sampath from comment #1)
> IMHO i feel the plugins needed to be supplemented with
> https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-
> controllers/#authenticate-apiservers to make them activated. But I dont find
> the apiVersion in OpenShift, guess we have an equivalent?

You'll see at the end of my comment above that is actually the workaround I use to get these enabled. However, that workaround should not be required to enable the admission webhooks.

As I suspected, it seems other plugins in the DefaultOffPlugins list can't be enabled as documented either. I tested locally with AlwaysPullImages plugin and it did not rewrite the ImagePullPolicy on a newly created pod from IfNotPresent to Always.

Comment 3 Nick Stielau 2018-10-04 19:23:52 UTC
Talking with OpenShift teams, I heard "the fixes required to enable by default were explicitly kicked from 3.11.  We didn't have them ready in time and they were too large... It was a large change that swept every admission plugin"  

Still TBD on if this will get in 3.11.z.  I think it's one of many things competing for time to rebase.

Comment 4 Ben Browning 2018-10-04 19:36:34 UTC
Just for clarity, fixing this bug does not require enabling the admission webhooks by default. This bug is that you can't enable the admission webhooks via the normal, documented method of supplying a DefaultAdmissionConfig config stanza with disabled:false. Instead, you have to supply custom WebhookAdmission config pointing to a fake kubeConfigFile to emulate the desired behavior of just enabling them.

See the references to DefaultAdmissionConfig here - https://docs.openshift.com/container-platform/3.10/architecture/additional_concepts/admission_controllers.html#admission-controllers-general-admission-rules

In 3.11, that will not work to enable the plugins. The documented example will not enable the AlwaysPullImages admission plugin, for example. The reason it won't work is due to this bug.

Comment 5 Nick Stielau 2020-01-07 19:51:32 UTC
Closing this as WONTFIX for 3.11.  Mutating and validating admission webhooks were not fully supported in 3.x, and the changeset to address this significant.

Note You need to log in before you can comment on or make changes to this bug.