Bug 1889873

Summary: MutatingAdmissionWebhook causing pod deployment failures and other errors
Product: OpenShift Container Platform Reporter: Luke Stanton <lstanton>
Component: kube-controller-managerAssignee: Maciej Szulik <maszulik>
Status: CLOSED INSUFFICIENT_DATA QA Contact: zhou ying <yinzhou>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.3.zCC: aos-bugs, mfojtik
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-05 14:37:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Luke Stanton 2020-10-20 18:45:30 UTC
Description of problem:

The addition of a MutatingAdmissionWebhook is causing pod deployment failures and other errors when running commands such as the following:

~~~~~~~~~~
$ oc debug node/...com
Error from server (InternalError): Internal error occurred: admission plugin "MutatingAdmissionWebhook" failed to complete mutation in 13s
~~~~~~~~~~


~~~~~~~~~~
2020-10-16T16:21:17.357064063Z I1016 16:21:17.357024       1 event.go:255] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"openshift-operator-lifecycle-manager", Name:"olm-operator-9d666b5b9", UID:"69c72a6d-f1b7-49b6-bc45-9e611c6bffda", APIVersion:"apps/v1", ResourceVersion:"142186292", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: Internal error occurred: admission plugin "MutatingAdmissionWebhook" failed to complete mutation in 13s
2020-10-16T16:21:17.357112317Z E1016 16:21:17.357103       1 replica_set.go:534] sync "openshift-operator-lifecycle-manager/olm-operator-9d666b5b9" failed with Internal error occurred: admission plugin "MutatingAdmissionWebhook" failed to complete mutation in 13s
~~~~~~~~~~


How reproducible:

Consistently


Actual results:

Adding MutatingAdmissionWebhook is causing problems for cluster components.


Expected results:

MutatingAdmissionWebhook would work as expected without adverse affects on the cluster.

Comment 1 Luke Stanton 2020-10-20 18:47:16 UTC
Meant to indicate the following error in the summary is from the kube-controller-manager pods:

~~~~~~~~~~
2020-10-16T16:21:17.357064063Z I1016 16:21:17.357024       1 event.go:255] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"openshift-operator-lifecycle-manager", Name:"olm-operator-9d666b5b9", UID:"69c72a6d-f1b7-49b6-bc45-9e611c6bffda", APIVersion:"apps/v1", ResourceVersion:"142186292", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: Internal error occurred: admission plugin "MutatingAdmissionWebhook" failed to complete mutation in 13s
2020-10-16T16:21:17.357112317Z E1016 16:21:17.357103       1 replica_set.go:534] sync "openshift-operator-lifecycle-manager/olm-operator-9d666b5b9" failed with Internal error occurred: admission plugin "MutatingAdmissionWebhook" failed to complete mutation in 13s
~~~~~~~~~~

Comment 7 Maciej Szulik 2020-11-13 11:03:58 UTC
I’m adding UpcomingSprint, because I was occupied by fixing bugs with higher priority/severity, developing new features with higher priority, or developing new features to improve stability at a macro level. I will revisit this bug next sprint.

Comment 8 Maciej Szulik 2020-12-04 16:29:54 UTC
I’m adding UpcomingSprint, because I was occupied by fixing bugs with higher priority/severity, developing new features with higher priority, or developing new features to improve stability at a macro level. I will revisit this bug next sprint.

Comment 9 Maciej Szulik 2021-01-15 13:43:28 UTC
I’m adding UpcomingSprint, because I was occupied by fixing bugs with higher priority/severity, developing new features with higher priority, or developing new features to improve stability at a macro level. I will revisit this bug next sprint.

Comment 10 Maciej Szulik 2021-02-05 14:37:54 UTC
The linked customer issue is closed and given this was most likely caused by a 3rd party mutating admission webhook I'm going to close this issue. 
Feel free to re-open if this re-appears.