Bug 2011369

Summary:	[4.9.z] caBundle not updated on kube-admission-webhook pods: etcdserver: request is too large
Product:	Container Native Virtualization (CNV)	Reporter:	Ram Lavi <ralavi>
Component:	Networking	Assignee:	Ram Lavi <ralavi>
Status:	CLOSED ERRATA	QA Contact:	Meni Yakove <myakove>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.9.0	CC:	cnv-qe-bugs, dporter, phoracek, ysegev
Target Milestone:	---
Target Release:	4.9.1
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	kubemacpool-v4.9.1-3 kubernetes-nmstate-handler-v4.9.1-4	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2011376 (view as bug list)		Environment:
Last Closed:	2021-12-13 19:59:01 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2011376, 2011746

Description Ram Lavi 2021-10-06 13:54:44 UTC

Description of problem:
caBundles are stored on the mutatingWebhookConfiguration instance.
Kubemacpool and kubernetes-nmstate components use kube-admission-webhook (https://github.com/qinqon/kube-admission-webhook) pod to rotate ca certs and append them to the appropriate mutatingWebhookConfiguration.
After a large amount of rotations (~700+) then the caBundle appended to mutatingWebhookConfiguration cannot be applied due to etcd's APPLY side restirction (1.5Mb)

This results with the kubemacpool/kubernetes pods to decline requests on 
```
x509: certificate has expired or is not yet valid
```

on the kube-admission-webhook logs one can see that it fails to apply the caBundle:
```
failed rotating all certs: failed adding new CA cert to CA bundle at webhook: failed to update webhook CABundle: failed to update webhook CABundle: etcdserver: request is too large
```


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. manually cause a lot of ca rotations on kubemacpool/kubernetes-nmstate.
2.
3.

Actual results:
VM cannot be created, due to "x509: certificate has expired or is not yet valid"

Expected results:
VM should be able to be created with no errors.

Additional info:

Comment 1 Ram Lavi 2021-10-06 14:08:35 UTC

https://github.com/qinqon/kube-admission-webhook/pull/54

Comment 2 Ram Lavi 2021-10-07 12:27:04 UTC

KAW bump on KMP: https://github.com/k8snetworkplumbingwg/kubemacpool/pull/338

Comment 3 Ram Lavi 2021-10-07 13:29:14 UTC

KAW bump in KNMSTATE: https://github.com/nmstate/kubernetes-nmstate/pull/851

Comment 4 Petr Horáček 2021-10-21 13:02:27 UTC

*** Bug 2011636 has been marked as a duplicate of this bug. ***

Comment 5 Ram Lavi 2021-10-31 21:14:03 UTC

change merged to CNV4.9 since http://cnv-version-explorer.apps.cnv.engineering.redhat.com/BundleDetails?ver=v4.9.0-260

Comment 6 Yossi Segev 2021-11-22 16:34:48 UTC

Verified the following way:
1. I deleted the nmstate secret (in openshift-cnv namespace) infinitely every 5 seconds using a script.
2. I sampled the output of the nmstate mutatingwebhookconfiguration after several iterations.
The output size kept growing, until it remained steady after ~100 iterations. On the point, I verified that the caBundle part in the continued to update.


OCP 4.9.6
CNV 4.9.1
nmstate-handler v4.9.1-7
kubemacpool v4.9.1-3

Comment 12 errata-xmlrpc 2021-12-13 19:59:01 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Virtualization 4.9.1 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:5091