Bug 2011376

Summary: [4.8.z] caBundle not updated on kube-admission-webhook pods: etcdserver: request is too large
Product: Container Native Virtualization (CNV) Reporter: Ram Lavi <ralavi>
Component: NetworkingAssignee: Petr Horáček <phoracek>
Status: CLOSED ERRATA QA Contact: Yossi Segev <ysegev>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.8.0CC: cnv-qe-bugs, gkapoor, myakove, ysegev
Target Milestone: ---   
Target Release: 4.8.4   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: v4.8.4-20 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2011369
: 2011746 (view as bug list) Environment:
Last Closed: 2022-01-20 17:21:21 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2011369    
Bug Blocks: 2011746    

Description Ram Lavi 2021-10-06 14:09:57 UTC 
+++ This bug was initially created as a clone of Bug #2011369 +++

Description of problem:
caBundles are stored on the mutatingWebhookConfiguration instance.
Kubemacpool and kubernetes-nmstate components use kube-admission-webhook (https://github.com/qinqon/kube-admission-webhook) pod to rotate ca certs and append them to the appropriate mutatingWebhookConfiguration.
After a large amount of rotations (~700+) then the caBundle appended to mutatingWebhookConfiguration cannot be applied due to etcd's APPLY side restirction (1.5Mb)

This results with the kubemacpool/kubernetes pods to decline requests on 
```
x509: certificate has expired or is not yet valid
```

on the kube-admission-webhook logs one can see that it fails to apply the caBundle:
```
failed rotating all certs: failed adding new CA cert to CA bundle at webhook: failed to update webhook CABundle: failed to update webhook CABundle: etcdserver: request is too large
```


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. manually cause a lot of ca rotations on kubemacpool/kubernetes-nmstate.
2.
3.

Actual results:
VM cannot be created, due to "x509: certificate has expired or is not yet valid"

Expected results:
VM should be able to be created with no errors.

Additional info:

--- Additional comment from Ram Lavi on 2021-10-06 14:08:35 UTC ---

https://github.com/qinqon/kube-admission-webhook/pull/54
Comment 1 Ram Lavi 2021-10-06 14:12:36 UTC 
https://github.com/qinqon/kube-admission-webhook/pull/54
Comment 2 Ram Lavi 2021-10-07 12:26:45 UTC 
KAW bump on KMP:  https://github.com/k8snetworkplumbingwg/kubemacpool/pull/339
Comment 3 Ram Lavi 2021-10-07 13:28:27 UTC 
KAW bump in KNMSTATE: https://github.com/nmstate/kubernetes-nmstate/pull/850
Comment 4 Petr Horáček 2021-11-03 10:36:14 UTC 
@ralavi are all the needed patches merged on U/S CNAO stable branch? We should be prepared, so the only missing thing to get this to D/S would be to merge a M/S patch.
Comment 5 Yossi Segev 2021-12-22 15:06:32 UTC 
Verified by running the following scenario:
1. Check the preliminary contents (specifically the size) of the nmstate mutatingwebhookconfiguration resource:
$ oc describe mutatingwebhookconfiguration nmstate > mutatingwebhookconfiguration-nmstate.orig
$ ll mutatingwebhookconfiguration-nmstate.orig 
-rw-rw-r--. 1 cnv-qe-jenkins cnv-qe-jenkins 12821 Dec 22 11:06 mutatingwebhookconfiguration-nmstate.orig

2. By running the attached script - I deleted the nmstate-ca secret (in openshift-cnv namespace) periodically every 10 seconds, for 100 iterations.
 The script includes storing the contents of the mutatingwebhookconfiguration resource in a file, every 10 iterations.

3. After the script finished - I checked the progress on the size of the mutatingwebhookconfiguration resource:
$ ll -tr
total 4048
-rw-rw-r--. 1 cnv-qe-jenkins cnv-qe-jenkins  12821 Dec 22 11:06 mutatingwebhookconfiguration-nmstate.orig
-rw-rw-r--. 1 cnv-qe-jenkins cnv-qe-jenkins  18470 Dec 22 11:47 mutatingwebhookconfiguration-nmstate.0
-rw-rw-r--. 1 cnv-qe-jenkins cnv-qe-jenkins  80535 Dec 22 11:49 mutatingwebhookconfiguration-nmstate.10
-rw-rw-r--. 1 cnv-qe-jenkins cnv-qe-jenkins 142599 Dec 22 11:50 mutatingwebhookconfiguration-nmstate.20
-rw-rw-r--. 1 cnv-qe-jenkins cnv-qe-jenkins 199031 Dec 22 11:52 mutatingwebhookconfiguration-nmstate.30
-rw-rw-r--. 1 cnv-qe-jenkins cnv-qe-jenkins 255463 Dec 22 11:54 mutatingwebhookconfiguration-nmstate.40
-rw-rw-r--. 1 cnv-qe-jenkins cnv-qe-jenkins 323175 Dec 22 11:55 mutatingwebhookconfiguration-nmstate.50
-rw-rw-r--. 1 cnv-qe-jenkins cnv-qe-jenkins 385239 Dec 22 11:57 mutatingwebhookconfiguration-nmstate.60
-rw-rw-r--. 1 cnv-qe-jenkins cnv-qe-jenkins 447303 Dec 22 11:59 mutatingwebhookconfiguration-nmstate.70
-rw-rw-r--. 1 cnv-qe-jenkins cnv-qe-jenkins 515015 Dec 22 12:00 mutatingwebhookconfiguration-nmstate.80
-rw-rw-r--. 1 cnv-qe-jenkins cnv-qe-jenkins 571448 Dec 22 12:02 mutatingwebhookconfiguration-nmstate.90
-rw-rw-r--. 1 cnv-qe-jenkins cnv-qe-jenkins 571448 Dec 22 12:04 mutatingwebhookconfiguration-nmstate.100
-rw-rw-r--. 1 cnv-qe-jenkins cnv-qe-jenkins 571448 Dec 22 12:06 mutatingwebhookconfiguration-nmstate.110

As can be seen - the resource size kept increasing, until after 80-90 iterations t remained steady.

4. I created a simple VM, and verified it is running, to ensure this CA rotation didn't break the CNV functionality.

OCP version: 4.10.0-202112011912.p0.g69c4d3f.assembly.stream-69c4d3f
CNV Version: 4.8.4 (HCO Version: v4.8.4-20, taken from the deployment job console output)
nmstate-handler: v4.8.4-3
kubemacpool: v4.8.4-3
Comment 12 errata-xmlrpc 2022-01-20 17:21:21 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Virtualization 4.8.4 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0213