Bug 2011636
| Summary: | nmstate-ca secret deleted and not recreated upon reinstall | ||
|---|---|---|---|
| Product: | Container Native Virtualization (CNV) | Reporter: | Dirk Porter <dporter> |
| Component: | Networking | Assignee: | Petr Horáček <phoracek> |
| Status: | CLOSED DUPLICATE | QA Contact: | Meni Yakove <myakove> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.9.0 | CC: | aos-bugs, arajapa, cnv-qe-bugs, stirabos |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-10-21 13:02:27 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Dirk Porter
2021-10-06 23:19:55 UTC
looks like an installation issue, moving to install component Directly deleting the namespace was definitely a bad idea, we already have have a validating webhook that should prevent that, but the user removed it removing the subscription then the csv and install plan. This is not a bug by itself. Did you already tried with? oc delete apiservice v1beta1.upload.cdi.kubevirt.io oc delete apiservice v1alpha1.upload.cdi.kubevirt.io oc delete apiservice v1alpha3.kubevirt.io Then removing the finalizer on the kubevirt-hyperconverged CR and then on the namespace? At that point you should be able to remove the namespace. The root cause that initially triggered the issue was `nmstate-ca secret deleted and not recreated upon reinstall`; moving this on network component. The team is looking into it. We believe we have identified the cause - certificate rotation was triggered too rapidly, we were not keeping up with cleaning them up and as a result our secret grew too big and was rejected by etcd. We are looking into our options how to backport this to 2.6. We think we have a workaround script available, but we want to verify it first. The only standing question is what lead into this rapid certificate rotation. This seems to be very similar to https://bugzilla.redhat.com/show_bug.cgi?id=2011369 except that the reference issue was caused by reducing the cert rotation interval. This current issue seems to be triggered by something else. Dirk, it seems that the issue we observe was caused by manual removal of the nmstate-ca Secret. That has triggered a hot-loop of rotating CA bundles which ended up by Kubernetes API rejecting our further attempts to rotate. Could you confirm that the Secret was removed manually? If so, I suppose there was a different issue prior to this, that lead us to the manual intervention. However, we will not be able to address the original issue until we recover the certificate rotation mechanism. In order to do so, please use following script (kudos to ellorent): wget -O force-cert-rotation.sh https://raw.githubusercontent.com/qinqon/kube-admission-webhook/v0.12.0/hack/force-cert-retation.sh chmod +x force-cert-rotation.sh HANDLER_NAMESPACE=openshift-cnv HANDLER_POD_LABEL=component=kubernetes-nmstate-webhook WEBHOOK_CONFIG_NAME=nmstate WEBHOOK_SECRET_NAME=nmstate-ca ./force-cert-rotation.sh Let me know if there would be any issues applying it. Petr, Yes the secret was deleted manually after reviewing the certificate and realizing that the certificate was expired. I have a call with the customer shortly and will try to use the script. Thank you! Regards, Dirk Porter Hello Petr,
After trying to run the script provided we are encountering the below errors. I believe it is because of the state of the operator, I believe we may need to remove or add some components to get back to a good state.
[corona.azusca21.vzwops.com ~]$ HANDLER_NAMESPACE=openshift-cnv HANDLER_POD_LABEL=component=kubernetes-nmstate-webhook WEBHOOK_CONFIG_NAME=nmstate WEBHOOK_SECRET_NAME=nmstate-ca ./force-cert-rotation.sh
+ KUBECTL=kubectl
+ parse_input_param
+ [[ ! -z '' ]]
+ check_input_env_var
+ [[ -z openshift-cnv ]]
+ [[ -z component=kubernetes-nmstate-webhook ]]
+ [[ -z nmstate ]]
+ [[ -z nmstate-ca ]]
+ for WEBHOOK_TYPE in "validationWebhookConfiguration" "mutatingWebhookConfiguration"
++ get_number_of_webhooks_in_webhook_config
+++ wc -w
+++ kubectl get validationWebhookConfiguration -n openshift-cnv nmstate -o 'jsonpath={.webhooks[*].name}'
I1007 14:13:36.676133 2073804 request.go:655] Throttling request took 1.157105718s, request: GET:https://api.kub1.azusca21.vzwops.com:6443/apis/networking.istio.io/v1alpha3?timeout=32s
error: the server doesn't have a resource type "validationWebhookConfiguration"
++ echo 0
+ number_of_webhooks=0
+ [[ 0 > 0 ]]
+ for WEBHOOK_TYPE in "validationWebhookConfiguration" "mutatingWebhookConfiguration"
++ get_number_of_webhooks_in_webhook_config
+++ wc -w
+++ kubectl get mutatingWebhookConfiguration -n openshift-cnv nmstate -o 'jsonpath={.webhooks[*].name}'
Error from server (NotFound): mutatingwebhookconfigurations.admissionregistration.k8s.io "nmstate" not found
++ echo 0
+ number_of_webhooks=0
+ [[ 0 > 0 ]]
[corona.azusca21.vzwops.com ~]$
[corona.azusca21.vzwops.com ~]$
[corona.azusca21.vzwops.com ~]$
[corona.azusca21.vzwops.com ~]$
[corona.azusca21.vzwops.com ~]$ oc get validationWebhookConfiguration
error: the server doesn't have a resource type "validationWebhookConfiguration"
[corona.azusca21.vzwops.com ~]$ oc get validationWebhookConfiguration -A
error: the server doesn't have a resource type "validationWebhookConfiguration"
[corona.azusca21.vzwops.com ~]$
[corona.azusca21.vzwops.com ~]$ kubectl get validationWebhookConfiguration
I1007 14:15:02.715086 2081406 request.go:655] Throttling request took 1.154036402s, request: GET:https://api.kub1.azusca21.vzwops.com:6443/apis/security.openshift.io/v1?timeout=32s
error: the server doesn't have a resource type "validationWebhookConfiguration"
[corona.azusca21.vzwops.com ~]$
[corona.azusca21.vzwops.com ~]$
[corona.azusca21.vzwops.com ~]$
[corona.azusca21.vzwops.com ~]$ oc get mutatingwebhookconfiguration -A
NAME WEBHOOKS AGE
aspen-mesh-cert-manager-webhook 1 138d
cdi-api-datavolume-mutate 1 138d
istio-sidecar-injector 1 138d
istio-sidecar-injector-canary 1 43h
machine-api 2 138d
mutate-ns-hco.kubevirt.io-68ktp 1 16h
network-resources-injector-config 1 138d
sriov-operator-webhook-config 1 36h
virt-api-mutator 3 16h
[corona.azusca21.vzwops.com ~]$ oc get validatingwebhookconfiguration -A
NAME WEBHOOKS AGE
aspen-mesh-cert-manager-webhook 1 138d
aspen-mesh-controlplane 2 138d
aspen-mesh-secure-ingress 1 43h
autoscaling.openshift.io 2 138d
cdi-api-datavolume-validate 1 138d
cdi-api-validate 1 138d
istiod-istio-system 1 43h
machine-api 2 138d
multus.openshift.io 1 138d
nodemaintenance-validation.kubevirt.io-2rzbx 1 16h
prometheusrules.openshift.io 1 138d
snapshot.storage.k8s.io 1 40h
sriov-operator-webhook-config 1 36h
traffic-claim-enforcer 1 43h
validate-hco.kubevirt.io-zsqb5 1 16h
validation-dns-updater-webhook-cfg 1 138d
virt-api-validator 11 16h
virt-operator-validator 2 16h
vssp.kb.io-8xrj5 1 16h
[corona.azusca21.vzwops.com ~]$
[corona.azusca21.vzwops.com ~]$
[corona.azusca21.vzwops.com ~]$
[corona.azusca21.vzwops.com ~]$ oc get validatingwebhookconfiguration -A | grep nmstate
*** This bug has been marked as a duplicate of bug 2011369 *** |