Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: During the WS 1.3.2-1 upgrade, CNV is also upgrade. Ran into an issue where the nmstate certs were expired and cnv install failed. Tried to clean up by deleting the subscription then the csv and install plan. Then reapplied the sub. But that did not clean up secrets. So repeated the process of delete and this time we delete the namespace. But VMI resources had not been stopped. This has resulted in the namespace being stuck in terminating. Removing finalizers etc has not helped. When trying to delete to reinstall: [corona.azusca21.vzwops.com tmp]$ oc delete hyperconverged -n openshift-cnv kubevirt-hyperconverged --force --grace-period 0 warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (Internal error occurred: failed calling webhook "cdi-validate.cdi.kubevirt.io": Post "https://cdi-api.openshift-cnv.svc:443/cdi-validate?timeout=30s": service "cdi-api" not found): admission webhook "validate-hco.kubevirt.io" denied the request: Internal error occurred: failed calling webhook "cdi-validate.cdi.kubevirt.io": Post "https://cdi-api.openshift-cnv.svc:443/cdi-validate?timeout=30s": service "cdi-api" not found [corona.azusca21.vzwops.com ~]$ [corona.azusca21.vzwops.com ~]$ oc get secrets -n openshift-cnv | grep nmstate nmstate-handler-dockercfg-d44zp kubernetes.io/dockercfg 1 75m nmstate-handler-token-7h44b kubernetes.io/service-account-token 4 75m nmstate-handler-token-sp52w kubernetes.io/service-account-token 4 75m nmstate-webhook kubernetes.io/tls 2 75m [corona.azusca21.vzwops.com ~]$ oc get secrets -n openshift-cnv -oyaml nmstate-webhook apiVersion: v1 data: tls.crt: YmFkIGNlcnRpZmljYXRlCg== tls.key: YmFkIGtleQo= kind: Secret metadata: annotations: networkaddonsoperator.network.kubevirt.io/rejectOwner: "" creationTimestamp: "2021-10-06T20:35:16Z" labels: networkaddonsoperator.network.kubevirt.io/version: sha256_e7c5148e1428dee26585ceb95273d4ea48396911d22858a74b483457 managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:data: .: {} f:tls.crt: {} f:tls.key: {} f:metadata: f:annotations: .: {} f:networkaddonsoperator.network.kubevirt.io/rejectOwner: {} f:labels: .: {} f:networkaddonsoperator.network.kubevirt.io/version: {} f:type: {} manager: cluster-network-addons-operator operation: Update time: "2021-10-06T20:35:16Z" name: nmstate-webhook namespace: openshift-cnv resourceVersion: "281490685" selfLink: /api/v1/namespaces/openshift-cnv/secrets/nmstate-webhook uid: 6d8d8c10-6786-4b5c-b07e-0eeb4c146017 type: kubernetes.io/tls [corona.azusca21.vzwops.com ~]$ echo "YmFkIGNlcnRpZmljYXRlCg==" | base64 -d bad certificate [corona.azusca21.vzwops.com ~]$ echo "YmFkIGtleQo=" | base64 -d bad key [corona.azusca21.vzwops.com ~]$ oc describe project openshift-cnv Name: openshift-cnv Created: 4 months ago Labels: control-plane=mac-controller-manager name=openshift-cnv networkaddonsoperator.network.kubevirt.io/version=sha256_e7c5148e1428dee26585ceb95273d4ea48396911d22858a74b483457 olm.operatorgroup.uid/b2427400-4359-44b9-98bf-9468141e4542= openshift.io/cluster-monitoring=true Annotations: openshift.io/display-name= openshift.io/sa.scc.mcs=s0:c24,c19 openshift.io/sa.scc.supplemental-groups=1000590000/10000 openshift.io/sa.scc.uid-range=1000590000/10000 Display Name: <none> Description: <none> Status: Terminating Node Selector: <none> Quota: <none> Resource limits: <none> [corona.azusca21.vzwops.com ~]$ Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
looks like an installation issue, moving to install component
Directly deleting the namespace was definitely a bad idea, we already have have a validating webhook that should prevent that, but the user removed it removing the subscription then the csv and install plan. This is not a bug by itself. Did you already tried with? oc delete apiservice v1beta1.upload.cdi.kubevirt.io oc delete apiservice v1alpha1.upload.cdi.kubevirt.io oc delete apiservice v1alpha3.kubevirt.io Then removing the finalizer on the kubevirt-hyperconverged CR and then on the namespace? At that point you should be able to remove the namespace.
The root cause that initially triggered the issue was `nmstate-ca secret deleted and not recreated upon reinstall`; moving this on network component.
The team is looking into it. We believe we have identified the cause - certificate rotation was triggered too rapidly, we were not keeping up with cleaning them up and as a result our secret grew too big and was rejected by etcd. We are looking into our options how to backport this to 2.6. We think we have a workaround script available, but we want to verify it first. The only standing question is what lead into this rapid certificate rotation. This seems to be very similar to https://bugzilla.redhat.com/show_bug.cgi?id=2011369 except that the reference issue was caused by reducing the cert rotation interval. This current issue seems to be triggered by something else.
Dirk, it seems that the issue we observe was caused by manual removal of the nmstate-ca Secret. That has triggered a hot-loop of rotating CA bundles which ended up by Kubernetes API rejecting our further attempts to rotate. Could you confirm that the Secret was removed manually? If so, I suppose there was a different issue prior to this, that lead us to the manual intervention. However, we will not be able to address the original issue until we recover the certificate rotation mechanism. In order to do so, please use following script (kudos to ellorent): wget -O force-cert-rotation.sh https://raw.githubusercontent.com/qinqon/kube-admission-webhook/v0.12.0/hack/force-cert-retation.sh chmod +x force-cert-rotation.sh HANDLER_NAMESPACE=openshift-cnv HANDLER_POD_LABEL=component=kubernetes-nmstate-webhook WEBHOOK_CONFIG_NAME=nmstate WEBHOOK_SECRET_NAME=nmstate-ca ./force-cert-rotation.sh Let me know if there would be any issues applying it.
Petr, Yes the secret was deleted manually after reviewing the certificate and realizing that the certificate was expired. I have a call with the customer shortly and will try to use the script. Thank you! Regards, Dirk Porter
Hello Petr, After trying to run the script provided we are encountering the below errors. I believe it is because of the state of the operator, I believe we may need to remove or add some components to get back to a good state. [corona.azusca21.vzwops.com ~]$ HANDLER_NAMESPACE=openshift-cnv HANDLER_POD_LABEL=component=kubernetes-nmstate-webhook WEBHOOK_CONFIG_NAME=nmstate WEBHOOK_SECRET_NAME=nmstate-ca ./force-cert-rotation.sh + KUBECTL=kubectl + parse_input_param + [[ ! -z '' ]] + check_input_env_var + [[ -z openshift-cnv ]] + [[ -z component=kubernetes-nmstate-webhook ]] + [[ -z nmstate ]] + [[ -z nmstate-ca ]] + for WEBHOOK_TYPE in "validationWebhookConfiguration" "mutatingWebhookConfiguration" ++ get_number_of_webhooks_in_webhook_config +++ wc -w +++ kubectl get validationWebhookConfiguration -n openshift-cnv nmstate -o 'jsonpath={.webhooks[*].name}' I1007 14:13:36.676133 2073804 request.go:655] Throttling request took 1.157105718s, request: GET:https://api.kub1.azusca21.vzwops.com:6443/apis/networking.istio.io/v1alpha3?timeout=32s error: the server doesn't have a resource type "validationWebhookConfiguration" ++ echo 0 + number_of_webhooks=0 + [[ 0 > 0 ]] + for WEBHOOK_TYPE in "validationWebhookConfiguration" "mutatingWebhookConfiguration" ++ get_number_of_webhooks_in_webhook_config +++ wc -w +++ kubectl get mutatingWebhookConfiguration -n openshift-cnv nmstate -o 'jsonpath={.webhooks[*].name}' Error from server (NotFound): mutatingwebhookconfigurations.admissionregistration.k8s.io "nmstate" not found ++ echo 0 + number_of_webhooks=0 + [[ 0 > 0 ]] [corona.azusca21.vzwops.com ~]$ [corona.azusca21.vzwops.com ~]$ [corona.azusca21.vzwops.com ~]$ [corona.azusca21.vzwops.com ~]$ [corona.azusca21.vzwops.com ~]$ oc get validationWebhookConfiguration error: the server doesn't have a resource type "validationWebhookConfiguration" [corona.azusca21.vzwops.com ~]$ oc get validationWebhookConfiguration -A error: the server doesn't have a resource type "validationWebhookConfiguration" [corona.azusca21.vzwops.com ~]$ [corona.azusca21.vzwops.com ~]$ kubectl get validationWebhookConfiguration I1007 14:15:02.715086 2081406 request.go:655] Throttling request took 1.154036402s, request: GET:https://api.kub1.azusca21.vzwops.com:6443/apis/security.openshift.io/v1?timeout=32s error: the server doesn't have a resource type "validationWebhookConfiguration" [corona.azusca21.vzwops.com ~]$ [corona.azusca21.vzwops.com ~]$ [corona.azusca21.vzwops.com ~]$ [corona.azusca21.vzwops.com ~]$ oc get mutatingwebhookconfiguration -A NAME WEBHOOKS AGE aspen-mesh-cert-manager-webhook 1 138d cdi-api-datavolume-mutate 1 138d istio-sidecar-injector 1 138d istio-sidecar-injector-canary 1 43h machine-api 2 138d mutate-ns-hco.kubevirt.io-68ktp 1 16h network-resources-injector-config 1 138d sriov-operator-webhook-config 1 36h virt-api-mutator 3 16h [corona.azusca21.vzwops.com ~]$ oc get validatingwebhookconfiguration -A NAME WEBHOOKS AGE aspen-mesh-cert-manager-webhook 1 138d aspen-mesh-controlplane 2 138d aspen-mesh-secure-ingress 1 43h autoscaling.openshift.io 2 138d cdi-api-datavolume-validate 1 138d cdi-api-validate 1 138d istiod-istio-system 1 43h machine-api 2 138d multus.openshift.io 1 138d nodemaintenance-validation.kubevirt.io-2rzbx 1 16h prometheusrules.openshift.io 1 138d snapshot.storage.k8s.io 1 40h sriov-operator-webhook-config 1 36h traffic-claim-enforcer 1 43h validate-hco.kubevirt.io-zsqb5 1 16h validation-dns-updater-webhook-cfg 1 138d virt-api-validator 11 16h virt-operator-validator 2 16h vssp.kb.io-8xrj5 1 16h [corona.azusca21.vzwops.com ~]$ [corona.azusca21.vzwops.com ~]$ [corona.azusca21.vzwops.com ~]$ [corona.azusca21.vzwops.com ~]$ oc get validatingwebhookconfiguration -A | grep nmstate
*** This bug has been marked as a duplicate of bug 2011369 ***