Description of problem: virt-template-validator ValidatingWebhookConfiguration is not restored after deletion Version-Release number of selected component (if applicable): kubevirt-ssp-operator-container-v2.5.0-55 How reproducible: 100% Steps to Reproduce: 1. Delete virt-template-validator ValidatingWebhookConfiguration $ oc delete ValidatingWebhookConfiguration virt-template-validator Actual results: virt-template-validator ValidatingWebhookConfiguration is not restored after deletion Expected results: virt-template-validator ValidatingWebhookConfiguration should be restored Additional info: $ oc get ValidatingWebhookConfiguration virt-template-validator -oyaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration metadata: annotations: operator-sdk/primary-resource: openshift-cnv/template-validator-kubevirt-hyperconverged operator-sdk/primary-resource-type: KubevirtTemplateValidator.ssp.kubevirt.io service.beta.openshift.io/inject-cabundle: "true" creationTimestamp: "2020-10-28T09:04:40Z" generation: 2 managedFields: - apiVersion: admissionregistration.k8s.io/v1beta1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:operator-sdk/primary-resource: {} f:operator-sdk/primary-resource-type: {} f:service.beta.openshift.io/inject-cabundle: {} f:webhooks: .: {} k:{"name":"virt-template-admission.kubevirt.io"}: .: {} f:admissionReviewVersions: {} f:clientConfig: .: {} f:service: .: {} f:name: {} f:namespace: {} f:path: {} f:port: {} f:failurePolicy: {} f:matchPolicy: {} f:name: {} f:namespaceSelector: {} f:objectSelector: {} f:rules: {} f:sideEffects: {} f:timeoutSeconds: {} manager: OpenAPI-Generator operation: Update time: "2020-10-28T09:04:40Z" - apiVersion: admissionregistration.k8s.io/v1 fieldsType: FieldsV1 fieldsV1: f:webhooks: k:{"name":"virt-template-admission.kubevirt.io"}: f:clientConfig: f:caBundle: {} manager: service-ca-operator operation: Update time: "2020-10-28T09:04:40Z" name: virt-template-validator resourceVersion: "7993360" selfLink: /apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations/virt-template-validator uid: f4ec8f5b-7651-46dc-af87-e7da2c8464ef webhooks: - admissionReviewVersions: - v1beta1 clientConfig: caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURVVENDQWptZ0F3SUJBZ0lJZEhTUDM4UzZPUFl3RFFZSktvWklodmNOQVFFTEJRQXdOakUwTURJR0ExVUUKQXd3cmIzQmxibk5vYVdaMExYTmxjblpwWTJVdGMyVnlkbWx1WnkxemFXZHVaWEpBTVRZd016WXlNemMyTXpBZQpGdzB5TURFd01qVXhNVEF5TkRKYUZ3MHlNakV5TWpReE1UQXlORE5hTURZeE5EQXlCZ05WQkFNTUsyOXdaVzV6CmFHbG1kQzF6WlhKMmFXTmxMWE5sY25acGJtY3RjMmxuYm1WeVFERTJNRE0yTWpNM05qTXdnZ0VpTUEwR0NTcUcKU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRRGhVakloTHFsZktGRFNUZmZVdkZvUFM0L3c2SmRtUExGSwp3cDNGb216N05vUzBERzB0V1p1UW51djFuNTNISFVabzZ2UVdzR3IxY2QzeE5jZ0NVQ2UrVzlWU3lxL044QTJ4CkJaUndhaW5XNjg4YlZ0R1lCNzUyTVY0cEpsWXFrVmhQRFRJeUFPckFHUU9IakRmOVFVRStBYTVSUy9Ob0VqYUUKNmtUY25mNW1idUo2VkxlZVE2RnQ5RFVSQWtTV1FoSHJvRnVobHY2eFQ3Q3BVanREdEdzNjYzUlpnRHlGSjN3Mgp6SlR2MzNTMVhsZWtodEMvWnpLamtSemQ1VTR5STJqRXhxTUNwK1dNajN6N3ZHYkN1Y05jVHhLSmxpN3Q2TzQyCjRxSzdmSTRQT2lNSy8yQ2VZVWhScFFBTXN1c3ZTOFhEbGMwQ3ErUXZTWTlJdUxwMWttR3hBZ01CQUFHall6QmgKTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlQwSExFWgpGc3dvOE9tclJyeFV3VE96dElUVHdqQWZCZ05WSFNNRUdEQVdnQlQwSExFWkZzd284T21yUnJ4VXdUT3p0SVRUCndqQU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFpTURhUU43Ykx2SHYxYS9BMlNuTGtvcjFZQzRrZndYb21Wc0QKZHEycU5vQUQ3MkduMlRZYitPdGdHUWdVNVRCS0ZUaDhoV0U2L015V3ZWeFRHZHpvQzY1K0tydERMc2ZURnMvTApSMEk4Q0IxbFBVSjJRV09ucXZES0hjNUl4eXVwczV0WWwyK3pkNW5lNDFudTRNakIzTnZ3M2xiWkxTdkFXL20rClZmY2FQaHpJeU5xOEM0bnRvWERPeTVaU2VYSVl2YXdkclhWdWE0KzZGcENZNlZJUkNOejdUNzZNQnhVTVZuQ2kKMSt3aFpsN1hWYVdJR2w5SkxsK1QrYmdzRlN2clhlaXJFSFZ2aVI2bklKdUN6YWxWTmV4anhBMWx4eHNCOXNPWQo5YzhHSUs2Z24vVkFsU1k1SWI1SmdjN1V2MElnUDVUUWtWb2J3citNeW9IcW51ZFQ0UT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K service: name: virt-template-validator namespace: openshift-cnv path: /virtualmachine-template-validate port: 443 failurePolicy: Fail matchPolicy: Exact name: virt-template-admission.kubevirt.io namespaceSelector: {} objectSelector: {} rules: - apiGroups: - kubevirt.io apiVersions: - v1alpha3 operations: - CREATE - UPDATE resources: - virtualmachines scope: '*' sideEffects: Unknown timeoutSeconds: 30
We can see that the ownership annotations set by operator-sdk exist, so the reconciliation should have been triggered automatically, we will investigate it further
Note: a workaround would be to delete the ssp-operator pod and let it recreate the webhook
Same applies to kubevirt-node-labeller SecurityContextConstraints; the resource is not reconsiled after deletion $ oc get SecurityContextConstraints -n openshift-cnv kubevirt-node-labeller -oyaml allowHostDirVolumePlugin: false allowHostIPC: false allowHostNetwork: false allowHostPID: false allowHostPorts: false allowPrivilegeEscalation: true allowPrivilegedContainer: true allowedCapabilities: null apiVersion: security.openshift.io/v1 defaultAddCapabilities: null fsGroup: type: RunAsAny groups: [] kind: SecurityContextConstraints metadata: annotations: operator-sdk/primary-resource: openshift-cnv/node-labeller-kubevirt-hyperconverged operator-sdk/primary-resource-type: KubevirtNodeLabellerBundle.ssp.kubevirt.io creationTimestamp: "2020-10-28T11:09:09Z" generation: 1 managedFields: - apiVersion: security.openshift.io/v1 fieldsType: FieldsV1 fieldsV1: f:allowPrivilegedContainer: {} f:metadata: f:annotations: .: {} f:operator-sdk/primary-resource: {} f:operator-sdk/primary-resource-type: {} f:runAsUser: .: {} f:type: {} f:seLinuxContext: .: {} f:type: {} f:users: {} manager: OpenAPI-Generator operation: Update time: "2020-10-28T11:09:09Z" name: kubevirt-node-labeller resourceVersion: "8249192" selfLink: /apis/security.openshift.io/v1/securitycontextconstraints/kubevirt-node-labeller uid: 9c2c8048-42cb-48ef-be36-f81a0657eba6 priority: null readOnlyRootFilesystem: false requiredDropCapabilities: null runAsUser: type: RunAsAny seLinuxContext: type: RunAsAny supplementalGroups: type: RunAsAny
We can consider to move the webhook handling to the csv level, then olm will take care of it. HCO did something similar
This should be retested when the 2.6 operator makes it downstream, I will update this bug with build information once it becomes available
Can be verified with kubevirt-ssp-operator-container-v2.6.0-31
Verified with: OCP Image: quay.io/openshift-release-dev/ocp-release@sha256:2f00e3016ca5678e51e9d79d4d3ac5a2926e0c09a8e75df19ea983b9cd6c5d05 Version: 4.7.0-fc.2 CNV-SSP Image: registry.redhat.io/container-native-virtualization/kubevirt-ssp-operator@sha256:40fe05888d7093164ab7f66d3a081a4bff877eb400934303e42a7b973a6c2a0c Version: kubevirt-ssp-operator-container-v2.6.0-35 Verify ValidatingWebhookConfiguration restoration after delete: # Download ValidatingWebhookConfiguration installed by operator $ oc get ValidatingWebhookConfiguration virt-template-validator -oyaml | tee virt-template-validator-webhook-config-original.yaml # Delete the configuration $ oc delete ValidatingWebhookConfiguration virt-template-validator validatingwebhookconfiguration.admissionregistration.k8s.io "virt-template-validator" deleted # Download the restored configuration $ oc get ValidatingWebhookConfiguration virt-template-validator -oyaml > virt-template-validator-webhook-config-restored.yaml # Verify manifests exist and contain data $ ls -l virt-template-validator-webhook-config-* -rw-r--r-- 1 user group 4011 Jan 19 15:02 virt-template-validator-webhook-config-original.yaml -rw-r--r-- 1 user group 4011 Jan 19 15:03 virt-template-validator-webhook-config-restored.yaml # List the differences between the two yamls. # Expected to only have timestamps, resourceVersion, and uid change $ diff virt-template-validator-webhook-config-* 8c8 < creationTimestamp: "2021-01-18T21:02:00Z" --- > creationTimestamp: "2021-01-19T20:03:19Z" 43c43 < time: "2021-01-18T21:02:00Z" --- > time: "2021-01-19T20:03:19Z" 53c53 < time: "2021-01-18T21:02:00Z" --- > time: "2021-01-19T20:03:19Z" 55c55 < resourceVersion: "1405295" --- > resourceVersion: "2381796" 57c57 < uid: bdecaeff-1644-4755-9c55-936d89f318f0 --- > uid: 3d821b06-fcc1-4312-a3d0-ddcdac7f22ed Verify SecurityContextConstraint restoration: # Download SecurityContextConstraints installed by operator oc -n openshift-cnv get SecurityContextConstraints kubevirt-node-labeller -oyaml | tee kubevirt-node-labeller-SecurityContextConstraints-original.yaml # Delete the SecurityContextConstraint $ oc delete SecurityContextConstraints kubevirt-node-labeller securitycontextconstraints.security.openshift.io "kubevirt-node-labeller" deleted # Download the restored SecurityContextConstraint $ oc get SecurityContextConstraints kubevirt-node-labeller -oyaml > kubevirt-node-labeller-SecurityContextConstraints-restored.yaml # Verify manifests exist and contain data $ ls -l kubevirt-node-labeller-SecurityContextConstraints-* -rw-r--r-- 1 user group 2233 Jan 21 14:49 kubevirt-node-labeller-SecurityContextConstraints-original.yaml -rw-r--r-- 1 user group 2233 Jan 21 14:51 kubevirt-node-labeller-SecurityContextConstraints-restored.yaml $ diff kubevirt-node-labeller-SecurityContextConstraints-* 19c19 < creationTimestamp: "2021-01-18T21:02:03Z" --- > creationTimestamp: "2021-01-21T19:50:27Z" 54c54 < time: "2021-01-18T21:02:03Z" --- > time: "2021-01-21T19:50:27Z" 56c56 < resourceVersion: "1405929" --- > resourceVersion: "4412988" 58c58 < uid: d8f97c06-6804-4832-b21f-5ccc6327eaef --- > uid: c0f3ba96-73ad-44c1-bbfe-684155f07662
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Virtualization 2.6.0 security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:0799