Bug 1892227 - [SSP] cluster scoped resources are not being reconciled
Summary: [SSP] cluster scoped resources are not being reconciled
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: SSP
Version: 2.5.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 2.6.0
Assignee: Omer Yahud
QA Contact: Sarah Bennert
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-10-28 09:15 UTC by Ruth Netser
Modified: 2021-03-10 11:19 UTC (History)
2 users (show)

Fixed In Version: kubevirt-ssp-operator-container-v2.6.0-31
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-03-10 11:18:56 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2021:0799 0 None None None 2021-03-10 11:19:54 UTC

Description Ruth Netser 2020-10-28 09:15:35 UTC
Description of problem:
virt-template-validator ValidatingWebhookConfiguration is not restored after deletion

Version-Release number of selected component (if applicable):
kubevirt-ssp-operator-container-v2.5.0-55

How reproducible:
100%

Steps to Reproduce:
1. Delete virt-template-validator ValidatingWebhookConfiguration
$ oc delete ValidatingWebhookConfiguration virt-template-validator

Actual results:
virt-template-validator ValidatingWebhookConfiguration is not restored after deletion

Expected results:
virt-template-validator ValidatingWebhookConfiguration should be restored

Additional info:
$ oc get ValidatingWebhookConfiguration virt-template-validator -oyaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  annotations:
    operator-sdk/primary-resource: openshift-cnv/template-validator-kubevirt-hyperconverged
    operator-sdk/primary-resource-type: KubevirtTemplateValidator.ssp.kubevirt.io
    service.beta.openshift.io/inject-cabundle: "true"
  creationTimestamp: "2020-10-28T09:04:40Z"
  generation: 2
  managedFields:
  - apiVersion: admissionregistration.k8s.io/v1beta1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:operator-sdk/primary-resource: {}
          f:operator-sdk/primary-resource-type: {}
          f:service.beta.openshift.io/inject-cabundle: {}
      f:webhooks:
        .: {}
        k:{"name":"virt-template-admission.kubevirt.io"}:
          .: {}
          f:admissionReviewVersions: {}
          f:clientConfig:
            .: {}
            f:service:
              .: {}
              f:name: {}
              f:namespace: {}
              f:path: {}
              f:port: {}
          f:failurePolicy: {}
          f:matchPolicy: {}
          f:name: {}
          f:namespaceSelector: {}
          f:objectSelector: {}
          f:rules: {}
          f:sideEffects: {}
          f:timeoutSeconds: {}
    manager: OpenAPI-Generator
    operation: Update
    time: "2020-10-28T09:04:40Z"
  - apiVersion: admissionregistration.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:webhooks:
        k:{"name":"virt-template-admission.kubevirt.io"}:
          f:clientConfig:
            f:caBundle: {}
    manager: service-ca-operator
    operation: Update
    time: "2020-10-28T09:04:40Z"
  name: virt-template-validator
  resourceVersion: "7993360"
  selfLink: /apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations/virt-template-validator
  uid: f4ec8f5b-7651-46dc-af87-e7da2c8464ef
webhooks:
- admissionReviewVersions:
  - v1beta1
  clientConfig:
    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURVVENDQWptZ0F3SUJBZ0lJZEhTUDM4UzZPUFl3RFFZSktvWklodmNOQVFFTEJRQXdOakUwTURJR0ExVUUKQXd3cmIzQmxibk5vYVdaMExYTmxjblpwWTJVdGMyVnlkbWx1WnkxemFXZHVaWEpBTVRZd016WXlNemMyTXpBZQpGdzB5TURFd01qVXhNVEF5TkRKYUZ3MHlNakV5TWpReE1UQXlORE5hTURZeE5EQXlCZ05WQkFNTUsyOXdaVzV6CmFHbG1kQzF6WlhKMmFXTmxMWE5sY25acGJtY3RjMmxuYm1WeVFERTJNRE0yTWpNM05qTXdnZ0VpTUEwR0NTcUcKU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRRGhVakloTHFsZktGRFNUZmZVdkZvUFM0L3c2SmRtUExGSwp3cDNGb216N05vUzBERzB0V1p1UW51djFuNTNISFVabzZ2UVdzR3IxY2QzeE5jZ0NVQ2UrVzlWU3lxL044QTJ4CkJaUndhaW5XNjg4YlZ0R1lCNzUyTVY0cEpsWXFrVmhQRFRJeUFPckFHUU9IakRmOVFVRStBYTVSUy9Ob0VqYUUKNmtUY25mNW1idUo2VkxlZVE2RnQ5RFVSQWtTV1FoSHJvRnVobHY2eFQ3Q3BVanREdEdzNjYzUlpnRHlGSjN3Mgp6SlR2MzNTMVhsZWtodEMvWnpLamtSemQ1VTR5STJqRXhxTUNwK1dNajN6N3ZHYkN1Y05jVHhLSmxpN3Q2TzQyCjRxSzdmSTRQT2lNSy8yQ2VZVWhScFFBTXN1c3ZTOFhEbGMwQ3ErUXZTWTlJdUxwMWttR3hBZ01CQUFHall6QmgKTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlQwSExFWgpGc3dvOE9tclJyeFV3VE96dElUVHdqQWZCZ05WSFNNRUdEQVdnQlQwSExFWkZzd284T21yUnJ4VXdUT3p0SVRUCndqQU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFpTURhUU43Ykx2SHYxYS9BMlNuTGtvcjFZQzRrZndYb21Wc0QKZHEycU5vQUQ3MkduMlRZYitPdGdHUWdVNVRCS0ZUaDhoV0U2L015V3ZWeFRHZHpvQzY1K0tydERMc2ZURnMvTApSMEk4Q0IxbFBVSjJRV09ucXZES0hjNUl4eXVwczV0WWwyK3pkNW5lNDFudTRNakIzTnZ3M2xiWkxTdkFXL20rClZmY2FQaHpJeU5xOEM0bnRvWERPeTVaU2VYSVl2YXdkclhWdWE0KzZGcENZNlZJUkNOejdUNzZNQnhVTVZuQ2kKMSt3aFpsN1hWYVdJR2w5SkxsK1QrYmdzRlN2clhlaXJFSFZ2aVI2bklKdUN6YWxWTmV4anhBMWx4eHNCOXNPWQo5YzhHSUs2Z24vVkFsU1k1SWI1SmdjN1V2MElnUDVUUWtWb2J3citNeW9IcW51ZFQ0UT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    service:
      name: virt-template-validator
      namespace: openshift-cnv
      path: /virtualmachine-template-validate
      port: 443
  failurePolicy: Fail
  matchPolicy: Exact
  name: virt-template-admission.kubevirt.io
  namespaceSelector: {}
  objectSelector: {}
  rules:
  - apiGroups:
    - kubevirt.io
    apiVersions:
    - v1alpha3
    operations:
    - CREATE
    - UPDATE
    resources:
    - virtualmachines
    scope: '*'
  sideEffects: Unknown
  timeoutSeconds: 30

Comment 1 Omer Yahud 2020-10-28 09:17:01 UTC
We can see that the ownership annotations set by operator-sdk exist, so the reconciliation should have been triggered automatically, we will investigate it further

Comment 2 Omer Yahud 2020-10-28 09:17:44 UTC
Note: a workaround would be to delete the ssp-operator pod and let it recreate the webhook

Comment 3 Ruth Netser 2020-10-28 11:59:26 UTC
Same applies to kubevirt-node-labeller SecurityContextConstraints; the resource is not reconsiled after deletion
$ oc get SecurityContextConstraints -n openshift-cnv kubevirt-node-labeller -oyaml
allowHostDirVolumePlugin: false
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: true
allowPrivilegedContainer: true
allowedCapabilities: null
apiVersion: security.openshift.io/v1
defaultAddCapabilities: null
fsGroup:
  type: RunAsAny
groups: []
kind: SecurityContextConstraints
metadata:
  annotations:
    operator-sdk/primary-resource: openshift-cnv/node-labeller-kubevirt-hyperconverged
    operator-sdk/primary-resource-type: KubevirtNodeLabellerBundle.ssp.kubevirt.io
  creationTimestamp: "2020-10-28T11:09:09Z"
  generation: 1
  managedFields:
  - apiVersion: security.openshift.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:allowPrivilegedContainer: {}
      f:metadata:
        f:annotations:
          .: {}
          f:operator-sdk/primary-resource: {}
          f:operator-sdk/primary-resource-type: {}
      f:runAsUser:
        .: {}
        f:type: {}
      f:seLinuxContext:
        .: {}
        f:type: {}
      f:users: {}
    manager: OpenAPI-Generator
    operation: Update
    time: "2020-10-28T11:09:09Z"
  name: kubevirt-node-labeller
  resourceVersion: "8249192"
  selfLink: /apis/security.openshift.io/v1/securitycontextconstraints/kubevirt-node-labeller
  uid: 9c2c8048-42cb-48ef-be36-f81a0657eba6
priority: null
readOnlyRootFilesystem: false
requiredDropCapabilities: null
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: RunAsAny
supplementalGroups:
  type: RunAsAny

Comment 5 Fabian Deutsch 2020-10-28 13:09:48 UTC
We can consider to move the webhook handling to the csv level, then olm will take care of it. HCO did something similar

Comment 6 Omer Yahud 2020-12-28 09:31:46 UTC
This should be retested when the 2.6 operator makes it downstream, I will update this bug with build information once it becomes available

Comment 7 Omer Yahud 2021-01-11 20:35:40 UTC
Can be verified with kubevirt-ssp-operator-container-v2.6.0-31

Comment 8 Sarah Bennert 2021-01-21 21:14:15 UTC
Verified with:

OCP
Image:   quay.io/openshift-release-dev/ocp-release@sha256:2f00e3016ca5678e51e9d79d4d3ac5a2926e0c09a8e75df19ea983b9cd6c5d05
Version: 4.7.0-fc.2

CNV-SSP
Image:  registry.redhat.io/container-native-virtualization/kubevirt-ssp-operator@sha256:40fe05888d7093164ab7f66d3a081a4bff877eb400934303e42a7b973a6c2a0c
Version: kubevirt-ssp-operator-container-v2.6.0-35


Verify ValidatingWebhookConfiguration restoration after delete:


# Download ValidatingWebhookConfiguration installed by operator
$ oc get ValidatingWebhookConfiguration virt-template-validator -oyaml | tee virt-template-validator-webhook-config-original.yaml


# Delete the configuration
$ oc delete ValidatingWebhookConfiguration virt-template-validator
validatingwebhookconfiguration.admissionregistration.k8s.io "virt-template-validator" deleted


# Download the restored configuration
$ oc get ValidatingWebhookConfiguration virt-template-validator -oyaml > virt-template-validator-webhook-config-restored.yaml


# Verify manifests exist and contain data
$ ls -l virt-template-validator-webhook-config-*
-rw-r--r--  1 user  group  4011 Jan 19 15:02 virt-template-validator-webhook-config-original.yaml
-rw-r--r--  1 user  group  4011 Jan 19 15:03 virt-template-validator-webhook-config-restored.yaml


# List the differences between the two yamls.
# Expected to only have timestamps, resourceVersion, and uid change
$ diff virt-template-validator-webhook-config-*
8c8
<   creationTimestamp: "2021-01-18T21:02:00Z"
---
>   creationTimestamp: "2021-01-19T20:03:19Z"
43c43
<     time: "2021-01-18T21:02:00Z"
---
>     time: "2021-01-19T20:03:19Z"
53c53
<     time: "2021-01-18T21:02:00Z"
---
>     time: "2021-01-19T20:03:19Z"
55c55
<   resourceVersion: "1405295"
---
>   resourceVersion: "2381796"
57c57
<   uid: bdecaeff-1644-4755-9c55-936d89f318f0
---
>   uid: 3d821b06-fcc1-4312-a3d0-ddcdac7f22ed



Verify SecurityContextConstraint restoration:



# Download SecurityContextConstraints installed by operator
oc -n openshift-cnv get SecurityContextConstraints kubevirt-node-labeller -oyaml | tee kubevirt-node-labeller-SecurityContextConstraints-original.yaml



# Delete the SecurityContextConstraint
$ oc delete SecurityContextConstraints kubevirt-node-labeller
securitycontextconstraints.security.openshift.io "kubevirt-node-labeller" deleted


# Download the restored SecurityContextConstraint
$ oc get SecurityContextConstraints kubevirt-node-labeller -oyaml > kubevirt-node-labeller-SecurityContextConstraints-restored.yaml


# Verify manifests exist and contain data
$ ls -l kubevirt-node-labeller-SecurityContextConstraints-*
-rw-r--r--  1 user  group  2233 Jan 21 14:49 kubevirt-node-labeller-SecurityContextConstraints-original.yaml
-rw-r--r--  1 user  group  2233 Jan 21 14:51 kubevirt-node-labeller-SecurityContextConstraints-restored.yaml

$ diff kubevirt-node-labeller-SecurityContextConstraints-*
19c19
<   creationTimestamp: "2021-01-18T21:02:03Z"
---
>   creationTimestamp: "2021-01-21T19:50:27Z"
54c54
<     time: "2021-01-18T21:02:03Z"
---
>     time: "2021-01-21T19:50:27Z"
56c56
<   resourceVersion: "1405929"
---
>   resourceVersion: "4412988"
58c58
<   uid: d8f97c06-6804-4832-b21f-5ccc6327eaef
---
>   uid: c0f3ba96-73ad-44c1-bbfe-684155f07662

Comment 11 errata-xmlrpc 2021-03-10 11:18:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 2.6.0 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:0799


Note You need to log in before you can comment on or make changes to this bug.