Bug 2036483 - HCO Enablement | reconciliation error adding a custom cron template
Summary: HCO Enablement | reconciliation error adding a custom cron template
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: SSP
Version: 4.10.0
Hardware: Unspecified
OS: Unspecified
urgent
unspecified
Target Milestone: ---
: 4.10.0
Assignee: Andrej Krejcir
QA Contact: ibesso
URL:
Whiteboard:
Depends On: 2037290
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-02 14:05 UTC by ibesso
Modified: 2022-03-16 16:05 UTC (History)
5 users (show)

Fixed In Version: kubevirt-ssp-operator-container-v4.10.0-39
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-16 16:05:38 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker CNV-15556 0 None None None 2022-01-07 08:37:38 UTC
Red Hat Product Errata RHSA-2022:0947 0 None Closed RHEL EUS Errata Documentation 2022-06-16 06:26:55 UTC

Description ibesso 2022-01-02 14:05:54 UTC
Description of problem:
----------------------
After successfully adding the dataImportCronTemplates data to HCO CR, SSP CR is reconciled endlessly.
Alternately, it becomes ready (Available: True, Progressing: False, etc.) and back to not ready every couple of seconds.

The ssp operator pod logs show the reconciliation error repeatedly:
2021-12-30T23:22:10.443Z	ERROR	controller-runtime.manager.controller.ssp	Reconciler error	{"reconciler group": "ssp.kubevirt.io", "reconciler kind": "SSP", "name": "ssp-kubevirt-hyperconverged", "namespace": "openshift-cnv", "error": "Operation cannot be fulfilled on ssps.ssp.kubevirt.io \"ssp-kubevirt-hyperconverged\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214



Version-Release number of selected component (if applicable):
------------------------------------------------------------
4.10.0-545 (pre-release)



How reproducible:
----------------
100%



Steps to Reproduce:
------------------
1. oc patch hco kubevirt-hyperconverged -n openshift-cnv --type merge -p '{"spec": {"featureGates": {"enableCommonBootImageImport": true}}}'

2. oc edit hco -n openshift-cnv

3. add the following entry under the spec:
  dataImportCronTemplates:
  - metadata:
      annotations:
        cdi.kubevirt.io/storage.bind.immediate.requested: "true"
      name: my-image-cron
    spec:
      garbageCollect: Outdated
      managedDataSource: my
      schedule: 59 23/3 * * *
      template:
        metadata: {}
        spec:
          source:
            registry:
              imageStream: my-guest
              pullMethod: node
          storage:
            resources:
              requests:
                storage: 10Gi
        status: {}

4. watch the ssp operator pod logs:
oc logs --follow ssp-operator-64b46b479b-2txj8 -n openshift-cnv --tail 1



Actual results:
--------------
1. repeatedly reconciliation errors in ssp operator pod logs:
2021-12-30T23:22:10.443Z	ERROR	controller-runtime.manager.controller.ssp	Reconciler error	{"reconciler group": "ssp.kubevirt.io", "reconciler kind": "SSP", "name": "ssp-kubevirt-hyperconverged", "namespace": "openshift-cnv", "error": "Operation cannot be fulfilled on ssps.ssp.kubevirt.io \"ssp-kubevirt-hyperconverged\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214

2. ssp status conditions alternately switches between ready conditions (Available: True, Progressing: False, Degraded: False) and not ready conditions (Available: False, Progressing: True, Degraded: True) every now and then (ready conditions for 1-2 seconds and then to not ready for a longer period)



Expected results:
----------------
Reconciliation should be successful.



Additional info:
---------------
Same results if enabling the featuregate "enableCommonBootImageImport".

Comment 1 Andrej Krejcir 2022-01-14 12:39:51 UTC
The cause is probably the same as in Bug 2037290.

Comment 2 Andrej Krejcir 2022-01-18 12:06:50 UTC
The bug that this one depends on was fixed.

Comment 3 ibesso 2022-01-19 12:24:08 UTC
Verification with a cluster running CNV 4.10.0-595 (the bundle contains the fixed ssp version 4.10.0-39)
--------------------------------------------------------------------------------------------------------
* With featuregate disabled - adding/modifying a custom entry in HCO CR - propagated to SSP CR as expected, no reconciliation loops in ssp-operator pod logs.
* Enabling the featuregate - SSP CR got the hard-coded templates.
* Executed several automation tests (still under development).

[cnv-qe-jenkins@c01-issac-410-hhrz5-executor ~]$ oc get dic -A
NAMESPACE                            NAME                        AGE
openshift-virtualization-os-images   centos-stream8-image-cron   21s
openshift-virtualization-os-images   centos-stream9-image-cron   21s
openshift-virtualization-os-images   fedora-image-cron           21s
openshift-virtualization-os-images   my-image-cron               3m18s
openshift-virtualization-os-images   rhel8-image-cron            21s
openshift-virtualization-os-images   rhel9-image-cron            21s


[cnv-qe-jenkins@c01-issac-410-hhrz5-executor ~]$ oc get is -n openshift-virtualization-os-images
NAME          IMAGE REPOSITORY                                                                                  TAGS     UPDATED
rhel8-guest   image-registry.openshift-image-registry.svc:5000/openshift-virtualization-os-images/rhel8-guest   latest   2 minutes ago
rhel9-guest   image-registry.openshift-image-registry.svc:5000/openshift-virtualization-os-images/rhel9-guest   latest   2 minutes ago
[cnv-qe-jenkins@c01-issac-410-hhrz5-executor ~]$ 


Moving to VERIFIED.

Comment 8 errata-xmlrpc 2022-03-16 16:05:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 4.10.0 Images security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0947


Note You need to log in before you can comment on or make changes to this bug.