Bug 2036483
| Summary: | HCO Enablement | reconciliation error adding a custom cron template | ||
|---|---|---|---|
| Product: | Container Native Virtualization (CNV) | Reporter: | ibesso <ibesso> |
| Component: | SSP | Assignee: | Andrej Krejcir <akrejcir> |
| Status: | CLOSED ERRATA | QA Contact: | ibesso <ibesso> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 4.10.0 | CC: | cnv-qe-bugs, dholler, nunnatsa, rnetser, stirabos |
| Target Milestone: | --- | ||
| Target Release: | 4.10.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | kubevirt-ssp-operator-container-v4.10.0-39 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-03-16 16:05:38 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 2037290 | ||
| Bug Blocks: | |||
The cause is probably the same as in Bug 2037290. The bug that this one depends on was fixed. Verification with a cluster running CNV 4.10.0-595 (the bundle contains the fixed ssp version 4.10.0-39) -------------------------------------------------------------------------------------------------------- * With featuregate disabled - adding/modifying a custom entry in HCO CR - propagated to SSP CR as expected, no reconciliation loops in ssp-operator pod logs. * Enabling the featuregate - SSP CR got the hard-coded templates. * Executed several automation tests (still under development). [cnv-qe-jenkins@c01-issac-410-hhrz5-executor ~]$ oc get dic -A NAMESPACE NAME AGE openshift-virtualization-os-images centos-stream8-image-cron 21s openshift-virtualization-os-images centos-stream9-image-cron 21s openshift-virtualization-os-images fedora-image-cron 21s openshift-virtualization-os-images my-image-cron 3m18s openshift-virtualization-os-images rhel8-image-cron 21s openshift-virtualization-os-images rhel9-image-cron 21s [cnv-qe-jenkins@c01-issac-410-hhrz5-executor ~]$ oc get is -n openshift-virtualization-os-images NAME IMAGE REPOSITORY TAGS UPDATED rhel8-guest image-registry.openshift-image-registry.svc:5000/openshift-virtualization-os-images/rhel8-guest latest 2 minutes ago rhel9-guest image-registry.openshift-image-registry.svc:5000/openshift-virtualization-os-images/rhel9-guest latest 2 minutes ago [cnv-qe-jenkins@c01-issac-410-hhrz5-executor ~]$ Moving to VERIFIED. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Virtualization 4.10.0 Images security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0947 |
Description of problem: ---------------------- After successfully adding the dataImportCronTemplates data to HCO CR, SSP CR is reconciled endlessly. Alternately, it becomes ready (Available: True, Progressing: False, etc.) and back to not ready every couple of seconds. The ssp operator pod logs show the reconciliation error repeatedly: 2021-12-30T23:22:10.443Z ERROR controller-runtime.manager.controller.ssp Reconciler error {"reconciler group": "ssp.kubevirt.io", "reconciler kind": "SSP", "name": "ssp-kubevirt-hyperconverged", "namespace": "openshift-cnv", "error": "Operation cannot be fulfilled on ssps.ssp.kubevirt.io \"ssp-kubevirt-hyperconverged\": the object has been modified; please apply your changes to the latest version and try again"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214 Version-Release number of selected component (if applicable): ------------------------------------------------------------ 4.10.0-545 (pre-release) How reproducible: ---------------- 100% Steps to Reproduce: ------------------ 1. oc patch hco kubevirt-hyperconverged -n openshift-cnv --type merge -p '{"spec": {"featureGates": {"enableCommonBootImageImport": true}}}' 2. oc edit hco -n openshift-cnv 3. add the following entry under the spec: dataImportCronTemplates: - metadata: annotations: cdi.kubevirt.io/storage.bind.immediate.requested: "true" name: my-image-cron spec: garbageCollect: Outdated managedDataSource: my schedule: 59 23/3 * * * template: metadata: {} spec: source: registry: imageStream: my-guest pullMethod: node storage: resources: requests: storage: 10Gi status: {} 4. watch the ssp operator pod logs: oc logs --follow ssp-operator-64b46b479b-2txj8 -n openshift-cnv --tail 1 Actual results: -------------- 1. repeatedly reconciliation errors in ssp operator pod logs: 2021-12-30T23:22:10.443Z ERROR controller-runtime.manager.controller.ssp Reconciler error {"reconciler group": "ssp.kubevirt.io", "reconciler kind": "SSP", "name": "ssp-kubevirt-hyperconverged", "namespace": "openshift-cnv", "error": "Operation cannot be fulfilled on ssps.ssp.kubevirt.io \"ssp-kubevirt-hyperconverged\": the object has been modified; please apply your changes to the latest version and try again"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214 2. ssp status conditions alternately switches between ready conditions (Available: True, Progressing: False, Degraded: False) and not ready conditions (Available: False, Progressing: True, Degraded: True) every now and then (ready conditions for 1-2 seconds and then to not ready for a longer period) Expected results: ---------------- Reconciliation should be successful. Additional info: --------------- Same results if enabling the featuregate "enableCommonBootImageImport".