Created attachment 1875704 [details]
virt_operator1

Created attachment 1875705 [details]
virt_operator2

Is it possible to reproduce this?

Deferring to the next release due to capacity.

Must gather is being attached. Since this is impacting smoke on 4.12, adding testblocker label.

When a node maintenance CR is created, I see kubevirt.status.conditions continuing to stay, till the cr is deleted:
============================
[cnv-qe-jenkins@c01-dbn-4012-8k679-executor ~]$ kubectl get kubevirt kubevirt-kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.conditions"
[
  {
    "lastProbeTime": "2022-06-29T21:32:36Z",
    "lastTransitionTime": "2022-06-29T21:32:36Z",
    "message": "Deploying version sha256:f9904655a1c579b7db4f55f621852795397c2a01c72d0c420c916ec8f0466024 with registry registry.redhat.io/container-native-virtualization",
    "reason": "DeploymentInProgress",
    "status": "False",
    "type": "Available"
  },
  {
    "lastProbeTime": "2022-06-29T21:32:36Z",
    "lastTransitionTime": "2022-06-29T21:32:36Z",
    "message": "Deploying version sha256:f9904655a1c579b7db4f55f621852795397c2a01c72d0c420c916ec8f0466024 with registry registry.redhat.io/container-native-virtualization",
    "reason": "DeploymentInProgress",
    "status": "True",
    "type": "Progressing"
  },
  {
    "lastProbeTime": "2022-06-29T21:32:36Z",
    "lastTransitionTime": "2022-06-29T21:32:36Z",
    "message": "Deploying version sha256:f9904655a1c579b7db4f55f621852795397c2a01c72d0c420c916ec8f0466024 with registry registry.redhat.io/container-native-virtualization",
    "reason": "DeploymentInProgress",
    "status": "False",
    "type": "Degraded"
  },
  {
    "lastProbeTime": "2022-06-24T03:59:37Z",
    "lastTransitionTime": null,
    "message": "All resources were created.",
    "reason": "AllResourcesCreated",
    "status": "True",
    "type": "Created"
  }
]
[cnv-qe-jenkins@c01-dbn-4012-8k679-executor ~]$
============================

(In reply to Debarati Basu-Nag from comment #15)
> When a node maintenance CR is created, I see kubevirt.status.conditions
> continuing to stay, till the cr is deleted:
> ============================
> [cnv-qe-jenkins@c01-dbn-4012-8k679-executor ~]$ kubectl get kubevirt
> kubevirt-kubevirt-hyperconverged -n openshift-cnv -o json | jq
> ".status.conditions"
> [
>   {
>     "lastProbeTime": "2022-06-29T21:32:36Z",
>     "lastTransitionTime": "2022-06-29T21:32:36Z",
>     "message": "Deploying version
> sha256:f9904655a1c579b7db4f55f621852795397c2a01c72d0c420c916ec8f0466024 with
> registry registry.redhat.io/container-native-virtualization",
>     "reason": "DeploymentInProgress",
>     "status": "False",
>     "type": "Available"
>   },
>   {
>     "lastProbeTime": "2022-06-29T21:32:36Z",
>     "lastTransitionTime": "2022-06-29T21:32:36Z",
>     "message": "Deploying version
> sha256:f9904655a1c579b7db4f55f621852795397c2a01c72d0c420c916ec8f0466024 with
> registry registry.redhat.io/container-native-virtualization",
>     "reason": "DeploymentInProgress",
>     "status": "True",
>     "type": "Progressing"
>   },
>   {
>     "lastProbeTime": "2022-06-29T21:32:36Z",
>     "lastTransitionTime": "2022-06-29T21:32:36Z",
>     "message": "Deploying version
> sha256:f9904655a1c579b7db4f55f621852795397c2a01c72d0c420c916ec8f0466024 with
> registry registry.redhat.io/container-native-virtualization",
>     "reason": "DeploymentInProgress",
>     "status": "False",
>     "type": "Degraded"
>   },
>   {
>     "lastProbeTime": "2022-06-24T03:59:37Z",
>     "lastTransitionTime": null,
>     "message": "All resources were created.",
>     "reason": "AllResourcesCreated",
>     "status": "True",
>     "type": "Created"
>   }
> ]
> [cnv-qe-jenkins@c01-dbn-4012-8k679-executor ~]$
> ============================

This is expected behavior as one of our handlers will be missing. We can look at his if we can improve here but this is not related to the issue described here.

*** Bug 2099635 has been marked as a duplicate of this bug. ***

Following my observations, I can see that upon changing the Kubevirt CR, Kubevirt unconditionally updates its operands with the `kubevirt.io/generation` annotation even though the operand doesn't require re-conciliation. For example when adding the `Spec.configuration.cpuModel` field in the KV CR, this will cause change in the `kubevirt.io/generation` field of each operand. 

This is indeed an issue, it may trigger some status flip-flop such as that described in the bug. 

However the status eventually recovers and very quickly, I can see it in the reproductions. IMO this bug is not a blocker one. It may be a blocker if the status in KV CR is really stuck at deploying, but this
is not the case.

Per Comment #19, the impact of this issue on clusters with OpenShift CNV deployed is that unnecessary reconciliations can occur. This was due to a conscious design decision to keep virt-operator operands with a common generation annotation. However, there doesn't appear to be an obvious reason why that is necessary. That is still being investigated.

Regardless of whether there is worth in reconciling resources that otherwise did not change, this transient will occur extremely rapidly and should not even be noticed in typical deployments.

Reconciliation also of course only occurs in the first place if the HCO CR was modified, so is generally expected to be infrequent.

Because there is no danger of loss of data, and the cluster's ability to upgrade will not be impaired, risk of disruption to a cluster is minimal. Consequently we've removed the blocker flag and deferred this BZ to the next major release.

To clarify comment #22, "extremely rapidly" isn't clear enough. The reconciliation is usually just a few seconds. Almost always less than 5 seconds.

Verified - we didn't see the issue during automation runs

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Virtualization 4.12.0 Images security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:0408