Bug 2073880

Created attachment 1871697 [details]
error_on_wizard

I think the better solution for this problem is enabling live migration on SNO instead of fixing the templates.

HCO started explicitly ignoring liveMigration feature gate livemigration eviction strategy on SNO (we a single node you cannot migrate a VM anywhere!) as a workaround for a virt-controller bug/behavior that was continuously and infinitely trying to migrate a VM on upgrades even with a single node, see: https://bugzilla.redhat.com/show_bug.cgi?id=2065308

Reverting that choice is going to reopen 2065308

The workloadUpdateStrategy in HCO looks like below:
```
    workloadUpdateStrategy:
      batchEvictionInterval: 1m0s
      batchEvictionSize: 10
      workloadUpdateMethods:
      - LiveMigrate
```

See the same error from CLI:
$ oc process rhel8-server-small -n openshift -p NAME=test -p DATA_SOURCE_NAME=rhel8-7b20b8e2e1c6 -p DATA_SOURCE_NAMESPACE=openshift-virtualization-os-images | oc create -f -
The request is invalid: spec.template.spec.evictionStrategy: LiveMigration feature gate is not enabled

Move the bug to virt as it's reproduced on CLI.
CNV version: CNV-v4.10.1-60

Raising the priority of this to Urgent as I concur that it is a blocker.

We've explored 3 different options:

1) remove the evictionStrategy from templates only in SNO mode. This has the downside that it requires an API change to the virt-ssp-operator, and is a temporary fix, so this is not ideal.
2) re-enable the feature gate in HCO. This is not so nice because it's highly visible and might have second order effects. See discussion in: https://bugzilla.redhat.com/show_bug.cgi?id=2065308
3) for SNO mode only, we will simply ignore evictionStrategy in virt-api's webhook.

We're planning to go with the third option since it has no real drawbacks as compared to the other strategies.

Reflecting on Comment #7, there is actually a significant downside to ignoring the evictionStrategy. If that is all that's done, it would block upgrades--which is also a big problem. While that could also be addressed, it's no longer a quick one-line fix.

Removing evictionStrategy from templates is likely the more sensible approach.

FYI Simone, Krzysztof.

If a new API is made available, HCO can easily communicate the SNO mode to the ssp-operator so that it can  dynamically fine tune the templates.

Patch https://github.com/kubevirt/ssp-operator/pull/343 posted by Federico.

[kbidarka@localhost auth]$ oc get templates -n openshift windows10-desktop-medium -o yaml | grep evictionStrategy
        evictionStrategy: LiveMigrate
[kbidarka@localhost auth]$ oc get templates -n openshift rhel8-server-medium -o yaml | grep evictionStrategy
        evictionStrategy: LiveMigrate
[kbidarka@localhost auth]$ 
[kbidarka@localhost auth]$ oc get kubevirt kubevirt-kubevirt-hyperconverged -n openshift-cnv -o yaml | grep LiveMigrate 
    - LiveMigrate
[kbidarka@localhost auth]$ oc get infrastructure.config.openshift.io cluster -o json | jq ".status.infrastructureTopology"
"SingleReplica"

VERIFIED with v4.10.1-97

[kbidarka@localhost auth]$ oc get infrastructure.config.openshift.io cluster -o json | jq ".status.infrastructureTopology"
"SingleReplica"

[kbidarka@localhost manifest]$ oc -n openshift process rhel8-server-medium -p NAME=vm-rhel8-ocs -p CLOUD_USER_PASSWORD=redhat -o yaml > vm-rhel8-ocs.yaml

[kbidarka@localhost manifest]$ cat vm-rhel8-ocs.yaml | grep eviction
        evictionStrategy: LiveMigrate

[kbidarka@localhost manifest]$ oc apply -f vm-rhel8-ocs.yaml
virtualmachine.kubevirt.io/vm-rhel8-ocs created

[kbidarka@localhost manifest]$ virtctl start vm-rhel8-ocs
VM vm-rhel8-ocs was scheduled to start

[kbidarka@localhost manifest]$ oc get vmi 
NAME               AGE    PHASE     IP             NODENAME                                         READY
rhel8-loud-swift   31m    Running   11.xx.yy.zz   node-23.redhat.com   True
vm-rhel8-ocs       4m8s   Running   12.yy.zz.aa   node-23.redhat.com   True

[kbidarka@localhost manifest]$ virtctl console vm-rhel8-ocs
Successfully connected to vm-rhel8-ocs console. The escape sequence is ^]

Red Hat Enterprise Linux 8.5 (Ootpa)
Kernel 4.18.0-348.23.1.el8_5.x86_64 on an x86_64

Activate the web console with: systemctl enable --now cockpit.socket

vm-rhel8-ocs login: cloud-user
Password: 
Last failed login: Sun May  1 10:19:11 EDT 2022 on ttyS0
There was 1 failed login attempt since the last successful login.
[cloud-user@vm-rhel8-ocs ~]$ 
[cloud-user@vm-rhel8-ocs ~]$ 
[cloud-user@vm-rhel8-ocs ~]$ 
[cloud-user@vm-rhel8-ocs ~]$ [kbidarka@localhost manifest]$ 
[kbidarka@localhost manifest]$

VERIFIED with v4.10.1-97

Adding known issue to the OpenShift Virtualization 4.10.1 release notes

Known issue:
When a VMI has the spec.evictionStrategy field set to LiveMigrate, the VMIs can't be migrated or evicted and the upgrade fails.

By default, VM Templates installed by the SSP operator have the spec.evictionStrategy field set to LiveMigrate. The LiveMigration feature gate is enabled even when the cluster, in SNO mode, has only 1 worker node. 

Workaround: 
You must implement one of the following workarounds to upgrade the cluster:

1. Remove the spec.evictionStrategy field from the VM declaration.

2. Manually stop the VM before starting a OpenShift Virtualization upgrade.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 4.10.1 Images security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:4668