I think there are two fixes needed:
1. Properly report upgrade failure when it occurs
2. Convert old useNamingPrefix string value to boolean automatically on upgrade

*** Bug 1907139 has been marked as a duplicate of this bug. ***

After talking with Alexander it seems we'll need a conversion webhook and thus certificates and thus certificate rotation in order to solve this properly.  Therefore it's not reasonable to finish in time for 2.5.3.  Pushing out.

After trying a few other things in an attempt to not use a conversion web hook (and the associated certificates), I have not been successful. Either people upgrading from 2.4.x to 2.5 will fail or people upgrading from 2.5.x -> 2.5.x+1 will fail due to the type conversion.

Honestly I don't think there are many people using 2.4.x Host path provisioner, so it might be easier to make a kbase article on how to fix the CR.

The fix is removing the "s from the useNamingPrefix value, so it is considered an actual boolean. It is really is a huge problem we can spend the time working out the certificates for the conversion webhook.

We need to keep in mind that this bug also contains the need to report upgrade failure when it occurs.

So IMO that is a bug in the HCO. The hpp CR is not updated by the operator at all. So the HCO upgraded to hpp operator v2.5.z but the hpp CR is reporting v2.4.z, the HCO should not report success. Then again HCO is NOT managing the hpp CR at all, so I doubt they even check any of that. So the reported success is on managed sub components and not the hpp.

So just to clarify, this bug can be resolved by a KBase article explaining how to resolve an upgrade failure when the old string form of the value was used on the HPP CR.

Audry, we would like to resolve this problem with a KBase article. Do you agree that this is the best path forward? if so, how should we proceed?

Hi Natalie, I think that the KB article idea is fine. The CNV doc writers do not author KB articles though; typically that's done by Engineering or Support. Hope this helps.

As bug scrub with Adam, we move the bug to ON_QA to check the KB and verify the KB on this bug.

Upgraded from OCP4.5.38 + CNV2.4.8 to OCP4.6.26 + CNV2.5.6, HPP provisioner operator Pod was upgraded successfully (v2.4.8-3 -> v2.5.6-3), while HPP provisioner Pods (v2.4.8-3) were not. Please check out the following step 6 for more information.


Here is the verification record. Thank Lukas Bednar and Oren Cohen for your assistance.

1. Check the current version.
[cnv-qe-jenkins@lbednar-4t4cl-executor ~]$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.38    True        False         9h      Cluster version is 4.5.38

[cnv-qe-jenkins@lbednar-4t4cl-executor ~]$ oc get csv -n openshift-cnv

NAME                                      DISPLAY                    VERSION   REPLACES   PHASE
kubevirt-hyperconverged-operator.v2.4.8   OpenShift Virtualization   2.4.8                Succeeded

[cnv-qe-jenkins@lbednar-4t4cl-executor ~]$ oc get ip -n openshift-cnv
NAME            CSV                                       APPROVAL   APPROVED
install-7djpp   kubevirt-hyperconverged-operator.v2.4.8   Manual     true


2. Upgrade OCP.
[cnv-qe-jenkins@lbednar-4t4cl-executor ~]$ oc adm upgrade --to-image quay.io/openshift-release-dev/ocp-release:4.6.26-x86_64 --allow-explicit-upgrade --allow-upgrade-with-warnings --force


3. Monitor OCP upgrade status. It will take over 1h.
[cnv-qe-jenkins@lbednar-4t4cl-executor ~]$ watch oc get clusterversion ; oc get clusteroperators ; oc get nodes


4. Upgrade CNV.
[cnv-qe-jenkins@lbednar-4t4cl-executor ~]$ oc adm catalog mirror registry-proxy.engineering.redhat.com/rh-osbs/iib:73986 registry-proxy.engineering.redhat.com/rh-osbs --manifests-only

It's a new cluster built from production, there is not ImageContentSourcePolicy iib.
[cnv-qe-jenkins@lbednar-4t4cl-executor ~]$ oc create -f iib-manifests/imageContentSourcePolicy.yaml

There's not a CatalogSource. Create a new one.

[cnv-qe-jenkins@lbednar-4t4cl-executor ~]$ cat catalog.yaml
kind: CatalogSource
apiVersion: operators.coreos.com/v1alpha1
metadata:
  name: hco-catalogsource
  namespace: openshift-marketplace
spec:
  displayName: OpenShift Virtualization Index Image
  sourceType: grpc
  image: registry-proxy.engineering.redhat.com/rh-osbs/iib:73986
  publisher: Red Hat

[cnv-qe-jenkins@lbednar-4t4cl-executor ~]$ oc create -f catalog.yaml

Update the current subscription in its spec.source, pointing to hco-catalogsource (the custom one) instead of the production redhat-operators.
[cnv-qe-jenkins@lbednar-4t4cl-executor ~]$ oc get subscription
NAME              PACKAGE                   SOURCE              CHANNEL
hco-operatorhub   kubevirt-hyperconverged   hco-catalogsource   stable

[cnv-qe-jenkins@lbednar-4t4cl-executor ~]$ oc get subscription -o yaml | grep -A7 spec
--
  spec:
    channel: stable
    installPlanApproval: Manual
    name: kubevirt-hyperconverged
    source: hco-catalogsource
    sourceNamespace: openshift-marketplace
    startingCSV: kubevirt-hyperconverged-operator.v2.5.0
  status:

[cnv-qe-jenkins@lbednar-4t4cl-executor ~]$ oc get ip
NAME            CSV                                       APPROVAL   APPROVED
install-7djpp   kubevirt-hyperconverged-operator.v2.4.8   Manual     true
install-bk9sq   kubevirt-hyperconverged-operator.v2.5.6   Manual     false

Approve the new InstallPlan.
[cnv-qe-jenkins@lbednar-4t4cl-executor ~]$ oc patch ip -n openshift-cnv --type merge -p '{"spec":{"approved":true}}' install-bk9sq

[cnv-qe-jenkins@lbednar-4t4cl-executor ~]$ oc get ip
NAME            CSV                                       APPROVAL   APPROVED
install-7djpp   kubevirt-hyperconverged-operator.v2.4.8   Manual     true
install-bk9sq   kubevirt-hyperconverged-operator.v2.5.6   Manual     true


5. The update will roll out automatically. Monitor till succeed.
[cnv-qe-jenkins@lbednar-4t4cl-executor ~]$ oc get csv
NAME                                      DISPLAY                    VERSION   REPLACES                                  PHASE
kubevirt-hyperconverged-operator.v2.5.6   OpenShift Virtualization   2.5.6     kubevirt-hyperconverged-operator.v2.4.8   Succeeded


6. Verify HPP Pods are upgraded.
[cnv-qe-jenkins@lbednar-4t4cl-executor ~]$ oc get pod -n openshift-cnv | grep hostp
hostpath-provisioner-operator-6cd8874d86-mdnzr        1/1     Running   0          1h
hostpath-provisioner-t5tgv                            1/1     Running   0          9h
hostpath-provisioner-xmlv2                            1/1     Running   0          9h
hostpath-provisioner-zxqj4                            1/1     Running   0          9h

[cnv-qe-jenkins@lbednar-4t4cl-executor ~]$ oc get pod -n openshift-cnv hostpath-provisioner-operator-6cd8874d86-mdnzr -o yaml | grep imageID
    imageID: registry.redhat.io/container-native-virtualization/hostpath-provisioner-rhel8-operator@sha256:5f5fa0954e403382e50c4d9d8194696b195f5d27f90d5b5f2a363d3f9e436a83
(It's build hostpath-provisioner-operator-container-v2.5.6-3)

[cnv-qe-jenkins@lbednar-4t4cl-executor ~]$ oc get pod -n openshift-cnv hostpath-provisioner-t5tgv -o yaml | grep imageID
    imageID: registry.redhat.io/container-native-virtualization/hostpath-provisioner-rhel8@sha256:46cada800534119068b117fcb1db06766c1e949a062b57e877d46988fba18031
(It's build hostpath-provisioner-container-v2.4.8-3)

[cnv-qe-jenkins@lbednar-4t4cl-executor ~]$ oc get hostpathprovisioner
NAME                   AGE
hostpath-provisioner   9h

[cnv-qe-jenkins@lbednar-4t4cl-executor ~]$ oc get hostpathprovisioner -o yaml
apiVersion: v1
items:
- apiVersion: hostpathprovisioner.kubevirt.io/v1beta1
  kind: HostPathProvisioner
  metadata:
    creationTimestamp: "2021-05-11T06:39:48Z"
    finalizers:
    - finalizer.delete.hostpath-provisioner
    generation: 42
    managedFields:
    - apiVersion: hostpathprovisioner.kubevirt.io/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:spec:
          .: {}
          f:imagePullPolicy: {}
          f:pathConfig:
            .: {}
            f:path: {}
            f:useNamingPrefix: {}
      manager: kubectl-create
      operation: Update
      time: "2021-05-11T06:39:48Z"
    - apiVersion: hostpathprovisioner.kubevirt.io/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:finalizers: {}
        f:status:
          .: {}
          f:conditions: {}
          f:observedVersion: {}
          f:operatorVersion: {}
          f:targetVersion: {}
      manager: hostpath-provisioner-operator
      operation: Update
      time: "2021-05-11T14:33:45Z"
    name: hostpath-provisioner
    resourceVersion: "566819"
    selfLink: /apis/hostpathprovisioner.kubevirt.io/v1beta1/hostpathprovisioners/hostpath-provisioner
    uid: 1db66826-76b5-41c9-aa7f-b0f7dd181dbf
  spec:
    imagePullPolicy: IfNotPresent
    pathConfig:
      path: /var/hpvolumes
      useNamingPrefix: "false"
  status:
    conditions:
    - lastHeartbeatTime: "2021-05-11T14:33:45Z"
      lastTransitionTime: "2021-05-11T06:40:01Z"
      status: "True"
      type: Available
    - lastHeartbeatTime: "2021-05-11T14:33:45Z"
      lastTransitionTime: "2021-05-11T06:40:01Z"
      status: "False"
      type: Progressing
    - lastHeartbeatTime: "2021-05-11T14:33:45Z"
      lastTransitionTime: "2021-05-11T11:19:11Z"
      status: "False"
      type: Degraded
    observedVersion: v2.4.8
    operatorVersion: v2.4.8
    targetVersion: v2.4.8
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Followed KBase https://access.redhat.com/solutions/5877411, move quote from useNamingPrefix section on HOSTPATHPROVISIONER: useNamingPrefix: "false" -> useNamingPrefix: false, HPP provisioner Pods restarted.

[cnv-qe-jenkins@lbednar-4t4cl-executor ~]$ oc get pod -n openshift-cnv | grep hostp
hostpath-provisioner-8ms26                            1/1     Running   0          41s
hostpath-provisioner-bk6cf                            1/1     Running   0          28s
hostpath-provisioner-bq52x                            1/1     Running   0          20s
hostpath-provisioner-operator-6cd8874d86-mdnzr        1/1     Running   0          15h

[cnv-qe-jenkins@lbednar-4t4cl-executor ~]$ oc get pod -n openshift-cnv hostpath-provisioner-8ms26 -o yaml | grep image
            f:image: {}
            f:imagePullPolicy: {}
    image: registry.redhat.io/container-native-virtualization/hostpath-provisioner-rhel8@sha256:78eb1767a467c4af256a14bd0de3e959acbbac3f9b68df98d0b99969c088e7fc
    imagePullPolicy: IfNotPresent
  imagePullSecrets:
    image: registry.redhat.io/container-native-virtualization/hostpath-provisioner-rhel8@sha256:78eb1767a467c4af256a14bd0de3e959acbbac3f9b68df98d0b99969c088e7fc
    imageID: registry.redhat.io/container-native-virtualization/hostpath-provisioner-rhel8@sha256:544bd201bd6f3e684c6cf194eaf131edfd31023e33fdbc91286ae7e87cbbf266

It's build hostpath-provisioner-container-v2.5.6-2


The bug has been fixed.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Virtualization 2.5.6 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:2045