Bug 1882394

PR [1] has been approved, waiting for patch manager to tag it.

[1] https://github.com/openshift/cluster-storage-operator/pull/91

Hi Fabio, I performed an downgrade from 4.6.0-rc.0 to 4.5.0-0.nightly-2020-10-07-231808(should contain the fix) but re-produced this problem.

Storage co did not roll out: 
    [wduan@MINT 01_general]$ oc get clusterversion
    NAME      VERSION      AVAILABLE   PROGRESSING   SINCE   STATUS
    version   4.6.0-rc.0   True        True          40m     Unable to apply 4.5.0-0.nightly-2020-10-07-231808: the cluster operator storage has not yet successfully rolled out

    [wduan@MINT verification-tests]$ oc get co storage
    NAME      VERSION      AVAILABLE   PROGRESSING   DEGRADED   SINCE
    storage   4.6.0-rc.0   True        False         False      3h34m

From cluster-storage-operator log, ConfigMap lock looks like be deleted(?), but operator was not able to become the leader:
    [wduan@MINT 01_general]$ oc -n openshift-cluster-storage-operator logs pod/cluster-storage-operator-86d6fbc996-7l8z7
    {"level":"info","ts":1602214988.450045,"logger":"cmd","msg":"Go Version: go1.13.4"}
    {"level":"info","ts":1602214988.450071,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
    {"level":"info","ts":1602214988.4500754,"logger":"cmd","msg":"Version of operator-sdk: v0.4.0"}
    {"level":"info","ts":1602214988.897236,"logger":"cmd","msg":"Found ConfigMap lock without metadata.ownerReferences, deleting"}
    {"level":"info","ts":1602214988.950875,"logger":"leader","msg":"Trying to become the leader."}
    {"level":"info","ts":1602214989.1881082,"logger":"leader","msg":"Not the leader. Waiting."}
    {"level":"info","ts":1602214990.3181906,"logger":"leader","msg":"Not the leader. Waiting."}
    ...
    {"level":"info","ts":1602226999.4777808,"logger":"leader","msg":"Not the leader. Waiting."}
    {"level":"info","ts":1602227016.1669917,"logger":"leader","msg":"Not the leader. Waiting."}

cluster-storage-operator-lock CM:
    [wduan@MINT verification-tests]$ oc -n openshift-cluster-storage-operator get cm cluster-storage-operator-lock -o yaml
    apiVersion: v1
    kind: ConfigMap
    metadata:
    annotations:
        control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"","leaseDurationSeconds":0,"acquireTime":null,"renewTime":null,"leaderTransitions":0}'
    creationTimestamp: "2020-10-09T03:43:08Z"
    managedFields:
    - apiVersion: v1
        fieldsType: FieldsV1
        fieldsV1:
        f:metadata:
            f:annotations:
            .: {}
            f:control-plane.alpha.kubernetes.io/leader: {}
        manager: cluster-storage-operator
        operation: Update
        time: "2020-10-09T03:43:09Z"
    name: cluster-storage-operator-lock
    namespace: openshift-cluster-storage-operator
    resourceVersion: "54649"
    selfLink: /api/v1/namespaces/openshift-cluster-storage-operator/configmaps/cluster-storage-operator-lock
    uid: 59894bf2-5b37-48be-9006-6cf08b427e2c

CSO 4.5 deleted the ConfigMap here:

> {"level":"info","ts":1602214988.897236,"logger":"cmd","msg":"Found ConfigMap lock without metadata.ownerReferences, deleting"}

This epoch translates to: Friday, October 9, 2020 3:43:08.950 AM

However, the ConfigMap from the description was created on "2020-10-09T03:43:08Z".

This means that something else (CSO 4.6) created the ConfigMap right after it was deleted by CSO 4.5.

Apparently both CSOs were running at the same time for a short period of time. This is possible because they utilize different leader election approaches.

Hi Fabio, 

With the https://github.com/openshift/cluster-storage-operator/pull/96, we verified pass for upgrade/downgrade for path 4.5 -> 4.6 -> 4.5

*** Bug 1877899 has been marked as a duplicate of this bug. ***

Verified pass.
Perform upgarde/downgrade successfully for 4.5 <-> 4.5 and 4.5 <-> 4.6, also checked ci upgrade from 4.4 is success. So change the status to VERIFIED.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.5.16 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4268

*** Bug 1877899 has been marked as a duplicate of this bug. ***