Bug 1825349

Summary: The installed operator cannot be upgraded after the cluster upgraded to 4.4 from 4.3
Product: OpenShift Container Platform Reporter: Alexander Greene <agreene>
Component: OLMAssignee: Alexander Greene <agreene>
OLM sub component: OLM QA Contact: Jian Zhang <jiazha>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: urgent CC: ecordell, jiazha, scuppett
Version: 4.4   
Target Milestone: ---   
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-04 11:49:21 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1825088    
Bug Blocks:    

Description Alexander Greene 2020-04-17 18:47:51 UTC
This bug was initially created as a copy of Bug #1825088

I am copying this bug because: 
This bug exists in 4.4 as well


Description of problem:
The installed operator cannot be upgraded due to the lack of 'generation' field of the InstallPlan after the cluster upgraded to 4.4 from 4.3, errors as follows:

E0416 16:32:18.613181       1 queueinformer_operator.go:290] sync {"update" "openshift-logging/install-tmllk"} failed: error updating InstallPlan status: InstallPlan.operators.coreos.com "install-tmllk" is invalid: spec.generation: Required value

mac:~ jianzhang$ oc get ip -n openshift-logging install-tmllk  -o=jsonpath='{.spec}'
map[approval:Automatic approved:true clusterServiceVersionNames:[clusterlogging.4.3.10-202004010435]]




Version-Release number of selected component (if applicable):
4.4.0-0.nightly-2020-04-16-094508

OLM version:
$oc exec catalog-operator-56b67964ff-cchtq  -- olm --version
OLM version: 0.14.2
git commit: 2248cc6bea92a03b450e773b84ef8fa924c9c1c9

How reproducible:
This cluster. 

Steps to Reproduce:
1, Install OCP 4.3
2. Deploy cluster-logging from redhat-operators 
3, Upgrade the OCP 4.3 to 4.4
4. Create a OperatorSource to use QE app registry.
mac:~ jianzhang$ oc get operatorsource -n openshift-marketplace qe-app-registry  -o yaml
apiVersion: operators.coreos.com/v1
kind: OperatorSource
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"operators.coreos.com/v1","kind":"OperatorSource","metadata":{"annotations":{},"name":"qe-app-registry","namespace":"openshift-marketplace"},"spec":{"authorizationToken":{"secretName":"qesecret"},"endpoint":"https://quay.io/cnr","registryNamespace":"openshift-qe-optional-operators","type":"appregistry"}}
  creationTimestamp: "2020-04-16T06:50:36Z"
  finalizers:
  - finalizer.operatorsources.operators.coreos.com
  generation: 75
  name: qe-app-registry
  namespace: openshift-marketplace
  resourceVersion: "675372"
  selfLink: /apis/operators.coreos.com/v1/namespaces/openshift-marketplace/operatorsources/qe-app-registry
  uid: 9389b6aa-7fae-11ea-8773-02505d469966
spec:
  authorizationToken:
    secretName: qesecret
  endpoint: https://quay.io/cnr
  registryNamespace: openshift-qe-optional-operators
  type: appregistry
status:
  currentPhase:
    lastTransitionTime: "2020-04-17T01:01:33Z"
    lastUpdateTime: "2020-04-17T01:01:33Z"
    phase:
      message: The object has been successfully reconciled
      name: Succeeded
  packages: openshifttemplateservicebroker,metering-ocp,nfd,ptp-operator,elasticsearch-operator,cluster-logging,local-storage-operator,sriov-network-operator,clusterresourceoverride,cluster-kube-descheduler-operator,openshiftansibleservicebroker
  
3. Check the csv if upgraded successfully.


Actual results:
# the csv wasn't upgraded to 4.4.0
oc get csv
NAME                                         DISPLAY                  VERSION               REPLACES   PHASE
clusterlogging.4.3.10-202004010435           Cluster Logging          4.3.10-202004010435              Succeeded
elasticsearch-operator.4.3.10-202003311428   Elasticsearch Operator   4.3.10-202003311428              Succeeded

Expected result
# the csv is upgraded to 4.4.0-202004150031
oc get csv
NAME                                         DISPLAY                  VERSION               REPLACES   PHASE
clusterlogging.4.4.0-202004150031           Cluster Logging          4.4.0-202004150031              Succeeded
elasticsearch-operator.4.4.0-202004150031   Elasticsearch Operator   4.4.0-202004150031              Succeeded

Addtional info:

Comment 1 Stephen Cuppett 2020-04-20 17:32:44 UTC
*** Bug 1825845 has been marked as a duplicate of this bug. ***

Comment 6 Jian Zhang 2020-04-23 07:19:34 UTC
1, Create a 4.3 cluster.
mac:~ jianzhang$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.3.15    True        False         113m    Cluster version is 4.3.15

2, Install a new OperatorSource called `qe-app-registry` that contains the cluster-logging operator.
mac:~ jianzhang$ oc get operatorsource -n openshift-marketplace qe-app-registry -o yaml
apiVersion: operators.coreos.com/v1
kind: OperatorSource
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"operators.coreos.com/v1","kind":"OperatorSource","metadata":{"annotations":{},"name":"qe-app-registry","namespace":"openshift-marketplace"},"spec":{"authorizationToken":{"secretName":"qesecret"},"endpoint":"https://quay.io/cnr","registryNamespace":"openshift-qe-optional-operators","type":"appregistry"}}
  creationTimestamp: "2020-04-23T03:50:24Z"
  finalizers:
  - finalizer.operatorsources.operators.coreos.com
  generation: 19
  name: qe-app-registry
  namespace: openshift-marketplace
  resourceVersion: "94168"
  selfLink: /apis/operators.coreos.com/v1/namespaces/openshift-marketplace/operatorsources/qe-app-registry
  uid: b75c4b23-8956-464b-a941-774f17a9381c
spec:
  authorizationToken:
    secretName: qesecret
  endpoint: https://quay.io/cnr
  registryNamespace: openshift-qe-optional-operators
  type: appregistry
status:
  currentPhase:
    lastTransitionTime: "2020-04-23T06:48:23Z"
    lastUpdateTime: "2020-04-23T06:48:23Z"
    phase:
      message: The object has been successfully reconciled
      name: Succeeded
  packages: clusterresourceoverride,cluster-kube-descheduler-operator,metering-ocp,nfd,ptp-operator,local-storage-operator,sriov-network-operator,elasticsearch-operator,cluster-logging,openshifttemplateservicebroker,openshiftansibleservicebroker

3, Subscribe the cluster-logging operator from the `redhat-operators` OperatorSource.
mac:~ jianzhang$ oc get sub 
NAME              PACKAGE           SOURCE             CHANNEL
cluster-logging   cluster-logging   redhat-operators   4.3
mac:~ jianzhang$ oc get ip
NAME            CSV                                  APPROVAL    APPROVED
install-bpbfz   clusterlogging.4.3.14-202004222058   Automatic   true
mac:~ jianzhang$ oc get csv
NAME                                         DISPLAY                  VERSION               REPLACES   PHASE
clusterlogging.4.3.14-202004222058           Cluster Logging          4.3.14-202004222058              Succeeded
elasticsearch-operator.4.3.14-202004221157   Elasticsearch Operator   4.3.14-202004221157              Succeeded

mac:~ jianzhang$ oc get sub  cluster-logging  -o=jsonpath='{.spec}'
map[channel:4.3 installPlanApproval:Automatic name:cluster-logging source:redhat-operators sourceNamespace:openshift-marketplace]

No `generation` field in it.
mac:~ jianzhang$ oc get ip install-bpbfz   -o=jsonpath='{.spec}'
map[approval:Automatic approved:true clusterServiceVersionNames:[clusterlogging.4.3.14-202004222058]

4, Upgrade this 4.3 cluster to 4.4.

mac:~ jianzhang$ oc patch clusterversion/version --patch '{"spec":{"upstream":"https://openshift-release.svc.ci.openshift.org/graph"}}' --type=merge
clusterversion.config.openshift.io/version patched
mac:~ jianzhang$ oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release:4.4.0-rc.10-x86_64 --force=true --allow-explicit-upgrade
Updating to release image quay.io/openshift-release-dev/ocp-release:4.4.0-rc.10-x86_64

mac:~ jianzhang$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.3.15    True        True          28s     Working towards 4.4.0-rc.10: 11% complete

mac:~ jianzhang$ oc get clusterversion
NAME      VERSION       AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-rc.10   True        False         4m11s   Cluster version is 4.4.0-rc.10

5, Update its subscription object, change the `channel` to 4.4, and the `source` to qe-app-registry.

mac:~ jianzhang$ oc edit sub cluster-logging
subscription.operators.coreos.com/cluster-logging edited
mac:~ jianzhang$ oc get sub cluster-logging -o=jsonpath='{.spec}'
map[channel:4.4 installPlanApproval:Automatic name:cluster-logging source:qe-app-registry sourceNamespace:openshift-marketplace]

The cluster-logging operator can be upgraded successfully. The `generation` field is optional now. LGTM, verify it.
mac:~ jianzhang$ oc get sub
NAME              PACKAGE           SOURCE            CHANNEL
cluster-logging   cluster-logging   qe-app-registry   4.4

mac:~ jianzhang$ oc get csv
NAME                                         DISPLAY                  VERSION               REPLACES                             PHASE
clusterlogging.4.4.0-202004211517            Cluster Logging          4.4.0-202004211517    clusterlogging.4.3.14-202004222058   Succeeded
elasticsearch-operator.4.3.14-202004221157   Elasticsearch Operator   4.3.14-202004221157                                        Succeeded

mac:~ jianzhang$ oc get pods
NAME                                        READY   STATUS    RESTARTS   AGE
cluster-logging-operator-579b4c8fb6-mqs9t   1/1     Running   0          5m46s

mac:~ jianzhang$ oc get ip
NAME            CSV                                  APPROVAL    APPROVED
install-4sjm7   clusterlogging.4.4.0-202004211517    Automatic   true
install-bpbfz   clusterlogging.4.3.14-202004222058   Automatic   true


mac:~ jianzhang$ oc get ip install-4sjm7 -o=jsonpath='{.spec}'
map[approval:Automatic approved:true clusterServiceVersionNames:[clusterlogging.4.4.0-202004211517] generation:1]
mac:~ jianzhang$ oc get ip install-bpbfz -o=jsonpath='{.spec}'
map[approval:Automatic approved:true clusterServiceVersionNames:[clusterlogging.4.3.14-202004222058]]

Comment 8 errata-xmlrpc 2020-05-04 11:49:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581