Bug 1731123

Summary: OLM packageserver upgrade failed from 4.1.6 to 4.2
Product: OpenShift Container Platform Reporter: Jian Zhang <jiazha>
Component: OLMAssignee: Evan Cordell <ecordell>
OLM sub component: OLM QA Contact: Jian Zhang <jiazha>
Status: CLOSED DUPLICATE Docs Contact:
Severity: medium    
Priority: high CC: bandrade, chezhang, chuo, jfan, scolange, zitang
Version: 4.1.z   
Target Milestone: ---   
Target Release: 4.1.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-07-19 02:22:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jian Zhang 2019-07-18 11:22:19 UTC 
Description of problem:
After the cluster upgrade to 4.2 from 4.1.6, the `operator-lifecycle-manager-packageserver` state in False status and no VERSION info, like below:
mac:~ jianzhang$ oc get clusteroperator
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
...
operator-lifecycle-manager                 4.2.0-0.nightly-2019-07-18-032637   True        False         False      161m
operator-lifecycle-manager-catalog         4.2.0-0.nightly-2019-07-18-032637   True        False         False      161m
operator-lifecycle-manager-packageserver                                       False       False         False      38m

Version-Release number of selected component (if applicable):
sh-4.2$ olm --version
OLM version: 0.10.1
git commit: 7367a8f

How reproducible:
always

Steps to Reproduce:
1. Install OCP 4.1.6
2. Upgrade it to 4.2


Actual results:
1) Clusteroperator packageserver failed, and no version info report, lacks `versions` field.
2) Packageserver csv failed.
mac:~ jianzhang$ oc get csv -n openshift-operator-lifecycle-manager
NAME                   DISPLAY          VERSION   REPLACES   PHASE
packageserver          Package Server   0.10.1               Failed
packageserver.v0.9.0   Package Server   0.9.0                Succeeded

mac:~ jianzhang$ oc get clusteroperator operator-lifecycle-manager-packageserver -o yaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: "2019-07-18T10:06:35Z"
  generation: 1
  name: operator-lifecycle-manager-packageserver
  resourceVersion: "66892"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/operator-lifecycle-manager-packageserver
  uid: b9eb8651-a943-11e9-9a61-066433738abc
spec: {}
status:
  conditions:
  - lastTransitionTime: "2019-07-18T10:06:35Z"
    status: "False"
    type: Degraded
  - lastTransitionTime: "2019-07-18T10:06:35Z"
    status: "False"
    type: Available
  - lastTransitionTime: "2019-07-18T10:08:47Z"
    message: Failed to deploy 0.10.1
    status: "False"
    type: Progressing
  extension: null
  relatedObjects:
  - group: ""
    name: openshift-operator-lifecycle-manager
    resource: namespaces
  - group: operators.coreos.com
    name: packageserver
    namespace: openshift-operator-lifecycle-manager
    resource: ClusterServiceVersion


Expected results:
OLM packageserver upgrade success.

Additional info:
Packageserver still is 0.9.0 version, failed to upgrade to 0.10.1. Like below:
mac:~ jianzhang$ oc get pods
NAME                               READY   STATUS    RESTARTS   AGE
catalog-operator-7cdff8f4d-w8nwc   1/1     Running   0          49m
olm-operator-69d54b799c-nln8f      1/1     Running   0          49m
olm-operators-26wsf                1/1     Running   0          80m
packageserver-8c65d6877-j9hh9      1/1     Running   0          75m
packageserver-8c65d6877-r77gn      1/1     Running   0          75m

mac:~ jianzhang$ oc rsh packageserver-8c65d6877-j9hh9
sh-4.2$ olm --version
OLM version: 0.9.0
git commit: 4878aa4

Failed packageserver csv info:  `reason: OwnerConflict`
mac:~ jianzhang$ oc get csv -n openshift-operator-lifecycle-manager packageserver -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
  annotations:
    olm.operatorGroup: olm-operators
    olm.operatorNamespace: openshift-operator-lifecycle-manager
    olm.targetNamespaces: openshift-operator-lifecycle-manager
  creationTimestamp: "2019-07-18T10:08:47Z"
  generation: 12
  labels:
    olm.api.4bca9f23e412d79d: provided
    olm.clusteroperator.name: operator-lifecycle-manager-packageserver
    olm.version: 0.10.1
  name: packageserver
  namespace: openshift-operator-lifecycle-manager
  resourceVersion: "81263"
  selfLink: /apis/operators.coreos.com/v1alpha1/namespaces/openshift-operator-lifecycle-manager/clusterserviceversions/packageserver
  uid: 083a5387-a944-11e9-9a61-066433738abc
spec:
  apiservicedefinitions:
    owned:
    - containerPort: 5443
      deploymentName: packageserver
      description: A PackageManifest is a resource generated from existing CatalogSources
        and their ConfigMaps
      displayName: PackageManifest
      group: packages.operators.coreos.com
      kind: PackageManifest
      name: packagemanifests
      version: v1
  customresourcedefinitions: {}
  description: Represents an Operator package that is available from a given CatalogSource
    which will resolve to a ClusterServiceVersion.
  displayName: Package Server
  install:
    spec:
      clusterPermissions:
      - rules:
        - apiGroups:
          - authorization.k8s.io
          resources:
          - subjectaccessreviews
          verbs:
          - create
          - get
        - apiGroups:
          - ""
          resources:
          - configmaps
          verbs:
          - get
          - list
          - watch
        - apiGroups:
          - operators.coreos.com
          resources:
          - catalogsources
          verbs:
          - get
          - list
          - watch
        - apiGroups:
          - packages.operators.coreos.com
          resources:
          - packagemanifests
          verbs:
          - get
          - list
        serviceAccountName: olm-operator-serviceaccount
      deployments:
      - name: packageserver
        spec:
          replicas: 2
          selector:
            matchLabels:
              app: packageserver
          strategy:
            type: RollingUpdate
          template:
            metadata:
              labels:
                app: packageserver
            spec:
              containers:
              - command:
                - /bin/package-server
                - -v=4
                - --secure-port
                - "5443"
                - --global-namespace
                - openshift-marketplace
                image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da6b38c9fcc4db7401ced116ff763e6c3adef4f1aa4d44f3daefba60328a0d3e
                imagePullPolicy: IfNotPresent
                livenessProbe:
                  httpGet:
                    path: /healthz
                    port: 5443
                    scheme: HTTPS
                name: packageserver
                ports:
                - containerPort: 5443
                readinessProbe:
                  httpGet:
                    path: /healthz
                    port: 5443
                    scheme: HTTPS
                terminationMessagePolicy: FallbackToLogsOnError
              nodeSelector:
                beta.kubernetes.io/os: linux
                node-role.kubernetes.io/master: ""
              priorityClassName: system-cluster-critical
              serviceAccountName: olm-operator-serviceaccount
              tolerations:
              - effect: NoSchedule
                key: node-role.kubernetes.io/master
                operator: Exists
              - effect: NoExecute
                key: node.kubernetes.io/unreachable
                operator: Exists
                tolerationSeconds: 120
              - effect: NoExecute
                key: node.kubernetes.io/not-ready
                operator: Exists
                tolerationSeconds: 120
    strategy: deployment
  installModes:
  - supported: true
    type: OwnNamespace
  - supported: true
    type: SingleNamespace
  - supported: true
    type: MultiNamespace
  - supported: true
    type: AllNamespaces
  keywords:
  - packagemanifests
  - olm
  - packages
  links:
  - name: Package Server
    url: https://github.com/operator-framework/operator-lifecycle-manager/tree/master/pkg/package-server
  maintainers:
  - email: openshift-operators
    name: Red Hat
  maturity: alpha
  minKubeVersion: 1.11.0
  provider:
    name: Red Hat
  version: 0.10.1
status:
  certsLastUpdated: null
  certsRotateAt: null
  conditions:
  - lastTransitionTime: "2019-07-18T10:08:47Z"
    lastUpdateTime: "2019-07-18T10:08:47Z"
    message: requirements not yet checked
    phase: Pending
    reason: RequirementsUnknown
  - lastTransitionTime: "2019-07-18T10:08:47Z"
    lastUpdateTime: "2019-07-18T10:08:47Z"
    message: unable to adopt APIService
    phase: Failed
    reason: OwnerConflict
  lastTransitionTime: "2019-07-18T10:08:47Z"
  lastUpdateTime: "2019-07-18T10:08:47Z"
  message: unable to adopt APIService
  phase: Failed
  reason: OwnerConflict
  requirementStatus:
  - group: operators.coreos.com
    kind: ClusterServiceVersion
    message: CSV minKubeVersion (1.11.0) less than server version (v1.13.4+6569b4f)
    name: packageserver
    status: Present
    version: v1alpha1
  - group: apiregistration.k8s.io
    kind: APIService
    message: ""
    name: v1.packages.operators.coreos.com
    status: DeploymentFound
    version: v1
  - dependents:
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: cluster rule:{"verbs":["create","get"],"apiGroups":["authorization.k8s.io"],"resources":["subjectaccessreviews"]}
      status: Satisfied
      version: v1beta1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: cluster rule:{"verbs":["get","list","watch"],"apiGroups":[""],"resources":["configmaps"]}
      status: Satisfied
      version: v1beta1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: cluster rule:{"verbs":["get","list","watch"],"apiGroups":["operators.coreos.com"],"resources":["catalogsources"]}
      status: Satisfied
      version: v1beta1
    - group: rbac.authorization.k8s.io
      kind: PolicyRule
      message: cluster rule:{"verbs":["get","list"],"apiGroups":["packages.operators.coreos.com"],"resources":["packagemanifests"]}
      status: Satisfied
      version: v1beta1
    group: ""
    kind: ServiceAccount
    message: ""
    name: olm-operator-serviceaccount
    status: Present
    version: v1


mac:~ jianzhang$ oc get og -n openshift-operator-lifecycle-manager olm-operators -o yaml
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  annotations:
    olm.providedAPIs: PackageManifest.v1.packages.operators.coreos.com
  creationTimestamp: "2019-07-18T08:02:31Z"
  generation: 55
  name: olm-operators
  namespace: openshift-operator-lifecycle-manager
  resourceVersion: "83910"
  selfLink: /apis/operators.coreos.com/v1/namespaces/openshift-operator-lifecycle-manager/operatorgroups/olm-operators
  uid: 64e71542-a932-11e9-9b2c-02ff9ebba722
spec:
  targetNamespaces:
  - openshift-operator-lifecycle-manager
status:
  lastUpdated: "2019-07-18T08:04:06Z"
  namespaces:
  - openshift-operator-lifecycle-manager

mac:~ jianzhang$ oc get clusterrole packageserver.v0.9.0-tcln4 -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: "2019-07-18T08:04:29Z"
  labels:
    olm.owner: packageserver.v0.9.0
    olm.owner.kind: ClusterServiceVersion
    olm.owner.namespace: openshift-operator-lifecycle-manager
  name: packageserver.v0.9.0-tcln4
  resourceVersion: "4445"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/packageserver.v0.9.0-tcln4
  uid: ab2d5065-a932-11e9-92d4-02ef6ebf96d0
rules:
- apiGroups:
  - authorization.k8s.io
  resources:
  - subjectaccessreviews
  verbs:
  - create
  - get
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - operators.coreos.com
  resources:
  - catalogsources
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - packages.operators.coreos.com
  resources:
  - packagemanifests
  verbs:
  - get
  - list
Comment 1 Jian Zhang 2019-07-19 02:22:37 UTC 

*** This bug has been marked as a duplicate of bug 1723818 ***