Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1751645

Summary:	Marketplace operator upgrade failed during upgrade from 4.1.15 to 4.2
Product:	OpenShift Container Platform	Reporter:	Yanping Zhang <yanpzhan>
Component:	OLM	Assignee:	Evan Cordell <ecordell>
OLM sub component:	OperatorHub	QA Contact:	Fan Jia <jfan>
Status:	CLOSED DUPLICATE	Docs Contact:
Severity:	high
Priority:	high	CC:	krizza
Version:	4.2.0	Keywords:	Regression, TestBlocker
Target Milestone:	---
Target Release:	4.2.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-09-12 12:58:14 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Yanping Zhang 2019-09-12 10:12:48 UTC

Description of problem:
Marketplace operator pod is in CrashLoopBackOff status during upgrade from 4.1.15 to 4.2, which cause upgrade failing.

Version-Release number of selected component (if applicable):
Before upgrade:
oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.1.15    True        False         82m     Cluster version is 4.1.15
After:
Updating to release image registry.svc.ci.openshift.org/ocp/release:4.2.0-0.nightly-2019-09-12-034447
$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.1.15    True        True          66m     Unable to apply 4.2.0-0.nightly-2019-09-12-034447: the update could not be applied

How reproducible:


Steps to Reproduce:
1. Upgrade 4.1.15 to 4.2 with above version.
$ oc adm upgrade --force=true --to-image=****/ocp/release:4.2.0-0.nightly-2019-09-12-034447
2. Marketplace operator pod can not start:
$ oc get pod -n openshift-marketplace
NAME                                                              READY   STATUS             RESTARTS   AGE
certified-operators-5f44d9bf57-mwqdp                              1/1     Running            0          3h48m
community-operators-7f5f668658-fqv46                              1/1     Running            0          3h48m
elasticsearch-6bb585cd97-mq74z                                    1/1     Running            0          3h18m
installed-certified-openshift-operators-54456696c-78qsh           1/1     Running            0          80m
installed-community-default-6c799dc469-5wtp9                      1/1     Running            0          3h1m
installed-community-openshift-operators-5c5fc8d9fd-2cl79          0/1     Evicted            0          68m
installed-community-openshift-operators-5c5fc8d9fd-4xdnd          0/1     Evicted            0          68m
installed-community-openshift-operators-5c5fc8d9fd-5cm4c          0/1     Evicted            0          68m
installed-community-openshift-operators-5c5fc8d9fd-bz9rp          0/1     Evicted            0          68m
installed-community-openshift-operators-5c5fc8d9fd-f26bf          0/1     Evicted            0          68m
installed-community-openshift-operators-5c5fc8d9fd-mvwwr          0/1     Evicted            0          68m
installed-community-openshift-operators-5c5fc8d9fd-pbsjv          0/1     Evicted            0          80m
installed-community-openshift-operators-5c5fc8d9fd-pcssp          1/1     Running            0          68m
installed-community-openshift-operators-5c5fc8d9fd-rmt5j          0/1     Evicted            0          68m
installed-community-openshift-operators-5c5fc8d9fd-tkthr          0/1     Evicted            0          68m
installed-redhat-openshift-ansible-service-broker-675c74f49sfqm   1/1     Running            0          68m
installed-redhat-openshift-ansible-service-broker-675c74f4jfw6g   0/1     Evicted            0          112m
installed-redhat-openshift-logging-575d9c8b58-mdq2w               1/1     Running            0          3h16m
installed-redhat-openshift-operators-79df77786c-hp9vr             1/1     Running            0          3h1m
installed-redhat-openshift-template-service-broker-cd55746bxczc   1/1     Running            0          67m
installed-redhat-openshift-template-service-broker-cd55746gzpf7   0/1     Evicted            0          110m
marketplace-operator-6dbcf87ffb-47gx7                             0/1     CrashLoopBackOff   4          2m2s
multicsc-6d87c5554f-vjww8                                         1/1     Running            0          3h2m
redhat-operators-548b548495-p9z45                                 1/1     Running            0          3h48m
simplecsc-6f6bdc6bd8-fjw2t                                        1/1     Running            0          68m
simplecsc-6f6bdc6bd8-gwm2c                                        0/1     Evicted            0          80m
testkey-6466c567df-jmdnh                                          1/1     Running            0          3h3m

===========================================
$ oc get pod -n openshift-marketplace |grep marketplace
marketplace-operator-6dbcf87ffb-8k7bn                             0/1     CrashLoopBackOff   14         48m
$ oc logs marketplace-operator-6dbcf87ffb-47gx7 -n openshift-marketplace
time="2019-09-12T09:56:05Z" level=info msg="[metrics] Registering marketplace metrics"
time="2019-09-12T09:56:05Z" level=info msg="[metrics] Creating marketplace metrics RoundTripperFunc"
time="2019-09-12T09:56:05Z" level=info msg="[metrics] Serving marketplace metrics"
time="2019-09-12T09:56:05Z" level=info msg="Go Version: go1.11.13"
time="2019-09-12T09:56:05Z" level=info msg="Go OS/Arch: linux/amd64"
time="2019-09-12T09:56:05Z" level=info msg="operator-sdk Version: v0.8.0"
time="2019-09-12T09:56:05Z" level=info msg="Config API is available"
time="2019-09-12T09:56:05Z" level=info msg="Registering Components."
time="2019-09-12T09:56:06Z" level=info msg="Waiting to become leader."
time="2019-09-12T09:56:06Z" level=info msg="Elected leader."
time="2019-09-12T09:56:06Z" level=info msg="Starting the Cmd."
panic: runtime error: index out of range

goroutine 1 [running]:
github.com/operator-framework/operator-marketplace/pkg/migrator.ExtractCsName(0xc000c8df80, 0xd, 0x14e0283, 0xe)
	/go/src/github.com/operator-framework/operator-marketplace/pkg/migrator/migrator.go:255 +0x125
github.com/operator-framework/operator-marketplace/pkg/migrator.findCatalogSource(0xc000570a80, 0x1694720, 0xc000c6c1a0, 0x13, 0xc00009d3b8, 0xc000518401)
	/go/src/github.com/operator-framework/operator-marketplace/pkg/migrator/migrator.go:195 +0x88
github.com/operator-framework/operator-marketplace/pkg/migrator.(*migrator).updateSubscriptions(0xc0009a9f50, 0xc000ecdd18, 0x40b6cf, 0xc0005183f0, 0x70, 0x70, 0x14754e0)
	/go/src/github.com/operator-framework/operator-marketplace/pkg/migrator/migrator.go:100 +0x2e5
github.com/operator-framework/operator-marketplace/pkg/migrator.(*migrator).doMigrate(0xc0009a9f50, 0xc000042040, 0x15, 0x40bf58, 0x70, 0x14754e0)
	/go/src/github.com/operator-framework/operator-marketplace/pkg/migrator/migrator.go:58 +0x40
github.com/operator-framework/operator-marketplace/pkg/migrator.(*migrator).Migrate(0xc0009a9f50, 0xc000042040, 0x15, 0xc000455e00, 0x1694720)
	/go/src/github.com/operator-framework/operator-marketplace/pkg/migrator/migrator.go:68 +0x5d
main.main()
	/go/src/github.com/operator-framework/operator-marketplace/cmd/manager/main.go:157 +0xaae

====================
$ oc get co marketplace -o yaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: "2019-09-12T06:08:24Z"
  generation: 1
  name: marketplace
  resourceVersion: "113756"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/marketplace
  uid: bace361c-d523-11e9-8b9d-02d6118630c8
spec: {}
status:
  conditions:
  - lastTransitionTime: "2019-09-12T09:04:52Z"
    message: 'Performing migration logic to progress towards release version: 4.2.0-0.nightly-2019-09-12-034447'
    reason: Upgrading
    status: "True"
    type: Progressing
  - lastTransitionTime: "2019-09-12T09:04:31Z"
    message: Determining status
    reason: Upgrading
    status: "False"
    type: Available
  - lastTransitionTime: "2019-09-12T06:08:24Z"
    message: Determining status
    reason: Upgrading
    status: "False"
    type: Degraded
  - lastTransitionTime: "2019-09-12T09:04:52Z"
    message: Determining status
    reason: Upgrading
    status: "False"
    type: Upgradeable
  extension: null
  relatedObjects:
  - group: ""
    name: openshift-marketplace
    resource: namespaces
  - group: operators.coreos.com
    name: ""
    namespace: openshift-marketplace
    resource: OperatorSource
  - group: operators.coreos.com
    name: ""
    namespace: openshift-marketplace
    resource: CatalogSourceConfig
  - group: operators.coreos.com
    name: ""
    namespace: openshift-marketplace
    resource: CatalogSource
  versions:
  - name: operator
    version: 4.1.15


3. The rest operators(dns, machine-config, marketplace, network) could not upgrade since marketplace stuck in failure.
$ oc get clusteroperator
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.2.0-0.nightly-2019-09-12-034447   True        False         False      3h36m
cloud-credential                           4.2.0-0.nightly-2019-09-12-034447   True        False         False      3h54m
cluster-autoscaler                         4.2.0-0.nightly-2019-09-12-034447   True        False         False      3h54m
console                                    4.2.0-0.nightly-2019-09-12-034447   True        False         False      50m
dns                                        4.1.15                              True        False         False      3h53m
image-registry                             4.2.0-0.nightly-2019-09-12-034447   True        False         False      3h45m
ingress                                    4.2.0-0.nightly-2019-09-12-034447   True        False         False      3h45m
insights                                   4.2.0-0.nightly-2019-09-12-034447   True        False         False      53m
kube-apiserver                             4.2.0-0.nightly-2019-09-12-034447   True        False         False      3h51m
kube-controller-manager                    4.2.0-0.nightly-2019-09-12-034447   True        False         False      3h52m
kube-scheduler                             4.2.0-0.nightly-2019-09-12-034447   True        False         False      3h51m
machine-api                                4.2.0-0.nightly-2019-09-12-034447   True        False         False      3h54m
machine-config                             4.1.15                              True        False         False      3h53m
marketplace                                4.1.15                              False       True          False      53m
monitoring                                 4.2.0-0.nightly-2019-09-12-034447   True        False         False      51m
network                                    4.1.15                              True        False         False      3h54m
node-tuning                                4.2.0-0.nightly-2019-09-12-034447   True        False         False      52m
openshift-apiserver                        4.2.0-0.nightly-2019-09-12-034447   True        False         False      3h50m
openshift-controller-manager               4.2.0-0.nightly-2019-09-12-034447   True        False         False      3h53m
openshift-samples                          4.2.0-0.nightly-2019-09-12-034447   True        False         False      53m
operator-lifecycle-manager                 4.2.0-0.nightly-2019-09-12-034447   True        False         False      3h53m
operator-lifecycle-manager-catalog         4.2.0-0.nightly-2019-09-12-034447   True        False         False      3h53m
operator-lifecycle-manager-packageserver   4.2.0-0.nightly-2019-09-12-034447   True        False         False      50m
service-ca                                 4.2.0-0.nightly-2019-09-12-034447   True        False         False      3h54m
service-catalog-apiserver                  4.2.0-0.nightly-2019-09-12-034447   True        False         False      114m
service-catalog-controller-manager         4.2.0-0.nightly-2019-09-12-034447   True        False         False      114m
storage                                    4.2.0-0.nightly-2019-09-12-034447   True        False         False      53m

4. 
Actual results:


Expected results:


Additional info:

Comment 1 Kevin Rizza 2019-09-12 12:58:14 UTC

This bug is a duplicate of the issue described in https://bugzilla.redhat.com/show_bug.cgi?id=1749643

Closing as duplicate.

As a side note, this should not be marked as a test blocker as the mitigation in this environment should be simply to delete any user defined catalogsourceconfigs before running the upgrade.

*** This bug has been marked as a duplicate of bug 1749643 ***