Bug 1982991

Summary:	WMCO auto upgrade from v2.0.2 to v3.0.0 failed
Product:	OpenShift Container Platform	Reporter:	gaoshang <sgao>
Component:	Windows Containers	Assignee:	Mansi Kulkarni <mankulka>
Status:	CLOSED ERRATA	QA Contact:	gaoshang <sgao>
Severity:	high	Docs Contact:
Priority:	urgent
Version:	4.9	CC:	aos-bugs, rrasouli, team-winc, yli2
Target Milestone:	---
Target Release:	4.9.0
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1983153 (view as bug list)		Environment:
Last Closed:	2021-10-28 17:41:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1983153

Description gaoshang 2021-07-16 08:23:52 UTC

Description of problem:
When upgrading from v2.0.2 to v3.0.0, operator stuck in pending status
$ oc get csv -n openshift-windows-machine-config-operator
NAME                                     DISPLAY                            VERSION    REPLACES                                 PHASE
elasticsearch-operator.5.0.6-35          OpenShift Elasticsearch Operator   5.0.6-35                                            Succeeded
windows-machine-config-operator.v2.0.2   Windows Machine Config Operator    2.0.2      windows-machine-config-operator.v2.0.1   Replacing
windows-machine-config-operator.v3.0.0   Windows Machine Config Operator    3.0.0      windows-machine-config-operator.v2.0.2   Pending

Version-Release number of selected component (if applicable):
OCP upgrade from 4.7.0-0.nightly-2021-07-15-180731 to 4.8.0-0.nightly-2021-07-15-150855

How reproducible:
always

Steps to Reproduce:
1, Install WMCO 2.0.2 on OCP 4.7 with 4.7 stage catalogsource, set "update approve" as automatic
e.g.
$ oc get catalogsource wmco -oyaml -n openshift-marketplace | grep " image"
  image: brew.registry.redhat.io/rh-osbs/iib-pub-pending:v4.7

2, Upgrade OCP 4.7 to 4.8, refer to https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-21588

$ oc adm release info registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-07-15-150855 | grep 'Pull From:'
Pull From: registry.ci.openshift.org/ocp/release@sha256:c3ec3e58ec67160900a5fb8d8ee51b81567861e7bba3f809e45d8be81fe8673b

$ oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:c3ec3e58ec67160900a5fb8d8ee51b81567861e7bba3f809e45d8be81fe8673b --allow-explicit-upgrade --force

3, Change WMCO catalogsource to 4.8 stage
e.g.
$ oc get catalogsource wmco -oyaml -n openshift-marketplace | grep " image"
  image: brew.registry.redhat.io/rh-osbs/iib-pub-pending:v4.8

4, Wait and check WMCO auto upgrade

Actual results:
Operator stuck in pending status

Expected results:
Operator should upgrade to 3.0.0

Additional info:
Describe csv, found "RequirementsNotMet, one or more requirements couldn't be found"

$ oc describe csv windows-machine-config-operator.v3.0.0
...
Status:
  Cleanup:
  Conditions:
    Last Transition Time:  2021-07-15T14:30:21Z
    Last Update Time:      2021-07-15T14:30:21Z
    Message:               requirements not yet checked
    Phase:                 Pending
    Reason:                RequirementsUnknown
    Last Transition Time:  2021-07-15T14:30:21Z
    Last Update Time:      2021-07-15T14:30:21Z
    Message:               one or more requirements couldn't be found
    Phase:                 Pending
    Reason:                RequirementsNotMet
  Last Transition Time:    2021-07-15T14:30:21Z
  Last Update Time:        2021-07-15T14:30:21Z
  Message:                 one or more requirements couldn't be found
  Phase:                   Pending
  Reason:                  RequirementsNotMet
  Requirement Status:
    Group:    operators.coreos.com
    Kind:     ClusterServiceVersion
    Message:  CSV minKubeVersion (1.21.0) less than server version (v1.21.1+f36aa36)
    Name:     windows-machine-config-operator.v3.0.0
    Status:   Present
    Version:  v1alpha1
    Group:    
    Kind:     ServiceAccount
    Message:  Service account is owned by another ClusterServiceVersion
    Name:     windows-machine-config-operator
    Status:   PresentNotSatisfied
    Version:  v1
Events:
  Type    Reason               Age                From                        Message
  ----    ------               ----               ----                        -------
  Normal  RequirementsUnknown  33m (x2 over 33m)  operator-lifecycle-manager  requirements not yet checked
  Normal  RequirementsNotMet   33m (x2 over 33m)  operator-lifecycle-manager  one or more requirements couldn't be found

Comment 1 Aravindh Puthiyaparambil 2021-07-16 16:13:26 UTC

This is a blocker for the WMCO 3.0.0 release

Comment 2 gaoshang 2021-07-21 07:10:38 UTC

Upgrade to the new WMCO 3.0.0, operator still leaves in pending status, changed back bug status.

Comment 3 Aravindh Puthiyaparambil 2021-07-23 16:14:27 UTC

Marking as VERIFIED to allow the release-4.7 PR to merge. Will move this back to ON_QA once that PR merges.

Comment 4 Aravindh Puthiyaparambil 2021-07-27 16:26:44 UTC

Marking as VERIFIED to allow the release-4.8 PR to merge. Will move this back to ON_QA once that PR merges.

Comment 5 Ronnie Rasouli 2021-08-01 10:04:05 UTC

rrasouli@rrasouli-mac openshift-tests-private % oc get csv
NAME                                     DISPLAY                            VERSION   REPLACES                                 PHASE
elasticsearch-operator.5.0.7-9           OpenShift Elasticsearch Operator   5.0.7-9                                            Succeeded
windows-machine-config-operator.v3.0.0   Windows Machine Config Operator    3.0.0     windows-machine-config-operator.v2.0.3   Succeeded
rrasouli@rrasouli-mac openshift-tests-private % oc get catalogsource wmco -oyaml -n openshift-marketplace | grep " image"
  image: brew.registry.redhat.io/rh-osbs/iib-pub-pending:v4.8

Comment 7 Yu Li 2021-08-05 02:48:15 UTC

As discussed with Mansi and Aravindh, this bug is mainly used to trace code merge which was be fixed on master branch. The issue was happening on upgrade from WMCO 2.0.2 to WMCO 3.0.0. WMCO 2.0.x is running on OCP4.7. WMCO 3.0.x is running on OCP4.8. OCP4.9 is not in the picture right now. When have WMCO 4.y.z (for OCP4.9), QE will run upgrade test from WMCO 3.y.z to 4.y.z, then we will see whether we have new issues and open new bug based on that only.

Comment 12 errata-xmlrpc 2021-10-28 17:41:17 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Windows Container Support for Red Hat OpenShift 4.0.0 product release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3702