Bug 1804729

Summary:	Upgrade between internal builds with different Y version is not possible when 4.2 build is done after 4.3
Product:	[Red Hat Storage] Red Hat OpenShift Container Storage	Reporter:	Petr Balogh <pbalogh>
Component:	build	Assignee:	Christina Meno <gmeno>
Status:	CLOSED ERRATA	QA Contact:	Petr Balogh <pbalogh>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	4.3	CC:	aschoen, bniver, madam, ocs-bugs, ratamir, sostapov
Target Milestone:	---	Keywords:	Automation, TestBlocker, Upgrades
Target Release:	OCS 4.3.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ocs-olm-operator:4.3-348.bd4d48c1.release_4.3	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-04-14 09:45:56 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Petr Balogh 2020-02-19 14:17:40 UTC

Description of problem (please be detailed as possible and provide log
snippests):
If we go with deployment of 4.2.2 internal build and want to upgrade to latest build of 4.3 which was done before 4.2.2 build we have the problem with upgrade cause of CSV versions:

4.2.2 build for example has version of CSV: ocs-operator.v0.0.319

But 4.3 build is:
ocs-operator.v0.0.313

where 313 is < than 319 but 
4.3 > 4.2

So this leads to that upgrade is not performing.

I already suggested other versioning of internal builds on our automation mtg where I asked to have 4.X in the internal build of CSV as well.

If possible something like 4.2.2-319 but not sure if it is possible to have - there or 4.2.2.319 than it should be < than 4.3.0.313

The jenkins job which failed:
https://ocs4-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/qe-deploy-ocs-cluster/4774//consoleFull

Version of all relevant components (if applicable):
4.2 and 4.3 have the same issue


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Block upgrade testing between 4.2 and 4.3


Is there any workaround available to the best of your knowledge?
No


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1


Can this issue reproducible?
Yes if the 4.2 build is done after 4.3.

Can this issue reproduce from the UI?
Haven't tried


If this is a regression, please provide more details to justify this:
No


Steps to Reproduce:
1. Install 4.2 internal build which was built after some 4.3 build
2. Change catalogSource to the new 4.3 build
3. Upgrade is not rolled


Actual results:
No upgrade happens


Expected results:
See upgrade in progress on CSV

Additional info:
Logs:
http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/pbalogh-upgrade/pbalogh-upgrade_20200218T213710/logs/failed_testcase_ocs_logs_1582065043/

Comment 3 Andrew Schoen 2020-02-27 22:52:42 UTC

Petr,

Could you please test with ocs-olm-operator:4.3-348.bd4d48c1.release_4.3? The versioning
for the dev builds will now follow the format of Major.minor.patch-ci.build, for example
the version for 4.3-348.bd4d48c1.release_4.3 is 4.3.0-ci.348.

Thanks,
Andrew

Comment 4 Petr Balogh 2020-02-28 09:55:12 UTC

Hey Andrew,

running verification of upgrade here:

https://ocs4-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/qe-deploy-ocs-cluster/5020/consoleFull

This is against the cluster which was upgraded from 4.2.1 RC build to 4.2.2 RC build and now upgrading to requested build.

Petr

Comment 5 Petr Balogh 2020-02-28 09:57:18 UTC

$ oc get csv -n openshift-storage
NAME                            DISPLAY                       VERSION        REPLACES                PHASE
lib-bucket-provisioner.v1.0.0   lib-bucket-provisioner        1.0.0                                  Succeeded
ocs-operator.v0.0.319           OpenShift Container Storage   0.0.319        ocs-operator.v0.0.286   Replacing
ocs-operator.v4.3.0-ci.348      OpenShift Container Storage   4.3.0-ci.348   ocs-operator.v0.0.319   Installing

So far looks OK ;)

Would be great to have also such build for 4.2 from which we can also verify upgrade to 4.3.

Comment 6 Petr Balogh 2020-02-28 12:56:08 UTC

Upgrade itself succeeded.

The job failed with some our verification which Jilju did.

I told him about the issue and he is working on fix to ocs-ci as seems like the data structure changed for the data where he was checking.

Comment 7 Andrew Schoen 2020-02-28 19:27:40 UTC

(In reply to Petr Balogh from comment #5)
> $ oc get csv -n openshift-storage
> NAME                            DISPLAY                       VERSION       
> REPLACES                PHASE
> lib-bucket-provisioner.v1.0.0   lib-bucket-provisioner        1.0.0         
> Succeeded
> ocs-operator.v0.0.319           OpenShift Container Storage   0.0.319       
> ocs-operator.v0.0.286   Replacing
> ocs-operator.v4.3.0-ci.348      OpenShift Container Storage   4.3.0-ci.348  
> ocs-operator.v0.0.319   Installing
> 
> So far looks OK ;)
> 
> Would be great to have also such build for 4.2 from which we can also verify
> upgrade to 4.3.

You can use this 4.2 build to test with: ocs-olm-operator:4.2-351.88017e99.release_4.2

The version number for that build should be 4.2.2-ci.351

Thanks,
Andrew

Comment 8 Petr Balogh 2020-03-02 15:00:31 UTC

Running verification here:
https://ocs4-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/qe-deploy-ocs-cluster/5096/console

Comment 9 Petr Balogh 2020-03-03 07:36:39 UTC

Upgrade finished. The job itself failed on unrelated check which Jilju is already fixing in one of the PR.

Marking as verified.

Comment 14 errata-xmlrpc 2020-04-14 09:45:56 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1437