Bug 2052007

*** Bug 2052006 has been marked as a duplicate of this bug. ***

Was this working in 4.9? If yes, how was it performed?

In that bug I was verifying the upgrade between different Z version of internal builds is working. So 4.9.0 to 4.9.1 . 
This actually work but upgrade for the same Z version like build 4.10.0-160 4.10.0-164 doesn't work as the CSV is still the same version 4.10.0 so OLM will not see any upgrade.
I know Umanga recommended some other steps how to invoke upgrade but when I tried it didn't work. But cannot find those steps now.

Umanga, please provide some steps if possible.

I am closing this BZ because there is nothing we can do through code/build here to provide this functionality and it never existed in the past too.

This worked before we changed the CSV name in 4.9 to this format to not contain the -PATCH_VERSION in the CSV name for internal builds.

This has stopped work in 4.9, hence Bipin opened the other BZ.

The main problem is now with DR setup - this is really time consuming to spin such environment up. In the case we need to update to new build for same X.Y.Z version but next build basically we need to invest a lot of time to do new setup from scratch which is really concerning here.

Any change in the versioning would be a big concern for the upgrades. The new bundle layout with the dependencies between the bundles was one of the reasons why we dropped the -PATCH_VERSION (well, -BUILD_NUMBER) in the CSV version.

Previuosly, we have only used the -BUILD_NUMBER in the mirrored builds but not the actual builds that we pushed live. There was a change to minimize the amount of changes between the mirrored builds and the released builds -- now, there is pretty much no change between the builds but the registry change to quay.io for the mirrored builds and that is done by IIB for us.

If we wanted to implement this, we should do it for both mirrored as well as released builds and that has never actually worked before.

I am reopening the bug to look at options on how we could support upgrades within the internal builds.  

1) We had this working in the past until 4.9
2) As mentioned by Petr in comment#8, having the ability to support internal build upgrades would mean QE doesn't have to work on creating setups, especially time-consuming ones like DR.

The current system was introduced in Sep 2021[1]

> This was done in order to fully align the CI and RC/production builds (up to
> the mirroring bits). We have also had other technical reasons to do this like
> the move to CPaaS (the CPaaS-enabled 4.8.3 builds will also have this change
> in them) and odf-operator now handling the dependencies on other CSV packages
> in code and storing them in the config map. @Nitin Goyal can elaborate a bit
> more on this.

And I didn't find it optimal neither[2]:

> Why is that a good thing? I'm asking since neither OCP itself nor OLM
> operator are versioned like this (there are clearly tagged nightly builds,
> RCs ...).

Does it mean we are reopening this discussion? I'm not sure why we can't try to
follow the same approach as LSO operator does.

Eg. to get a version of LSO operator, one can just asks for it's version directly:

```
$ oc get csv -n openshift-local-storage -o=jsonpath="{.items[0].spec.version}" 2>/dev/null
```

But doing the same for ODF operator doesn't provide the full info, one needs to inspect
it's labels:

```
$ oc get csv -n openshift-storage -o=jsonpath="{.items[0].spec.version}"
$ oc get csv -n openshift-storage -o json | jq '.items[].metadata.labels["full_version"]' | grep -v null | sort | uniq
```

Which obvious confuses OLM which believes that 2 different ci builds of ODF are the
same.

And if I read comment 9 from Boris right, we have some releng problem with GA'd builds
as well, which prevents us to have a reasonable versioning in CI builds?

[1] https://post-office.corp.redhat.com/archives/ocs-qe/2021-September/msg00342.html
[2] https://post-office.corp.redhat.com/archives/ocs-qe/2021-September/msg00358.html

QE acceptance
-------------

The following command reports full ODF operator version:

```
$ oc get csv -n openshift-storage -o=jsonpath="{.items[0].spec.version}"
```

No work was done in 4.11 for this BZ, moving it out. We still don't know how to resolve it without breaking the existing things.

@branto Can you please provide us custom builds for verifciation?

I have just merged the following MR:

https://gitlab.cee.redhat.com/ceph/rhodf/-/merge_requests/878

We should be providing upgradeable 4.99/Master/Main builds from now on.

I am also working on related changes for the CPaaS build pipeline here:

https://gitlab.cee.redhat.com/ceph/rhcs-jenkins-jobs/-/merge_requests/1115

This introduces the notion of RC builds to the CPaaS build pipeline. All the non-RC builds (default when creating a new y-stream release and for the 4.99/Master/Main builds) will be upgradeable once we merge this. This includes the current releases like 4.12.0 (it will also require a similar MR like the 4.99/Master/Main builds). All the other supported releases (4.11 and less) are already in maintenance mode so they will be tagged as RC builds and will only be upgradeable from the previous z-stream (or y-stream) release like before.

All the MRs are merged now. All the new 4.12.0 builds (until we reach RC phase) will also be upgradeable.