Bug 2112643

Summary: Nightly build: v4.11.0-578: index format was changed in 4.11 to file-based instead of sqlite-based
Product: Container Native Virtualization (CNV) Reporter: Eli <ebattat>
Component: InstallationAssignee: Oren Cohen <ocohen>
Status: CLOSED ERRATA QA Contact: SATHEESARAN <sasundar>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.11.0CC: cnv-qe-bugs, dbasunag, kmajcher, ocohen, sasundar, stirabos
Target Milestone: ---   
Target Release: 4.11.1   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-12-01 21:12:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Eli 2022-07-31 09:12:58 UTC
Description of problem:
CNV Nightly build installation failed due to index format was changed in 4.11 to file-based instead of sqlite-based

$ oc get InstallPlan -n openshift-cnv 
NAME            CSV                                           APPROVAL    APPROVED
install-j88pg   kubevirt-hyperconverged-operator.4.11.0-578   Automatic   true


$ oc patch InstallPlan -n openshift-cnv $(oc get InstallPlan -n openshift-cnv -ojsonpath={.items[0].metadata.name}) -p '{"spec":{"approved":true}}' --type merge


Apply patch didnt work.

According to Oren Cohen investigation:

"it's because the index format was changed in 4.11 to file-based instead of sqlite-based."

Comment 1 Oren Cohen 2022-07-31 18:53:48 UTC
'opm index' command is no longer working for file-based catalogs, which v4.11 index image has been transitioned to.
we need to update the nightly channel workflow to reflect the changes from sqlite-based to file-based catalogs.

https://github.com/redhat-openshift-ecosystem/community-operators-prod/discussions/512
https://olm.operatorframework.io/docs/reference/file-based-catalogs/

Comment 2 Oren Cohen 2022-08-15 17:00:28 UTC
An MR that migrates the cnv nightly channel workflow to use File-Based catalog (FBC) format, rather than the old SQLite-based catalog, has been merged.
cnv 4.11 and 4.12 nightly catalogs have been migrated to FBC already.

To verify:
1. Get the latest opm tool at: https://github.com/operator-framework/operator-registry/releases
2. use it to verify the integrity of the index image that is using FBC. for example: "opm alpha list bundles quay.io/openshift-cnv/nightly-catalog:4.12 kubevirt-hyperconverged"
This command should return a table with the production contents, as well as the nightly bundle in "nightly-4.12" channel.
When running this command against an sqlite-based index, a warning is shown on the console. for example:
$ opm alpha list bundles quay.io/openshift-cnv/nightly-catalog:4.10 kubevirt-hyperconverged
WARN[0021] DEPRECATION NOTICE:
Sqlite-based catalogs and their related subcommands are deprecated. Support for
them will be removed in a future release. Please migrate your catalog workflows
to the new file-based catalog format. 


3. Verify that it is actually works by using it in a cluster, according to instructions on: https://access.redhat.com/articles/6070641
4. Verify that a cluster that is already subscribed to the nightly channel can upgrade its CNV nightly version seamlessly when a new build is available.

Comment 3 Krzysztof Majcher 2022-10-04 12:56:33 UTC
Hi, this looks a bit old, when we plan to verify it?

Comment 4 Oren Cohen 2022-10-04 13:14:14 UTC
This is just to verify that the index images we are publishing on the nightly channel[1] for CNV 4.11 and CNV 4.12 are using the new file-based catalog[2], rather than the deprecated sqlite-based catalogs.

it can be verified by examining the contents of the index images, i.e.

$ podman run --entrypoint='["cat", "/configs/nightly_catalog.json"]' quay.io/openshift-cnv/nightly-catalog:4.11
$ podman run --entrypoint='["cat", "/configs/nightly_catalog.json"]' quay.io/openshift-cnv/nightly-catalog:4.12

these commands will print the contents of the file-based catalogs for 4.11 and 4.12 (in 4.10 and below we're still using sqlite-based catalogs).

Also, worth to verify we can actually install CNV 4.11 and 4.12 from these index images, according the procedure in [1]. It should install the last version in each minor version that passed QE smoke tests.


----
[1] https://access.redhat.com/articles/6070641
[2] https://olm.operatorframework.io/docs/reference/file-based-catalogs/

Comment 5 SATHEESARAN 2022-10-04 18:07:49 UTC
(In reply to Krzysztof Majcher from comment #3)
> Hi, this looks a bit old, when we plan to verify it?

Yes, this will be verified as part of the current active sprint.
I am working on it now

Comment 6 SATHEESARAN 2022-10-05 10:07:17 UTC
Thank you Oren for all the required information that you provided in comment2 and comment4.

I could download the latest opm tool to 'list' the contents in nightly-build which is now using file-based catalog
The following command worked good and listed the package, channel, bundle information.
# opm alpha list bundles quay.io/openshift-cnv/nightly-catalog:4.12 kubevirt-hyperconverged
The same when tried with 4.10 has provided the 'depreciation' warning of sqlite db

File based catalog contents are accessed and verified for 4.11 and 4.12 nightly-build channel using the command:
$ podman run --entrypoint='["cat", "/configs/nightly_catalog.json"]' quay.io/openshift-cnv/nightly-catalog:4.11
$ podman run --entrypoint='["cat", "/configs/nightly_catalog.json"]' quay.io/openshift-cnv/nightly-catalog:4.12

Pending thing for verification is:
1. Install the latest nightly from 4.11
2. Install the latest nightly from 4.12
3. Updating the nightly build.

Above testing is in progress and completed by this week

Comment 7 SATHEESARAN 2022-10-10 18:15:41 UTC
With information from comment2 and comment4, verified that CNV 4.11 and 4.12 nightly builds are using file based catalog.
Also verified with installing and upgrading CNV 4.11 nightly builds on OCP 4.11.

Installation and update testing of 4.12 builds is blocked as the CNV 4.12 stable interim builds are not yet available.
And the same will be taken care with the separate JIRA story 

So, as the installation and upgrade of 4.11 nightly builds is working fine and also with the verification that CNV 4.12 nightly
build is using file based catalog, verifying this bug

Comment 9 SATHEESARAN 2022-11-07 10:12:43 UTC
(In reply to SATHEESARAN from comment #7)
> Installation and update testing of 4.12 builds is blocked as the CNV 4.12
> stable interim builds are not yet available.
> And the same will be taken care with the separate JIRA story 

Verified that 4.12 nightly-build is also installable and upgradable.

Comment 16 errata-xmlrpc 2022-12-01 21:12:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 4.11.1 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8750