Bug 2069307

Summary: oc mirror hangs when processing the Red Hat 4.10 catalog
Product: OpenShift Container Platform Reporter: Alex <aflom>
Component: ocAssignee: Alex <aflom>
oc sub component: oc-mirror QA Contact: RamaKasturi <knarra>
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: unspecified CC: anbhatta, aos-bugs, jpower, knarra, vdinh, yinzhou
Version: 4.11   
Target Milestone: ---   
Target Release: 4.11.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2065832 Environment:
Last Closed: 2022-08-10 11:02:19 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2065832    

Description Alex 2022-03-28 16:54:57 UTC 
+++ This bug was initially created as a clone of Bug #2065832 +++

Description of problem:

When running opm alpha diff with this Red Hat 4.10 catalog it hangs

opm alpha diff registry.redhat.io/redhat/redhat-operator-index:v4.10
...
WARN[0034] DEPRECATION NOTICE:
Sqlite-based catalogs and their related subcommands are deprecated. Support for
them will be removed in a future release. Please migrate your catalog workflows
to the new file-based catalog format. 
...

This does not occur with other catalogs (certified catalog) or with an include.yaml.


Version-Release number of selected component (if applicable):

Version: version.Version{OpmVersion:"v1.21.0", GitCommit:"9999f796", BuildDate:"2022-03-03T21:23:12Z", GoOs:"linux", GoArch:"amd64"}


How reproducible:
100%


Steps to Reproduce:
1.opm alpha diff registry.redhat.io/redhat/redhat-operator-index:v4.10


Actual results:

The application hangs with no output


Expected results:

The application to render a catalog successfully


Additional info:

GitHub issue https://github.com/operator-framework/operator-registry/issues/936

--- Additional comment from Anik on 2022-03-21 21:45:56 UTC ---

upstream PR: https://github.com/operator-framework/operator-registry/pull/937

--- Additional comment from Anik on 2022-03-24 16:10:33 UTC ---

Moving this over to oc since oc-mirror (which uses the `operator-registry/declcfg` package that has the opm alpha diff implementation) is what we're interested in fixing/testing.
Comment 1 RamaKasturi 2022-04-01 17:54:02 UTC 
Verified bug in the version below and i see that it no more hangs.

[knarra@knarra oc-mirror]$ ./oc-mirror version
Client Version: version.Info{Major:"0", Minor:"2", GitVersion:"v0.2.0-alpha.1", GitCommit:"c072b9dd6c9f85e07557ffb126db76803c712355", GitTreeState:"clean", BuildDate:"2022-04-01T08:19:56Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}

Below are the steps followed to verify the bug:
====================================================
1) Run command export RELEASE_IMAGE='registry.ci.openshift.org/ocp/release:4.11.0-0.nightly-2022-04-01-113941'
2) Run command oc image extract $(oc adm release info $RELEASE_IMAGE  -o=json | jq -r '.references.spec.tags[] | select(.name=="oc-mirror") | .from.name') --path=/usr/bin/oc-mirror:.
3) Once we have the oc-mirror image run 'chmod +x oc-mirror'
4) create a config.yaml with the contents below
[knarra@knarra oc-mirror]$ cat /tmp/config.yaml 
apiVersion: mirror.openshift.io/v1alpha2
kind: ImageSetConfiguration
mirror:
  operators:
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.10
5) Create a dir called mirror_411 using the command mkdir mirror_411
6) Now run the command ./oc-mirror --config /tmp/config.yaml file://mirror_411

Below is the ouput i see:
===========================
[knarra@knarra oc-mirror]$ ./oc-mirror --config /tmp/config.yaml file://mirror_411 --dry-run
INFO Found: mirror_411/oc-mirror-workspace/src/publish 
INFO Found: mirror_411/oc-mirror-workspace/src/v2 
INFO Found: mirror_411/oc-mirror-workspace/src/charts 
INFO Found: mirror_411/oc-mirror-workspace/src/release-signatures 
WARNING backend is not configured in /tmp/config.yaml, using stateless mode 
INFO No metadata detected, creating new workspace 
Sqlite-based catalogs and their related subcommands are deprecated. Support for
them will be removed in a future release. Please migrate your catalog workflows
to the new file-based catalog format. 
wrote mirroring manifests to mirror_411/oc-mirror-workspace/operators.1648834515/manifests-redhat-operator-index

To upload local images to a registry, run:

	oc adm catalog mirror file://redhat/redhat-operator-index:v4.10 REGISTRY/REPOSITORY

<dir>
  3scale-amp2/3scale-rhel7-operator
    blobs:
      registry.redhat.io/3scale-amp2/3scale-rhel7-operator sha256:d6d5f18a5b85c57a54ff88c6e57a17c9ea93b025aac94add7887415279ac4338 1.867KiB
      registry.redhat.io/3scale-amp2/3scale-rhel7-operator sha256:16398b11489027c045f66ce95dd0f2749ffee0c24b66e85a172af98ebbb1fd7f 1.887KiB
      registry.redhat.io/3scale-amp2/3scale-rhel7-operator sha256:0de52b06c009d537b1588c8ae3dcb26b9b62a3f8a8d0a48f060795c49244aceb 4.035KiB
      registry.redhat.io/3scale-amp2/3scale-rhel7-operator sha256:3149b20058934fb991ac36ed2a12c9c131aa456733f407af734691c28b865c96 4.045KiB
      registry.redhat.io/3scale-amp2/3scale-rhel7-operator sha256:ba7db50f56ddb2788c6f99b539cdee6bfa1ae7bf35c6ee3306eadeaf5f75484b 19.1MiB
      registry.redhat.io/3scale-amp2/3scale-rhel7-operator sha256:73256e4899544da239d8db93229e4453e80c05a88b9d488a03e3472400849ef3 20.88MiB
      registry.redhat.io/3scale-amp2/3scale-rhel7-operator sha256:868993c36870deb05702d8fbe02a66bd852e84db0cdb434c458f7f09a00a8df6 30.84MiB
      registry.redhat.io/3scale-amp2/3scale-rhel7-operator sha256:d65ad77b6fa504ab77dafd5dd8c0118fc8ce87abae0f122ff269e299fbac8377 33.39MiB
    manifests:
      sha256:06878a946d0011bcf45c35d6454e64054a7d7b54444edfc2af3769b95eb0c218
      sha256:69e1546805e3e51bf2cd90363125e556ef219eae11e50f42d56dff5a228cb650
      sha256:e2f786eb91752c1ad4c379cf5493cf547cc802dca611e92d3262fb261959f674
      sha256:06878a946d0011bcf45c35d6454e64054a7d7b54444edfc2af3769b95eb0c218 -> 3ecf3228
  3scale-amp2/3scale-rhel7-operator-metadata
    blobs:
      registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata sha256:9d5da3053918285b6e00670389bcf9e6d61b24f67c583952d83218d112837d4d 4.46KiB
      registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata sha256:34e2a3e6e75d14de6d6480406cbfb9e7635851a4c903423e0e8e6007a22f679f 50.6KiB
    manifests:
      sha256:bd95d5f4346063b9fb729da0efd4f6b223f4b9453bb8c2ee4d179ce8b7851f3e -> c0e5357f
  3scale-amp2/apicast-gateway-rhel8
    blobs:

 manifests:
      sha256:209c8f43ffb954e7a500e58823992ecd631c8f9c4c83ece10eaaed20f024139b
      sha256:8b606d5c487f4e016cfbf2160c6bee2276243e97da0cd16792458be656cc8835
      sha256:209c8f43ffb954e7a500e58823992ecd631c8f9c4c83ece10eaaed20f024139b -> 33b50287
  stats: shared=307 unique=3475 size=116.9GiB ratio=0.92

error: unable to retrieve source image registry.redhat.io/rhpam-7/rhpam-kogito-rhel8-operator manifest sha256:7aa5491e5d26b29cd0e271d24996e587a65b50f5a8d871bd004b53da65ee0038: error parsing HTTP 404 response body: invalid character 'N' looking for beginning of value: "Not found\n"
error: unable to retrieve source image registry.redhat.io/container-native-virtualization/kubevirt-velero-plugin-rhel8 manifest sha256:0c5c9c6a12f906a092d105842158a1eda8c21b9681b0bbd6f51a829d927bde60: unknown: Not Found
error: unable to retrieve source image registry.access.redhat.com/ubi8/openjdk-11 manifest sha256:03ba993c5d7459eb2d92b4c88ac3561633d05716760e62e5da5f04292ae287b8: error parsing HTTP 404 response body: invalid character 'N' looking for beginning of value: "Not found\n"
error: unable to retrieve source image registry.redhat.io/openshift4/ose-cloud-event-proxy manifest sha256:1ea23957c50885f5c26dce74ea6c88bb579dc684cbb29043ece6aece33073504: error parsing HTTP 404 response body: invalid character 'N' looking for beginning of value: "Not found\n"



Tried reproducing it with older oc-mirror version using the same steps as above and i see that command hangs:
============================================================================================================
[knarra@knarra Downloads]$ ./oc-mirror version
Client Version: version.Info{Major:"0", Minor:"1", GitVersion:"v0.1.0", GitCommit:"63d070c19e8f5b6c84b0b3339102c307ad1cac05", GitTreeState:"clean", BuildDate:"2022-03-31T17:05:48Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}


[knarra@knarra Downloads]$ ./oc-mirror --config /tmp/config.yaml file://mirror
WARNING backend is not configured in /tmp/config.yaml, using stateless mode 
INFO Creating directory: mirror/oc-mirror-workspace/src/publish 
INFO Creating directory: mirror/oc-mirror-workspace/src/v2 
INFO Creating directory: mirror/oc-mirror-workspace/src/charts 
WARN[0108] DEPRECATION NOTICE:
Sqlite-based catalogs and their related subcommands are deprecated. Support for
them will be removed in a future release. Please migrate your catalog workflows
to the new file-based catalog format. 

Based on the above moving bug to verified state.
Comment 3 errata-xmlrpc 2022-08-10 11:02:19 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069