Bug 1841885

Summary: oc adm catalog mirror command attempts to pull from registry.redhat.io when using --from-dir option
Product: OpenShift Container Platform Reporter: Dean Lystra <dlystra>
Component: OLMAssignee: Evan Cordell <ecordell>
OLM sub component: OLM QA Contact: Bruno Andrade <bandrade>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: aos-bugs, dageoffr, erich, jcall, jhultz, jiazha, jokerman, jrickard, kmorgan, kodonnel, krizza, ljenkin, marobrie, maszulik, mfojtik, nhale
Version: 4.4   
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-24 15:12:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1917550    

Description Dean Lystra 2020-05-29 17:57:40 UTC 
Description of problem:

After using the oc adm catalog build command to build a catalog image using the --to-file option, I transferred a tar of the the directory to an offline registry. When using the oc adm catalog mirror command with the --from-dir option, the output indicated that pulls are still being attempted from registry.redhat.io instead of from the directory.

Version-Release number of selected component (if applicable):

Client Version: 4.4.5

How reproducible:

Always

Steps to Reproduce:
1. Run oc adm catalog build with --to-file option

#oc adm catalog build --appregistry-org redhat-operators --from=registry.redhat.io/openshift4/ose-operator-registry:v4.4 --to=file://offline/redhat-operators:4.4

...
INFO[0018] directory                                     dir=/tmp/cache-947871507/manifests-339782904 file=4.4 load=package
Uploading ... 22.92MB/s
Uploading 1.598kB ...
Uploading 3.501MB ...
Uploading 76.28MB ...
Uploading 8.229MB ...
Uploading 86.75MB ...
Pushed sha256:1aa50fed30c4015e3802bf841afbeed618fb357669000ff4b873982264770241 to file://offline/redhat-operators:4.4

2. Tar the image directory and scp to offline registry

3. Extract and run oc adm catalog mirror command against the image directory using --from-dir option

#oc adm catalog mirror --from-dir=operators file://offline/redhat-operators:4.4 registry.cluster.dean.io

I0529 17:43:26.689924   12783 mirror.go:231] wrote database to /tmp/003437671/bundles.db                             

error: unable to connect to registry.redhat.io/rhacm1-tech-preview/openshift-hive-rhel7-operator: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W0529 17:43:41.697565   12783 mirror.go:194] error mirroring image: an error occurred during planning                

error: unable to connect to registry.redhat.io/openshift-service-mesh/proxy-init-rhel7: Get https://registry.redhat.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)       
W0529 17:43:56.697942   12783 mirror.go:194] error mirroring image: an error occurred during planning                

error: unable to connect to registry.redhat.io/3scale-amp2/3scale-rhel7-operator: Get https://registry.redhat.io/v2/:
net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)             
W0529 17:44:11.698327   12783 mirror.go:194] error mirroring image: an error occurred during planning     

Actual results:

System attempts Gets from registry.redhat.io even when --from-dir option is used.

Expected results:

Successful catalog image mirroring from a directory

Additional info:
Comment 10 Jian Zhang 2020-10-27 03:18:54 UTC 
Hi Dean,

> 2. Tar the image directory and scp to offline registry

Sorry, I'm not familiar with it. I tried it but failed. Could you help give more info about this step? Thanks! 

[root@preserve-olm-env data]# oc adm catalog build --appregistry-org redhat-operators --from=registry.redhat.io/openshift4/ose-operator-registry:v4.4 --to=file://offline/redhat-operators:4.4
...
INFO[0018] directory                                     dir=/tmp/cache-975303688/manifests-616634257 file=web-terminal-_656cyb8 load=package
INFO[0018] directory                                     dir=/tmp/cache-975303688/manifests-616634257 file=1.0.1 load=package
Uploading ... 26.3MB/s
Uploading 3.509MB ...
Uploading 76.25MB ...
Uploading 9.59MB ...
Uploading 1.72kB ...
Uploading 86.76MB ...
Pushed sha256:8e0dcf29be1ba5dcb73aa2c71ce56c850b5eb5fed8e59d320c48a7cffbac09b0 to file://offline/redhat-operators:4.4

[root@preserve-olm-env data]# docker save -o rh.tar file://offline/redhat-operators:4.4
Error response from daemon: invalid reference format


[root@preserve-olm-env data]# tar -czvf 44.tar /tmp/cache-975303688/manifests-616634257
tar: Removing leading `/' from member names
/tmp/cache-975303688/manifests-616634257/
/tmp/cache-975303688/manifests-616634257/amq-streams/
...

[root@preserve-olm-env data]# docker load < 44.tar 
open /data/docker/tmp/docker-import-823274727/tmp/json: no such file or directory

[root@preserve-olm-env data]# oc adm catalog mirror --from-dir=operators file://offline/redhat-operators:4.4 registry.cluster.dean.io
using database path mapping: /:/tmp/309457861
errors during mirroring. the full contents of the catalog may not have been mirrored: unable to read image file://offline/redhat-operators:4.4: unknown blob
wrote mirroring manifests to redhat-operators-manifests
Comment 11 Evan Cordell 2020-10-27 13:49:32 UTC 
There's a new example added to the help text:

  # Mirror to an airgapped registry by first mirroring to files
  oc adm catalog mirror quay.io/my/image:latest file:///local/index
  oc adm catalog mirror file:///local/index/my/image:latest my-airgapped-registry.com

So for example:

oc adm catalog mirror quay.io/olmtest/busybox-dependencies-index:2.0.0-with-ListBundles-method file:///local/index

then you can take the directory (v2/local/index) and move it to another location (i.e. inside the airgap) and run

oc adm catalog mirror file:///local/index localhost:5000
Comment 12 Jian Zhang 2020-10-28 02:27:32 UTC 
Thanks Evan! Now, the latest oc doesn't contains the fixed PR, change the Status to MODIFIED.
[root@preserve-olm-env client]# ./oc version -o yaml
clientVersion:
  buildDate: "2020-10-26T17:03:22Z"
  compiler: gc
  gitCommit: 7b3fb8676abf29200a732676477c96f095a9fb27
  gitTreeState: clean
  gitVersion: openshift-clients-4.6.0-202006250705.p0-184-g7b3fb8676
  goVersion: go1.15.0
  major: ""
  minor: ""
  platform: linux/amd64
releaseClientVersion: 4.7.0-0.nightly-2020-10-27-051128
Comment 14 Bruno Andrade 2020-11-03 23:26:00 UTC 
Verified on Client Version: 4.7.0-0.ci-2020-11-03-102229


oc adm catalog mirror quay.io/olmtest/busybox-dependencies-index:2.0.0-with-ListBundles-method file:///local/index                                                                                                                    
src image has index label for database path: /database/index.db
using database path mapping: /database/index.db:/tmp/542896758
wrote database to /tmp/542896758
using database at: /tmp/542896758/index.db
<dir>
  local/index/library/busybox
    blobs:
      docker.io/library/busybox sha256:f0b02e9d092d905d0d87a8455a1ae3e9bb47b4aa3dc125125ca5cd10d6441c9f 1.458KiB
      docker.io/library/busybox sha256:9758c28807f21c13d05c704821fdd56c0b9574912f9b916c65e1df3e6b8bc572 746.7KiB
    manifests:
      sha256:8274214f1bb9614b494e01847ffea66ad68c916dd4465872e9c0e70ce229d01d
      sha256:c9249fdf56138f0d929e2080ae98ee9cb2946f71498fc1484288e6a935b5e5bc
      sha256:a9286defaba7b3a519d585ba0e37d0b2cbee74ebfe590960b0b1d6a5e97d1e1d -> latest
  local/index/olmtest/busybox-bundle
    blobs:
      quay.io/olmtest/busybox-bundle sha256:89829e959fd165ab400b794778c37930fd150b54c911460def28ab1760b8bdab 294B
      quay.io/olmtest/busybox-bundle sha256:97d90f4f8fa970f79ca5b95cc0cbcb42c3bedf1bdd283c6a9f8063bbc2defba2 294B
      quay.io/olmtest/busybox-bundle sha256:8dcfdd43df48d6a6feb9ef52698c56376acb6954ae367629346916d314524364 577B
      quay.io/olmtest/busybox-bundle sha256:91d574cfa4840e788fa63294ad6a7163d10906e51189a0abc13e6347f76239e9 611B
      quay.io/olmtest/busybox-bundle sha256:5ce916832c3bcc96fce4bc8b4f90225225eab25a5ac47e0a6a0b5bf6dd78b57b 3.281KiB
      quay.io/olmtest/busybox-bundle sha256:fcfd2e9805ff132aa7d5d1a07be7ec2b772ce9bbc3ea27b2216e57652add5bde 3.281KiB
    manifests:
      sha256:44e261c8eac2835027b11896a961c03d9d9e0bef6d79f9a9b4268a525013acc8 -> 2.0.0
      sha256:909ea461c8d033b14667b273ba7b6dd188a293c92bf076824836c03627982f2d -> 1.0.0
  local/index/olmtest/busybox-dependencies-index
    blobs:
      quay.io/olmtest/busybox-dependencies-index sha256:203a3d8f57dc03844a44cd5dd598b70d9e4496153b24d558f0fa75ebd88d8bfd 3.215KiB
      quay.io/olmtest/busybox-dependencies-index sha256:4434eb1de60bdbd4fc1339457e859db23ff1dc975e8f4b3e44724e79096acbfa 4.603KiB
      quay.io/olmtest/busybox-dependencies-index sha256:03260e9c3f626c4f2822688eda12971a44cd0bedeb21c194b5e22cd478db1126 1.963MiB
      quay.io/olmtest/busybox-dependencies-index sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c 2.668MiB
      quay.io/olmtest/busybox-dependencies-index sha256:f359ba578c53c12b5a58647e7d7aec4ffc65b108a858ec033051813735293915 3.602MiB
      quay.io/olmtest/busybox-dependencies-index sha256:1f60d16de36442f7a8c23be95f6026ab1966e83be25f9bc78d614705062ac0a3 15.27MiB
    manifests:
      sha256:1cae286a91e6e2c6974589a893d222215c7b17377beef059dc3ab76a6836f907 -> 2.0.0-with-ListBundles-method
  local/index/olmtest/busybox-dependency-bundle
    blobs:
      quay.io/olmtest/busybox-dependency-bundle sha256:1667f566a89541611bd418d879f7ed8a5abbc3710891d3b79cd97d9d1fb1239a 302B
      quay.io/olmtest/busybox-dependency-bundle sha256:bc9d67a63c6a4678671c82f0fc9bcd0805e832fa66d64b03e0b1439f14ded77c 304B
      quay.io/olmtest/busybox-dependency-bundle sha256:215a2e144e04bd609c19d76c6ea7eb1e99b6ca464dffd6bb6c3d84351c6b9953 812B
      quay.io/olmtest/busybox-dependency-bundle sha256:7a00a6fda1e2c115d3f6ef9dc9dc2a70f322946c35b5289bd343eafc96bd16cf 843B
      quay.io/olmtest/busybox-dependency-bundle sha256:3225c4caf073fb1676df5b4869953276cbe182835ce989489b8f18df0382a1ee 3.312KiB
      quay.io/olmtest/busybox-dependency-bundle sha256:7334dfc42a1c04a7691808d39110e4e006dfb49604d4a39d961dd1233b7273cf 3.314KiB
    manifests:
      sha256:6ea4735cc75a86d2a837a71e548cda9d21fcd9fc55e456e994b08f4b051f9ac3 -> 1.0.0
      sha256:eb4b13cba7a85a99055c9e177f5116066951e9ad7cde8dd8a7287ce79cc037f0 -> 2.0.0
  stats: shared=0 unique=20 size=24.26MiB ratio=1.00

phase 0:
   local/index/olmtest/busybox-dependencies-index blobs=6 mounts=0 manifests=1 shared=0
   local/index/olmtest/busybox-dependency-bundle  blobs=6 mounts=0 manifests=2 shared=0
   local/index/library/busybox                    blobs=2 mounts=0 manifests=3 shared=0
   local/index/olmtest/busybox-bundle             blobs=6 mounts=0 manifests=2 shared=0

info: Planning completed in 5.43s
uploading: file://local/index/olmtest/busybox-dependencies-index sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c 2.668MiB
uploading: file://local/index/library/busybox sha256:9758c28807f21c13d05c704821fdd56c0b9574912f9b916c65e1df3e6b8bc572 746.7KiB
uploading: file://local/index/olmtest/busybox-dependencies-index sha256:03260e9c3f626c4f2822688eda12971a44cd0bedeb21c194b5e22cd478db1126 1.963MiB
uploading: file://local/index/olmtest/busybox-dependencies-index sha256:1f60d16de36442f7a8c23be95f6026ab1966e83be25f9bc78d614705062ac0a3 15.27MiB
uploading: file://local/index/olmtest/busybox-dependencies-index sha256:f359ba578c53c12b5a58647e7d7aec4ffc65b108a858ec033051813735293915 3.602MiB
sha256:c9249fdf56138f0d929e2080ae98ee9cb2946f71498fc1484288e6a935b5e5bc file://local/index/library/busybox
sha256:8274214f1bb9614b494e01847ffea66ad68c916dd4465872e9c0e70ce229d01d file://local/index/library/busybox
sha256:1cae286a91e6e2c6974589a893d222215c7b17377beef059dc3ab76a6836f907 file://local/index/olmtest/busybox-dependencies-index:2.0.0-with-ListBundles-method
sha256:8274214f1bb9614b494e01847ffea66ad68c916dd4465872e9c0e70ce229d01d file://local/index/library/busybox:latest
sha256:eb4b13cba7a85a99055c9e177f5116066951e9ad7cde8dd8a7287ce79cc037f0 file://local/index/olmtest/busybox-dependency-bundle:2.0.0
sha256:6ea4735cc75a86d2a837a71e548cda9d21fcd9fc55e456e994b08f4b051f9ac3 file://local/index/olmtest/busybox-dependency-bundle:1.0.0
sha256:44e261c8eac2835027b11896a961c03d9d9e0bef6d79f9a9b4268a525013acc8 file://local/index/olmtest/busybox-bundle:2.0.0
sha256:909ea461c8d033b14667b273ba7b6dd188a293c92bf076824836c03627982f2d file://local/index/olmtest/busybox-bundle:1.0.0
info: Mirroring completed in 4.62s (5.505MB/s)
wrote mirroring manifests to manifests-busybox-dependencies-index-1604420049

To upload local images to a registry, run:

	oc adm catalog mirror file://local/index/olmtest/busybox-dependencies-index:2.0.0-with-ListBundles-method REGISTRY/REPOSITORY


  oc adm catalog mirror --insecure file://local/index/olmtest/busybox-dependencies-index:2.0.0-with-ListBundles-method localhost:5000/me:1.0
  src image has index label for database path: /database/index.db
  using database path mapping: /database/index.db:/tmp/169946936
  wrote database to /tmp/169946936
  using database at: /tmp/169946936/index.db
  localhost:5000/
    me/local-index-olmtest-busybox-dependencies-index
      blobs:
        file://local/index/olmtest/busybox-dependencies-index sha256:203a3d8f57dc03844a44cd5dd598b70d9e4496153b24d558f0fa75ebd88d8bfd 3.215KiB
        file://local/index/olmtest/busybox-dependencies-index sha256:4434eb1de60bdbd4fc1339457e859db23ff1dc975e8f4b3e44724e79096acbfa 4.603KiB
        file://local/index/olmtest/busybox-dependencies-index sha256:03260e9c3f626c4f2822688eda12971a44cd0bedeb21c194b5e22cd478db1126 1.963MiB
        file://local/index/olmtest/busybox-dependencies-index sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c 2.668MiB
        file://local/index/olmtest/busybox-dependencies-index sha256:f359ba578c53c12b5a58647e7d7aec4ffc65b108a858ec033051813735293915 3.602MiB
        file://local/index/olmtest/busybox-dependencies-index sha256:1f60d16de36442f7a8c23be95f6026ab1966e83be25f9bc78d614705062ac0a3 15.27MiB
      manifests:
        sha256:1cae286a91e6e2c6974589a893d222215c7b17377beef059dc3ab76a6836f907 -> 2.0.0-with-ListBundles-method
    stats: shared=0 unique=6 size=23.51MiB ratio=1.00

  info: Planning completed in 10ms
  uploading: localhost:5000/me/local-index-olmtest-busybox-dependencies-index sha256:1f60d16de36442f7a8c23be95f6026ab1966e83be25f9bc78d614705062ac0a3 15.27MiB
  uploading: localhost:5000/me/local-index-olmtest-busybox-dependencies-index sha256:03260e9c3f626c4f2822688eda12971a44cd0bedeb21c194b5e22cd478db1126 1.963MiB
  uploading: localhost:5000/me/local-index-olmtest-busybox-dependencies-index sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c 2.668MiB
  uploading: localhost:5000/me/local-index-olmtest-busybox-dependencies-index sha256:f359ba578c53c12b5a58647e7d7aec4ffc65b108a858ec033051813735293915 3.602MiB
  sha256:1cae286a91e6e2c6974589a893d222215c7b17377beef059dc3ab76a6836f907 localhost:5000/me/local-index-olmtest-busybox-dependencies-index:2.0.0-with-ListBundles-method
  info: Mirroring completed in 450ms (54.21MB/s)
  wrote mirroring manifests to manifests-index/olmtest/busybox-dependencies-index-1604420956
Comment 17 errata-xmlrpc 2021-02-24 15:12:15 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633