Bug 1919168
| Summary: | `oc adm catalog mirror` doesn't work for the air-gapped cluster | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Jian Zhang <jiazha> | |
| Component: | OLM | Assignee: | Evan Cordell <ecordell> | |
| OLM sub component: | OLM | QA Contact: | Jian Zhang <jiazha> | |
| Status: | CLOSED ERRATA | Docs Contact: | ||
| Severity: | medium | |||
| Priority: | medium | CC: | akaris, ansulliv, cshepher, dsover, ecordell, krizza | |
| Version: | 4.7 | Keywords: | Triaged | |
| Target Milestone: | --- | |||
| Target Release: | 4.8.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | Enhancement | ||
| Doc Text: |
Feature: Air-gapped mirroring of operator catalogs
Reason: Prior to this, the `oc adm catalog mirror` command could not be easily used to mirror across an airgap.
Result: The contents of a catalog can be mirrored to a file system, placed onto removable media, and then mirrored back from the file system into a real registry for consumption by an airgapped cluster.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1990442 (view as bug list) | Environment: | ||
| Last Closed: | 2021-07-27 22:36:39 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1942059, 1990442 | |||
[jzhang@dhcp-140-36 4.6.8]$ oc adm catalog mirror --dir=v2/ file://local/index/olmqe/redhat-operator-index:v4.6-2 quay.io/jiazha error: unable to read image file://local/index/olmqe/redhat-operator-index:v4.6-2: unknown blob [jzhang@dhcp-140-36 4.6.8]$ oc image mirror --dir=v2/ file://local/index/olmqe/redhat-operator-index:v4.6-2 quay.io/jiazha error: unable to retrieve source image file://local/index/olmqe/redhat-operator-index by tag v4.6-2: unknown blob error: an error occurred during planning Use the `oc` client that fixed PR merged for testing. [root@preserve-olm-env client]# oc version -o yaml clientVersion: buildDate: "2021-03-23T20:25:48Z" compiler: gc gitCommit: 314b90b414e3048a243d828435ffc48e1549d862 gitTreeState: clean gitVersion: 4.8.0-202103231739.p0-314b90b goVersion: go1.15.7 major: "" minor: "" platform: linux/amd64 openshiftVersion: 4.8.0-0.nightly-2021-03-22-104536 releaseClientVersion: 4.8.0-0.nightly-2021-03-23-225416 serverVersion: buildDate: "2021-03-19T19:56:04Z" compiler: gc gitCommit: 39c0afe26e6c66d57a6ee53b925c16a6c4d272c7 gitTreeState: clean gitVersion: v1.20.0+39c0afe goVersion: go1.15.7 major: "1" minor: "20" platform: linux/amd64 1. Mirror an index images to local. [root@preserve-olm-env bug1919168]# oc adm catalog mirror quay.io/olmqe/redhat-operator-index:v4.6-2 file:///local/index src image has index label for database path: /database/index.db using database path mapping: /database/index.db:/tmp/095063026 wrote database to /tmp/095063026 using database at: /tmp/095063026/index.db <dir> local/index/olmqe/redhat-operator-index ... info: Mirroring completed in 40.21s (40.68MB/s) wrote mirroring manifests to manifests-redhat-operator-index-1616556527 To upload local images to a registry, run: oc adm catalog mirror file://local/index/olmqe/redhat-operator-index:v4.6-2 REGISTRY/REPOSITORY 2. Check the images mirrored to the local [root@preserve-olm-env bug1919168]# ls manifests-redhat-operator-index-1616556527 v2 3. Switch to the parent folder, and tar this folder. [root@preserve-olm-env bug1919168]# cd .. [root@preserve-olm-env airgapped]# ls 4.6.8 4.6.8.tar bug1919168 [root@preserve-olm-env airgapped]# tar -zcvf bug1919168.tar bug1919168 bug1919168/ bug1919168/manifests-redhat-operator-index-1616556527/ bug1919168/manifests-redhat-operator-index-1616556527/mapping.txt bug1919168/v2/ ... bug1919168/v2/local/index/olmqe/redhat-operator-index/manifests/v4.6-2 4, Copy this tar package to another host. In real env, we should copy it to the Flash-Disk or Portable Hard Disk, and then copy it to the disconnected host. 5, Unzip this tar file. [root@preserve-olm-env airgapped]# ls 4.6.8.tar bug1919168.tar [root@preserve-olm-env airgapped]# tar -xvf bug1919168.tar ... bug1919168/v2/local/index/olmqe/redhat-operator-index/manifests/v4.6-2 [root@preserve-olm-env airgapped]# ls 4.6.8.tar bug1919168 bug1919168.tar 6, `cd` this folder and mirror the image to the remote registry via `oc adm catalog mirror` [root@preserve-olm-env airgapped]# cd bug1919168/ [root@preserve-olm-env bug1919168]# ls manifests-redhat-operator-index-1616556527 v2 [root@preserve-olm-env 4.6.8]# oc adm catalog mirror file://local/index/olmqe/redhat-operator-index:v4.6-2 quay.io/jiazha ... sha256:c1105fb39c658762597c1537ee9c1176a63c887dbdb7dc0709fe39ca9d0a4599 quay.io/jiazha/openshift4-ose-logging-curator5:7d0bc7a4 info: Mirroring completed in 1m38s (16.69MB/s) no digest mapping available for file://local/index/olmqe/redhat-operator-index:v4.6-2, skip writing to ImageContentSourcePolicy wrote mirroring manifests to manifests-index/olmqe/redhat-operator-index-1616557104 [root@preserve-olm-env bug1919168]# ls manifests-index manifests-redhat-operator-index-1616556527 v2 [root@preserve-olm-env bug1919168]# tree manifests-index/ manifests-index/ └── olmqe └── redhat-operator-index-1616557104 ├── catalogSource.yaml ├── imageContentSourcePolicy.yaml └── mapping.txt 2 directories, 3 files LGTM, verify it. *** Bug 1926457 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438 |
Description of problem: Failed to mirror image to the remote registry from the local. error: unable to retrieve source image file://local/index/olmqe/redhat-operator-index/openshift4/ose-logging-fluentd manifest sha256:11ccb42f3d96b065f7d94879611a7aefabbe509b522c11ac36be7a1c959a34d6: open v2/local/index/olmqe/redhat-operator-index/openshift4/ose-logging-fluentd/manifests/sha256:11ccb42f3d96b065f7d94879611a7aefabbe509b522c11ac36be7a1c959a34d6: no such file or directory Version-Release number of selected component (if applicable): [root@preserve-olm-env 4.6.8]# oc version -o yaml clientVersion: buildDate: "2021-01-21T00:15:43Z" compiler: gc gitCommit: 6f8f260853ad23a1edeb7ee622da764e6b711e37 gitTreeState: clean gitVersion: 4.7.0-202101202207.p0-6f8f260 goVersion: go1.15.5 major: "" minor: "" platform: linux/amd64 releaseClientVersion: 4.7.0-0.nightly-2021-01-21-012810 ... How reproducible: always Steps to Reproduce: 1. Mirror an index images to local. [root@preserve-olm-env 4.6.8]# oc adm catalog mirror quay.io/olmqe/redhat-operator-index:v4.6-2 file:///local/index src image has index label for database path: /database/index.db using database path mapping: /database/index.db:/tmp/347746165 wrote database to /tmp/347746165 using database at: /tmp/347746165/index.db <dir> local/index/olmqe/redhat-operator-index blobs: quay.io/olmqe/redhat-operator-index sha256:0b204e20a8c130523318d64af9072f601f5a5276ed2e3990594500db47dbf070 quay.io/olmqe/redhat-operator-index sha256:188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964 quay.io/olmqe/redhat-operator-index sha256:68ddc78a89ffedf0abdb66252299e6599feaedfb3244bf84e29b04c4335fa788 quay.io/olmqe/redhat-operator-index sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 ... ... sha256:298a87856d08f3e04687d7f3316f2308e59d7ef1121b03fe6d9bb779a7279ee4 file://local/index/openshift4/ose-logging-curator5 sha256:73884604ac4506bcfb2a3c112eb621f40e0cd53fede8118e0f7a1b292ac8f924 file://local/index/openshift4/ose-logging-curator5:f6789123 info: Mirroring completed in 40.88s (38.39MB/s) wrote mirroring manifests to manifests-redhat-operator-index-1611303212 To upload local images to a registry, run: oc adm catalog mirror file://local/index/olmqe/redhat-operator-index:v4.6-2 REGISTRY/REPOSITORY 2. Check the images mirrored to the local [root@preserve-olm-env 4.6.8]# ls manifests-redhat-operator-index-1611303212 v2 [root@preserve-olm-env 4.6.8]# tree manifests-redhat-operator-index-1611303212/ manifests-redhat-operator-index-1611303212/ └── mapping.txt 0 directories, 1 file [root@preserve-olm-env 4.6.8]# tree v2/ v2/ └── local └── index ├── olmqe │ └── redhat-operator-index │ ├── blobs │ │ ├── sha256:0b204e20a8c130523318d64af9072f601f5a5276ed2e3990594500db47dbf070 ... │ └── manifests │ ├── sha256:58d8f83a209cc7a1dd485c7ecc0264f1e2437ffd5992322c83c64d043b85031c │ └── v4.6-2 -> sha256:58d8f83a209cc7a1dd485c7ecc0264f1e2437ffd5992322c83c64d043b85031c └── openshift4 ├── ose-cluster-logging-operator │ ├── blobs │ │ ├── sha256:04ef0e69dcba4088b009d14d21e86cda00ddbf8e84a4d9746c5d8ec9d61803af ... ... [root@preserve-olm-env 4.6.8]# du -h --max-depth=1 v2 1.5G v2/local 1.5G v2 [root@preserve-olm-env 4.6.8]# du -h --max-depth=1 manifests-redhat-operator-index-1611303212/ 4.0K manifests-redhat-operator-index-1611303212/ Look good. 3. Switch to the parent folder, and tar this folder. [root@preserve-olm-env 4.6.8]# cd .. [root@preserve-olm-env release]# tar -zcvf 4.6.8.tar 4.6.8 4.6.8/ 4.6.8/manifests-redhat-operator-index-1611303212/ 4.6.8/manifests-redhat-operator-index-1611303212/mapping.txt 4.6.8/v2/ 4.6.8/v2/local/ 4.6.8/v2/local/index/ 4.6.8/v2/local/index/olmqe/ 4.6.8/v2/local/index/olmqe/redhat-operator-index/ 4.6.8/v2/local/index/olmqe/redhat-operator-index/blobs/ ... 4.6.8/v2/local/index/openshift4/ose-logging-curator5/manifests/sha256:73884604ac4506bcfb2a3c112eb621f40e0cd53fede8118e0f7a1b292ac8f924 4.6.8/v2/local/index/openshift4/ose-logging-curator5/manifests/f6789123 [root@preserve-olm-env release]# ls 4.6.8 4.6.8.tar [root@preserve-olm-env release]# ls -lh 4.6.8.tar -rw-r--r--. 1 root root 1.5G Jan 22 08:37 4.6.8.tar looks good. 4, Copy this tar package to another host. In real env, we should copy it to the Flash-Disk or Portable Hard Disk, and then copy it to the disconnected host. 5, Unzip this tar file. [root@preserve-olm-env airgapped]# tar -xvf 4.6.8.tar 4.6.8/ 4.6.8/manifests-redhat-operator-index-1611303212/ 4.6.8/manifests-redhat-operator-index-1611303212/mapping.txt 4.6.8/v2/ 4.6.8/v2/local/ 4.6.8/v2/local/index/ 4.6.8/v2/local/index/olmqe/ 4.6.8/v2/local/index/olmqe/redhat-operator-index/ 4.6.8/v2/local/index/olmqe/redhat-operator-index/blobs/ ... ... 6, `cd` this folder and mirror the image to the remote registry via `oc adm catalog mirror` [root@preserve-olm-env airgapped]# cd 4.6.8/ [root@preserve-olm-env 4.6.8]# oc adm catalog mirror file://local/index/olmqe/redhat-operator-index:v4.6-2 quay.io/jiazha ... Actual results: The index image was mirrored successfully, but the operators' images failed. Get the "no such file or directory" error. error: unable to retrieve source image file://local/index/olmqe/redhat-operator-index/openshift4/ose-logging-fluentd manifest sha256:11ccb42f3d96b065f7d94879611a7aefabbe509b522c11ac36be7a1c959a34d6: open v2/local/index/olmqe/redhat-operator-index/openshift4/ose-logging-fluentd/manifests/sha256:11ccb42f3d96b065f7d94879611a7aefabbe509b522c11ac36be7a1c959a34d6: no such file or directory I think the root cause is here: It open and non-exist folder, for example, v2/local/index/olmqe/redhat-operator-index/openshift4/ose-logging-fluentd ... It should open v2/local/index/openshift4/ose-logging-fluentd, see: [root@preserve-olm-env 4.6.8]# tree v2/local/index/openshift4/ose-logging-fluentd/ v2/local/index/openshift4/ose-logging-fluentd/ ├── blobs │ ├── sha256:024e289f9d3059db8f47e89e0e13314df01f8651dcf9afb50c4c68d58cb92520 ... Should remove the "olmqe/redhat-operator-index/" from the operator images' mirror path. Expected results: The mirroring from the local disk should work well: ``` To upload local images to a registry, run: oc adm catalog mirror file://local/index/olmqe/redhat-operator-index:v4.6-2 REGISTRY/REPOSITORY ``` Additional info: 1, The help info: [root@preserve-olm-env 4.6.8]# oc adm catalog mirror --help Usage: oc adm catalog mirror SRC DEST [flags] Examples: ... # Mirror to an airgapped registry by first mirroring to files oc adm catalog mirror quay.io/my/image:latest file:///local/index oc adm catalog mirror file:///local/index/my/image:latest my-airgapped-registry.com Options: --dir='': The directory on disk that file:// images will be copied under. --from-dir='': The directory on disk that file:// images will be read from. Overrides --dir ... 2, The full logs: [root@preserve-olm-env 4.6.8]# oc adm catalog mirror file://local/index/olmqe/redhat-operator-index:v4.6-2 quay.io/jiazha src image has index label for database path: /database/index.db using database path mapping: /database/index.db:/tmp/475068864 wrote database to /tmp/475068864 using database at: /tmp/475068864/index.db quay.io/ jiazha/local-index-olmqe-redhat-operator-index blobs: file://local/index/olmqe/redhat-operator-index sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 32B file://local/index/olmqe/redhat-operator-index sha256:e83dfb2924e762e794606fdf7568137ea31bafb0c4bd280078cb1e0559760103 158B file://local/index/olmqe/redhat-operator-index sha256:4f41bd1637dcd854c3804d74453e7ecc9c47a64516812574fc02e2f61a5d47ae 1.651KiB file://local/index/olmqe/redhat-operator-index sha256:0b204e20a8c130523318d64af9072f601f5a5276ed2e3990594500db47dbf070 1.968MiB file://local/index/olmqe/redhat-operator-index sha256:68ddc78a89ffedf0abdb66252299e6599feaedfb3244bf84e29b04c4335fa788 2.572MiB file://local/index/olmqe/redhat-operator-index sha256:188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964 2.667MiB file://local/index/olmqe/redhat-operator-index sha256:acdafdf2d92fada644d2a2aa8c8ae80a96dd049357d182aca2fd866d06ce8233 3.602MiB file://local/index/olmqe/redhat-operator-index sha256:e3628f8eb74cc4427c0f9b643118eee323543e3f746835e1578e197e08ff0439 15.52MiB manifests: sha256:58d8f83a209cc7a1dd485c7ecc0264f1e2437ffd5992322c83c64d043b85031c -> v4.6-2 stats: shared=0 unique=8 size=26.34MiB ratio=1.00 error: unable to retrieve source image file://local/index/olmqe/redhat-operator-index/openshift4/ose-logging-fluentd manifest sha256:11ccb42f3d96b065f7d94879611a7aefabbe509b522c11ac36be7a1c959a34d6: open v2/local/index/olmqe/redhat-operator-index/openshift4/ose-logging-fluentd/manifests/sha256:11ccb42f3d96b065f7d94879611a7aefabbe509b522c11ac36be7a1c959a34d6: no such file or directory error: unable to retrieve source image file://local/index/olmqe/redhat-operator-index/openshift4/ose-cluster-logging-operator manifest sha256:540b0d087c5e5529bab555030310478630249a9339a4a4c3fac6d0d7037d5eac: open v2/local/index/olmqe/redhat-operator-index/openshift4/ose-cluster-logging-operator/manifests/sha256:540b0d087c5e5529bab555030310478630249a9339a4a4c3fac6d0d7037d5eac: no such file or directory error: unable to retrieve source image file://local/index/olmqe/redhat-operator-index/openshift4/ose-logging-curator5 manifest sha256:73884604ac4506bcfb2a3c112eb621f40e0cd53fede8118e0f7a1b292ac8f924: open v2/local/index/olmqe/redhat-operator-index/openshift4/ose-logging-curator5/manifests/sha256:73884604ac4506bcfb2a3c112eb621f40e0cd53fede8118e0f7a1b292ac8f924: no such file or directory error: unable to retrieve source image file://local/index/olmqe/redhat-operator-index/openshift4/ose-cluster-logging-operator-bundle manifest sha256:61aca61840dcf1d50f4a17fc9b2e10b7855c563bd6680b4dc77e3e9283c81369: open v2/local/index/olmqe/redhat-operator-index/openshift4/ose-cluster-logging-operator-bundle/manifests/sha256:61aca61840dcf1d50f4a17fc9b2e10b7855c563bd6680b4dc77e3e9283c81369: no such file or directory phase 0: quay.io jiazha/local-index-olmqe-redhat-operator-index blobs=8 mounts=0 manifests=1 shared=0 info: Planning completed in 990ms uploading: quay.io/jiazha/local-index-olmqe-redhat-operator-index sha256:188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964 2.667MiB uploading: quay.io/jiazha/local-index-olmqe-redhat-operator-index sha256:0b204e20a8c130523318d64af9072f601f5a5276ed2e3990594500db47dbf070 1.968MiB uploading: quay.io/jiazha/local-index-olmqe-redhat-operator-index sha256:e3628f8eb74cc4427c0f9b643118eee323543e3f746835e1578e197e08ff0439 15.52MiB uploading: quay.io/jiazha/local-index-olmqe-redhat-operator-index sha256:acdafdf2d92fada644d2a2aa8c8ae80a96dd049357d182aca2fd866d06ce8233 3.602MiB uploading: quay.io/jiazha/local-index-olmqe-redhat-operator-index sha256:68ddc78a89ffedf0abdb66252299e6599feaedfb3244bf84e29b04c4335fa788 2.572MiB sha256:58d8f83a209cc7a1dd485c7ecc0264f1e2437ffd5992322c83c64d043b85031c quay.io/jiazha/local-index-olmqe-redhat-operator-index:v4.6-2 info: Mirroring completed in 4.68s (5.898MB/s) error mirroring image: one or more errors occurred no digest mapping available for file://local/index/olmqe/redhat-operator-index:v4.6-2, skip writing to ImageContentSourcePolicy wrote mirroring manifests to manifests-index/olmqe/redhat-operator-index-1611305625