1919168 – `oc adm catalog mirror` doesn't work for the air-gapped cluster

Bug 1919168 - `oc adm catalog mirror` doesn't work for the air-gapped cluster

Summary: `oc adm catalog mirror` doesn't work for the air-gapped cluster

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	OLM
Sub Component:
Version:	4.7
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	4.8.0
Assignee:	Evan Cordell
QA Contact:	Jian Zhang
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	1926457 (view as bug list)
Depends On:
Blocks:	1942059 1990442
TreeView+	depends on / blocked

Reported:	2021-01-22 10:07 UTC by Jian Zhang
Modified:	2024-10-01 17:22 UTC (History)
CC List:	6 users (show)
Fixed In Version:
Doc Type:	Enhancement
Doc Text:	Feature: Air-gapped mirroring of operator catalogs Reason: Prior to this, the `oc adm catalog mirror` command could not be easily used to mirror across an airgap. Result: The contents of a catalog can be mirrored to a file system, placed onto removable media, and then mirrored back from the file system into a real registry for consumption by an airgapped cluster.
Clone Of:
Clones:	1990442 (view as bug list)
Environment:
Last Closed:	2021-07-27 22:36:39 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Github	openshift oc pull 772	None	open	Bug 1919168: when mirroring to a file destination, mount images under the index location	2021-03-19 21:02:37 UTC
Red Hat Knowledge Base (Solution)	6242391	None	None	None	2021-08-05 12:43:57 UTC
Red Hat Product Errata	RHSA-2021:2438	None	None	None	2021-07-27 22:36:56 UTC

Description Jian Zhang 2021-01-22 10:07:00 UTC

Description of problem:
Failed to mirror image to the remote registry from the local.
error: unable to retrieve source image file://local/index/olmqe/redhat-operator-index/openshift4/ose-logging-fluentd manifest sha256:11ccb42f3d96b065f7d94879611a7aefabbe509b522c11ac36be7a1c959a34d6: open v2/local/index/olmqe/redhat-operator-index/openshift4/ose-logging-fluentd/manifests/sha256:11ccb42f3d96b065f7d94879611a7aefabbe509b522c11ac36be7a1c959a34d6: no such file or directory

Version-Release number of selected component (if applicable):
[root@preserve-olm-env 4.6.8]# oc version -o yaml
clientVersion:
  buildDate: "2021-01-21T00:15:43Z"
  compiler: gc
  gitCommit: 6f8f260853ad23a1edeb7ee622da764e6b711e37
  gitTreeState: clean
  gitVersion: 4.7.0-202101202207.p0-6f8f260
  goVersion: go1.15.5
  major: ""
  minor: ""
  platform: linux/amd64
releaseClientVersion: 4.7.0-0.nightly-2021-01-21-012810
...


How reproducible:
always

Steps to Reproduce:
1. Mirror an index images to local.

[root@preserve-olm-env 4.6.8]# oc adm catalog mirror quay.io/olmqe/redhat-operator-index:v4.6-2 file:///local/index
src image has index label for database path: /database/index.db
using database path mapping: /database/index.db:/tmp/347746165
wrote database to /tmp/347746165
using database at: /tmp/347746165/index.db
<dir>
  local/index/olmqe/redhat-operator-index
    blobs:
      quay.io/olmqe/redhat-operator-index sha256:0b204e20a8c130523318d64af9072f601f5a5276ed2e3990594500db47dbf070
      quay.io/olmqe/redhat-operator-index sha256:188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964
      quay.io/olmqe/redhat-operator-index sha256:68ddc78a89ffedf0abdb66252299e6599feaedfb3244bf84e29b04c4335fa788
      quay.io/olmqe/redhat-operator-index sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
...
...
sha256:298a87856d08f3e04687d7f3316f2308e59d7ef1121b03fe6d9bb779a7279ee4 file://local/index/openshift4/ose-logging-curator5
sha256:73884604ac4506bcfb2a3c112eb621f40e0cd53fede8118e0f7a1b292ac8f924 file://local/index/openshift4/ose-logging-curator5:f6789123
info: Mirroring completed in 40.88s (38.39MB/s)
wrote mirroring manifests to manifests-redhat-operator-index-1611303212

To upload local images to a registry, run:

	oc adm catalog mirror file://local/index/olmqe/redhat-operator-index:v4.6-2 REGISTRY/REPOSITORY

2. Check the images mirrored to the local
[root@preserve-olm-env 4.6.8]# ls
manifests-redhat-operator-index-1611303212  v2

[root@preserve-olm-env 4.6.8]# tree manifests-redhat-operator-index-1611303212/
manifests-redhat-operator-index-1611303212/
└── mapping.txt

0 directories, 1 file
[root@preserve-olm-env 4.6.8]# tree v2/
v2/
└── local
    └── index
        ├── olmqe
        │   └── redhat-operator-index
        │       ├── blobs
        │       │   ├── sha256:0b204e20a8c130523318d64af9072f601f5a5276ed2e3990594500db47dbf070
...
        │       └── manifests
        │           ├── sha256:58d8f83a209cc7a1dd485c7ecc0264f1e2437ffd5992322c83c64d043b85031c
        │           └── v4.6-2 -> sha256:58d8f83a209cc7a1dd485c7ecc0264f1e2437ffd5992322c83c64d043b85031c
        └── openshift4
            ├── ose-cluster-logging-operator
            │   ├── blobs
            │   │   ├── sha256:04ef0e69dcba4088b009d14d21e86cda00ddbf8e84a4d9746c5d8ec9d61803af
...
...

[root@preserve-olm-env 4.6.8]# du -h --max-depth=1 v2
1.5G	v2/local
1.5G	v2
[root@preserve-olm-env 4.6.8]# du -h --max-depth=1 manifests-redhat-operator-index-1611303212/
4.0K	manifests-redhat-operator-index-1611303212/

Look good.

3. Switch to the parent folder, and tar this folder.
[root@preserve-olm-env 4.6.8]# cd ..
[root@preserve-olm-env release]# tar -zcvf 4.6.8.tar 4.6.8
4.6.8/
4.6.8/manifests-redhat-operator-index-1611303212/
4.6.8/manifests-redhat-operator-index-1611303212/mapping.txt
4.6.8/v2/
4.6.8/v2/local/
4.6.8/v2/local/index/
4.6.8/v2/local/index/olmqe/
4.6.8/v2/local/index/olmqe/redhat-operator-index/
4.6.8/v2/local/index/olmqe/redhat-operator-index/blobs/
...
4.6.8/v2/local/index/openshift4/ose-logging-curator5/manifests/sha256:73884604ac4506bcfb2a3c112eb621f40e0cd53fede8118e0f7a1b292ac8f924
4.6.8/v2/local/index/openshift4/ose-logging-curator5/manifests/f6789123

[root@preserve-olm-env release]# ls
4.6.8  4.6.8.tar
[root@preserve-olm-env release]# ls -lh 4.6.8.tar 
-rw-r--r--. 1 root root 1.5G Jan 22 08:37 4.6.8.tar

looks good.

4, Copy this tar package to another host. In real env, we should copy it to the Flash-Disk or Portable Hard Disk, and then copy it to the disconnected host.

5, Unzip this tar file.
[root@preserve-olm-env airgapped]# tar -xvf 4.6.8.tar
4.6.8/
4.6.8/manifests-redhat-operator-index-1611303212/
4.6.8/manifests-redhat-operator-index-1611303212/mapping.txt
4.6.8/v2/
4.6.8/v2/local/
4.6.8/v2/local/index/
4.6.8/v2/local/index/olmqe/
4.6.8/v2/local/index/olmqe/redhat-operator-index/
4.6.8/v2/local/index/olmqe/redhat-operator-index/blobs/
...
...

6, `cd` this folder and mirror the image to the remote registry via `oc adm catalog mirror`
[root@preserve-olm-env airgapped]# cd 4.6.8/
[root@preserve-olm-env 4.6.8]# oc adm catalog mirror file://local/index/olmqe/redhat-operator-index:v4.6-2 quay.io/jiazha
...

Actual results:
The index image was mirrored successfully, but the operators' images failed. Get the "no such file or directory" error.

error: unable to retrieve source image file://local/index/olmqe/redhat-operator-index/openshift4/ose-logging-fluentd manifest sha256:11ccb42f3d96b065f7d94879611a7aefabbe509b522c11ac36be7a1c959a34d6: open v2/local/index/olmqe/redhat-operator-index/openshift4/ose-logging-fluentd/manifests/sha256:11ccb42f3d96b065f7d94879611a7aefabbe509b522c11ac36be7a1c959a34d6: no such file or directory

I think the root cause is here:
It open and non-exist folder, for example, v2/local/index/olmqe/redhat-operator-index/openshift4/ose-logging-fluentd ...

It should open v2/local/index/openshift4/ose-logging-fluentd, see:
[root@preserve-olm-env 4.6.8]# tree v2/local/index/openshift4/ose-logging-fluentd/
v2/local/index/openshift4/ose-logging-fluentd/
├── blobs
│   ├── sha256:024e289f9d3059db8f47e89e0e13314df01f8651dcf9afb50c4c68d58cb92520
...

Should remove the "olmqe/redhat-operator-index/" from the operator images' mirror path.


Expected results:
The mirroring from the local disk should work well:
```
To upload local images to a registry, run:

	oc adm catalog mirror file://local/index/olmqe/redhat-operator-index:v4.6-2 REGISTRY/REPOSITORY
```


Additional info:
1, The help info:
[root@preserve-olm-env 4.6.8]# oc adm catalog mirror --help

Usage:
  oc adm catalog mirror SRC DEST [flags]

Examples:
...  
  # Mirror to an airgapped registry by first mirroring to files
  oc adm catalog mirror quay.io/my/image:latest file:///local/index
  oc adm catalog mirror file:///local/index/my/image:latest my-airgapped-registry.com


Options:
      --dir='': The directory on disk that file:// images will be copied under.
      --from-dir='': The directory on disk that file:// images will be read from. Overrides --dir
...

2, The full logs:
[root@preserve-olm-env 4.6.8]# oc adm catalog mirror file://local/index/olmqe/redhat-operator-index:v4.6-2 quay.io/jiazha
src image has index label for database path: /database/index.db
using database path mapping: /database/index.db:/tmp/475068864
wrote database to /tmp/475068864
using database at: /tmp/475068864/index.db
quay.io/
  jiazha/local-index-olmqe-redhat-operator-index
    blobs:
      file://local/index/olmqe/redhat-operator-index sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 32B
      file://local/index/olmqe/redhat-operator-index sha256:e83dfb2924e762e794606fdf7568137ea31bafb0c4bd280078cb1e0559760103 158B
      file://local/index/olmqe/redhat-operator-index sha256:4f41bd1637dcd854c3804d74453e7ecc9c47a64516812574fc02e2f61a5d47ae 1.651KiB
      file://local/index/olmqe/redhat-operator-index sha256:0b204e20a8c130523318d64af9072f601f5a5276ed2e3990594500db47dbf070 1.968MiB
      file://local/index/olmqe/redhat-operator-index sha256:68ddc78a89ffedf0abdb66252299e6599feaedfb3244bf84e29b04c4335fa788 2.572MiB
      file://local/index/olmqe/redhat-operator-index sha256:188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964 2.667MiB
      file://local/index/olmqe/redhat-operator-index sha256:acdafdf2d92fada644d2a2aa8c8ae80a96dd049357d182aca2fd866d06ce8233 3.602MiB
      file://local/index/olmqe/redhat-operator-index sha256:e3628f8eb74cc4427c0f9b643118eee323543e3f746835e1578e197e08ff0439 15.52MiB
    manifests:
      sha256:58d8f83a209cc7a1dd485c7ecc0264f1e2437ffd5992322c83c64d043b85031c -> v4.6-2
  stats: shared=0 unique=8 size=26.34MiB ratio=1.00

error: unable to retrieve source image file://local/index/olmqe/redhat-operator-index/openshift4/ose-logging-fluentd manifest sha256:11ccb42f3d96b065f7d94879611a7aefabbe509b522c11ac36be7a1c959a34d6: open v2/local/index/olmqe/redhat-operator-index/openshift4/ose-logging-fluentd/manifests/sha256:11ccb42f3d96b065f7d94879611a7aefabbe509b522c11ac36be7a1c959a34d6: no such file or directory
error: unable to retrieve source image file://local/index/olmqe/redhat-operator-index/openshift4/ose-cluster-logging-operator manifest sha256:540b0d087c5e5529bab555030310478630249a9339a4a4c3fac6d0d7037d5eac: open v2/local/index/olmqe/redhat-operator-index/openshift4/ose-cluster-logging-operator/manifests/sha256:540b0d087c5e5529bab555030310478630249a9339a4a4c3fac6d0d7037d5eac: no such file or directory
error: unable to retrieve source image file://local/index/olmqe/redhat-operator-index/openshift4/ose-logging-curator5 manifest sha256:73884604ac4506bcfb2a3c112eb621f40e0cd53fede8118e0f7a1b292ac8f924: open v2/local/index/olmqe/redhat-operator-index/openshift4/ose-logging-curator5/manifests/sha256:73884604ac4506bcfb2a3c112eb621f40e0cd53fede8118e0f7a1b292ac8f924: no such file or directory
error: unable to retrieve source image file://local/index/olmqe/redhat-operator-index/openshift4/ose-cluster-logging-operator-bundle manifest sha256:61aca61840dcf1d50f4a17fc9b2e10b7855c563bd6680b4dc77e3e9283c81369: open v2/local/index/olmqe/redhat-operator-index/openshift4/ose-cluster-logging-operator-bundle/manifests/sha256:61aca61840dcf1d50f4a17fc9b2e10b7855c563bd6680b4dc77e3e9283c81369: no such file or directory
phase 0:
  quay.io jiazha/local-index-olmqe-redhat-operator-index blobs=8 mounts=0 manifests=1 shared=0

info: Planning completed in 990ms
uploading: quay.io/jiazha/local-index-olmqe-redhat-operator-index sha256:188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964 2.667MiB
uploading: quay.io/jiazha/local-index-olmqe-redhat-operator-index sha256:0b204e20a8c130523318d64af9072f601f5a5276ed2e3990594500db47dbf070 1.968MiB
uploading: quay.io/jiazha/local-index-olmqe-redhat-operator-index sha256:e3628f8eb74cc4427c0f9b643118eee323543e3f746835e1578e197e08ff0439 15.52MiB
uploading: quay.io/jiazha/local-index-olmqe-redhat-operator-index sha256:acdafdf2d92fada644d2a2aa8c8ae80a96dd049357d182aca2fd866d06ce8233 3.602MiB
uploading: quay.io/jiazha/local-index-olmqe-redhat-operator-index sha256:68ddc78a89ffedf0abdb66252299e6599feaedfb3244bf84e29b04c4335fa788 2.572MiB
sha256:58d8f83a209cc7a1dd485c7ecc0264f1e2437ffd5992322c83c64d043b85031c quay.io/jiazha/local-index-olmqe-redhat-operator-index:v4.6-2
info: Mirroring completed in 4.68s (5.898MB/s)
error mirroring image: one or more errors occurred
no digest mapping available for file://local/index/olmqe/redhat-operator-index:v4.6-2, skip writing to ImageContentSourcePolicy
wrote mirroring manifests to manifests-index/olmqe/redhat-operator-index-1611305625

Comment 1 Jian Zhang 2021-01-22 10:17:59 UTC

[jzhang@dhcp-140-36 4.6.8]$ oc adm catalog mirror --dir=v2/ file://local/index/olmqe/redhat-operator-index:v4.6-2 quay.io/jiazha 
error: unable to read image file://local/index/olmqe/redhat-operator-index:v4.6-2: unknown blob

[jzhang@dhcp-140-36 4.6.8]$ oc image  mirror --dir=v2/ file://local/index/olmqe/redhat-operator-index:v4.6-2 quay.io/jiazha 

error: unable to retrieve source image file://local/index/olmqe/redhat-operator-index by tag v4.6-2: unknown blob
error: an error occurred during planning

Comment 5 Jian Zhang 2021-03-24 04:02:16 UTC

Use the `oc` client that fixed PR merged for testing.
[root@preserve-olm-env client]# oc version -o yaml
clientVersion:
  buildDate: "2021-03-23T20:25:48Z"
  compiler: gc
  gitCommit: 314b90b414e3048a243d828435ffc48e1549d862
  gitTreeState: clean
  gitVersion: 4.8.0-202103231739.p0-314b90b
  goVersion: go1.15.7
  major: ""
  minor: ""
  platform: linux/amd64
openshiftVersion: 4.8.0-0.nightly-2021-03-22-104536
releaseClientVersion: 4.8.0-0.nightly-2021-03-23-225416
serverVersion:
  buildDate: "2021-03-19T19:56:04Z"
  compiler: gc
  gitCommit: 39c0afe26e6c66d57a6ee53b925c16a6c4d272c7
  gitTreeState: clean
  gitVersion: v1.20.0+39c0afe
  goVersion: go1.15.7
  major: "1"
  minor: "20"
  platform: linux/amd64

1. Mirror an index images to local.
[root@preserve-olm-env bug1919168]# oc adm catalog mirror quay.io/olmqe/redhat-operator-index:v4.6-2 file:///local/index
src image has index label for database path: /database/index.db
using database path mapping: /database/index.db:/tmp/095063026
wrote database to /tmp/095063026
using database at: /tmp/095063026/index.db
<dir>
  local/index/olmqe/redhat-operator-index
...
info: Mirroring completed in 40.21s (40.68MB/s)
wrote mirroring manifests to manifests-redhat-operator-index-1616556527

To upload local images to a registry, run:

	oc adm catalog mirror file://local/index/olmqe/redhat-operator-index:v4.6-2 REGISTRY/REPOSITORY

2. Check the images mirrored to the local
[root@preserve-olm-env bug1919168]# ls
manifests-redhat-operator-index-1616556527  v2

3. Switch to the parent folder, and tar this folder.
[root@preserve-olm-env bug1919168]# cd ..
[root@preserve-olm-env airgapped]# ls
4.6.8  4.6.8.tar  bug1919168
[root@preserve-olm-env airgapped]# tar -zcvf bug1919168.tar bug1919168
bug1919168/
bug1919168/manifests-redhat-operator-index-1616556527/
bug1919168/manifests-redhat-operator-index-1616556527/mapping.txt
bug1919168/v2/
...
bug1919168/v2/local/index/olmqe/redhat-operator-index/manifests/v4.6-2


4, Copy this tar package to another host. In real env, we should copy it to the Flash-Disk or Portable Hard Disk, and then copy it to the disconnected host.

5, Unzip this tar file.
[root@preserve-olm-env airgapped]# ls
4.6.8.tar  bug1919168.tar
[root@preserve-olm-env airgapped]# tar -xvf bug1919168.tar
...
bug1919168/v2/local/index/olmqe/redhat-operator-index/manifests/v4.6-2
[root@preserve-olm-env airgapped]# ls
4.6.8.tar  bug1919168  bug1919168.tar

6, `cd` this folder and mirror the image to the remote registry via `oc adm catalog mirror`
[root@preserve-olm-env airgapped]# cd bug1919168/
[root@preserve-olm-env bug1919168]# ls
manifests-redhat-operator-index-1616556527  v2

[root@preserve-olm-env 4.6.8]# oc adm catalog mirror file://local/index/olmqe/redhat-operator-index:v4.6-2 quay.io/jiazha
...
sha256:c1105fb39c658762597c1537ee9c1176a63c887dbdb7dc0709fe39ca9d0a4599 quay.io/jiazha/openshift4-ose-logging-curator5:7d0bc7a4
info: Mirroring completed in 1m38s (16.69MB/s)
no digest mapping available for file://local/index/olmqe/redhat-operator-index:v4.6-2, skip writing to ImageContentSourcePolicy
wrote mirroring manifests to manifests-index/olmqe/redhat-operator-index-1616557104
[root@preserve-olm-env bug1919168]# ls
manifests-index  manifests-redhat-operator-index-1616556527  v2
[root@preserve-olm-env bug1919168]# tree manifests-index/
manifests-index/
└── olmqe
    └── redhat-operator-index-1616557104
        ├── catalogSource.yaml
        ├── imageContentSourcePolicy.yaml
        └── mapping.txt

2 directories, 3 files

LGTM, verify it.

Comment 6 Kevin Rizza 2021-06-07 17:58:27 UTC

*** Bug 1926457 has been marked as a duplicate of this bug. ***

Comment 10 errata-xmlrpc 2021-07-27 22:36:39 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438

Note You need to log in before you can comment on or make changes to this bug.