Bug 1806780 - Cannot mirror a local release to a remote registry
Summary: Cannot mirror a local release to a remote registry
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oc
Version: 4.4
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: 4.4.z
Assignee: W. Trevor King
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On: 1806779
Blocks: 1806782
TreeView+ depends on / blocked
 
Reported: 2020-02-25 02:17 UTC by Clayton Coleman
Modified: 2020-06-17 22:26 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1806779
: 1806782 (view as bug list)
Environment:
Last Closed: 2020-06-17 22:26:03 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift oc pull 327 0 None closed [release-4.4] Bug 1806780: Allow a file:// release argument to release mirror 2021-01-28 00:15:08 UTC
Github openshift oc pull 446 0 None closed Bug 1806780: release: Mirroring from custom directory via oc adm release mirror fails 2021-01-28 00:15:08 UTC
Red Hat Product Errata RHBA-2020:2445 0 None None None 2020-06-17 22:26:28 UTC

Description Clayton Coleman 2020-02-25 02:17:43 UTC
+++ This bug was initially created as a clone of Bug #1806779 +++

The oc adm release mirror file support was intended to support downloading the release to disk and then pushing it to a remote repository after moving into an isolated network.  However, it is not possible to specify a from source for the release that includes both release and images today.

The intended usage is:

oc adm release mirror OPENSHIFT_VERSION --to file://openshift/release
# move across networks
oc adm release mirror file://openshift/release:OPENSHIFT_VERSION --to MIRROR_REPOSITORY

Because the location on disk involves changing the image location, and mirror cannot rewrite the release contents, we need oc adm release mirror to support a heuristic that checks the local store for a given digest within the release and uses that as the source if found.

This will allow mirror to correctly support both sides of the mirroring operation.

Backport to 4.3 so offline support can be completed. Low risk because this is an optional code path and the fallback is the previous behavior.

Comment 3 Johnny Liu 2020-03-03 13:41:35 UTC
est this bug with 4.4.0-0.nightly-2020-03-03-065638.

# oc version
Client Version: 4.4.0-0.nightly-2020-03-03-065638

# oc adm release mirror -a ~/mirror_pullsecret_config.json --from=registry.svc.ci.openshift.org/ocp/release:4.4.0-0.nightly-2020-03-03-065638 --to=file://test
<--snip-->
Success
Update image:  test:4.4.0-0.nightly-2020-03-03-065638
Mirror prefix: file://test

To upload local images to a registry, run:

    oc image mirror 'file://test:4.4.0-0.nightly-2020-03-03-065638*' REGISTRY/REPOSITORY


The command from the above command is working well.
# oc image mirror 'file://test:4.4.0-0.nightly-2020-03-03-065638*' upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release
upshift.mirror-registry.qe.devcluster.openshift.com:5000/
  ocp/release
    manifests:
      sha256:001fc4d0a20206748a3e0b3ce0ae3b6c9ee88cf508dda9db03ac78db5ebb20e5 -> 4.4.0-0.nightly-2020-03-03-065638-cluster-etcd-operator
<--snip-->
sha256:d1db4d6ea9acb3eb919025d924c98946dfa9188269e33888f5b278334632423d upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release:4.4.0-0.nightly-2020-03-03-065638-telemeter
info: Mirroring completed in 930ms (0B/s)


But the following questions are about how to consume the mirror image.
1. No 'imageContentSources' output when uploading disk file to internal registry (this is different when mirror image from public registry to internal registry). That is inconvenient to use this mirror image for installation.
2. The mirror command is using "oc adm release mirror" to mirror image to disk, but output is suggesting to use 'oc image mirror' to upload image to private registry, this is not intended usage, Should also use "oc adm release mirror" for uploading, isn't it? 
3. I also tried to use "oc adm release mirror" command to upload the disk files to private registry, command return success, but still no 'imageContentSources' output.
# oc adm release mirror 'file://test:4.4.0-0.nightly-2020-03-03-065638*' --to=upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release
error: unable to retrieve release image info: "test:4.4.0-0.nightly-2020-03-03-065638*" is not a valid image reference: invalid reference format

# oc adm release mirror --from=file://test:4.4.0-0.nightly-2020-03-03-065638 --to=upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release
info: Mirroring 108 images to upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release ...
upshift.mirror-registry.qe.devcluster.openshift.com:5000/
  ocp/release
    manifests:
<--snip-->
info: Mirroring completed in 1.7s (0B/s)

Success
Update image:  upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release:4.4.0-0.nightly-2020-03-03-065638
Mirror prefix: upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release

Comment 4 Johnny Liu 2020-03-04 14:20:15 UTC
One more issue, seem like --from-dir does not work.

when pwd == root, run the following command:
# oc adm release mirror --from-dir=/opt/mirror-to-disk/jialiu441_2020-03-04-07-27-41/data --from=file://test:4.4 --to=upshift-nointernet.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release --to-release-image=upshift-nointernet.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release:4.4'

error: unable to retrieve release image info: unable to read image file://test:4.4: unknown blob

Once I changed pwd to /opt/mirror-to-disk/jialiu441_2020-03-04-07-27-41/data, run this command again, then it works.

Comment 5 Maciej Szulik 2020-03-06 10:43:33 UTC
The problems outline in here, are not super important to block 4.4, moving to 4.4.z

Comment 7 W. Trevor King 2020-06-01 18:47:42 UTC
> Backport to 4.3 so offline support can be completed.

We're into POST for 4.4 now, but 4.3 is about to enter maintenance mode.  My impression is that it is probably not worth backporting this to 4.3 at this point, but we could revisit if someone can explain why they can't use a 4.4 oc to work around 4.3 not having the backport.

Comment 10 Johnny Liu 2020-06-08 02:27:04 UTC
Verified this bug with 4.4.0-0.nightly-2020-06-07-075345, and PASS.


1. mirror image to local disk file
$  /home/installer3/mnt/2020-06-07-21-38-15/jialiu23_2020-06-07-21-38-15/oc adm release mirror -a /home/installer3/mirror_pullsecret_config.json --from=registry.svc.ci.openshift.org/ocp/release:4.4.0-0.nightly-2020-06-07-075345 --to-dir=/home/installer3/mnt/2020-06-07-21-38-15/jialiu23_2020-06-07-21-38-15/data --to=file://test
<--snip-->
info: Mirroring completed in 1m14.54s (78.66MB/s)

Success
Update image:  test:4.4.0-0.nightly-2020-06-07-075345
Mirror prefix: file://test

To upload local images to a registry, run:

    oc image mirror --from-dir=/home/installer3/mnt/2020-06-07-21-38-15/jialiu23_2020-06-07-21-38-15/data 'file://test:4.4.0-0.nightly-2020-06-07-075345*' REGISTRY/REPOSITORY
2. move disk cross firewall
3. mirror to private cluster from the local disk files via `oc adm release mirror`
$  /opt/mirror-to-disk/jialiu23_2020-06-07-21-38-15/oc adm release mirror --from-dir=/opt/mirror-to-disk/jialiu23_2020-06-07-21-38-15/data --from=file://test:4.4.0-0.nightly-2020-06-07-075345 --to=upshift-nointernet.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release --to-release-image=upshift-nointernet.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release:4.4.0-0.nightly-2020-06-07-075345
<--snip-->
Success
Update image:  upshift-nointernet.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release:4.4.0-0.nightly-2020-06-07-075345
Mirror prefix: upshift-nointernet.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release
4. Installation get successfull.

Comment 12 errata-xmlrpc 2020-06-17 22:26:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2445


Note You need to log in before you can comment on or make changes to this bug.