Bug 1878972 - oc adm release mirror removes the architecture information [NEEDINFO]
Summary: oc adm release mirror removes the architecture information
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oc
Version: 4.6
Hardware: Unspecified
OS: Unspecified
low
medium
Target Milestone: ---
: 4.7.0
Assignee: Jack Ottofaro
QA Contact: Johnny Liu
URL:
Whiteboard: LifecycleReset
: 1914712 (view as bug list)
Depends On:
Blocks: ocp-47-z-tracker 1907755
TreeView+ depends on / blocked
 
Reported: 2020-09-15 05:29 UTC by Frederic Giloux
Modified: 2021-02-24 15:18 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously when a release was mirrored to disk using 'oc adm release mirror' the manifest file names did not contain the architecture extension, e.g. "-x86_64". This did not align with our documentation and does not allow for mirroring multiple architectures to the same repository without having tag-name collisions. File names now contain the appropriate architecture extension.
Clone Of:
: 1907755 (view as bug list)
Environment:
Last Closed: 2021-02-24 15:18:03 UTC
Target Upstream Version:
Embargoed:
mfojtik: needinfo?


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift oc pull 646 0 None closed Bug 1878972: adm release mirror: add architecture information 2021-02-11 15:17:55 UTC
Github openshift oc pull 680 0 None closed Bug 1878972: pkg/cli/admin/release: Use '-', not '=', for untranslated arches 2021-02-11 15:17:54 UTC
Red Hat Knowledge Base (Solution) 5450081 0 None None None 2020-09-30 22:04:49 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:18:40 UTC

Description Frederic Giloux 2020-09-15 05:29:13 UTC
Description of problem:
When running the following command the architecture information is missing:
./oc adm release mirror -a ${LOCAL_SECRET_JSON} --to-dir=./removable-registry quay.io/openshift-release-dev/ocp-release:4.6.0-fc.5-x86_64
info: Mirroring 121 images to file://openshift/release ...
<dir>
[...]
 manifests:
      sha256:007637513a06eebb324561d234d64b9dc614adcb1c79b4efb3e9818bea09b108 -> 4.6.0-fc.5-ironic-inspector
      sha256:010737a642e6da90ac55edeb1aebe1e308dda607561cf04b2b9085d7277c0cd0 -> 4.6.0-fc.5-ironic-machine-os-downloader
      sha256:01a8e4ac77a3f534adfb8a3174b8d30ea63065aa5370f8f3f92446e46a8130a8 -> 4.6.0-fc.5-docker-registry
      sha256:0489c92a0845a2330062c4f10e45a5dcdc8b6d7982ea41c5b7e5485960943be9 -> 4.6.0-fc.5-pod
      sha256:058f695f28fa5cd43f98bbb4b187d3ce8d37f24771e43834bd5a119b4e3ff0a2 -> 4.6.0-fc.5-cli-artifacts
[...]

This can also be seen in the local directory:
$ ls removable-registry/mirror/v2/openshift/release/manifests/
4.6.0-fc.5                                                 sha256:007637513a06eebb324561d234d64b9dc614adcb1c79b4efb3e9818bea09b108
4.6.0-fc.5-aws-ebs-csi-driver                              sha256:010737a642e6da90ac55edeb1aebe1e308dda607561cf04b2b9085d7277c0cd0
4.6.0-fc.5-aws-ebs-csi-driver-operator                     sha256:01a8e4ac77a3f534adfb8a3174b8d30ea63065aa5370f8f3f92446e46a8130a8
4.6.0-fc.5-aws-machine-controllers                         sha256:0489c92a0845a2330062c4f10e45a5dcdc8b6d7982ea41c5b7e5485960943be9
[...]

Version-Release number of selected component (if applicable):
$ ./oc version
Client Version: 4.6.0-0-2020-07-31-080025

How reproducible:
Always

Steps to Reproduce:
1. Run the command above

Actual results:
Manifests without architecture

Expected results:
Manifests with architecture

Additional info:
This prevents among others the documented command to work for extracting the installation program:
$ oc adm -a ${LOCAL_SECRET_JSON} release extract --command=openshift-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}"
This works without -${ARCHITECTURE}
See also doc bug 1878830

Comment 2 Jack Ottofaro 2020-09-30 15:25:01 UTC
What leads you to believe that the mirror command is "removing" the architecture information? From my initial testing the mirror command is simply naming the manifest links using the info from the release and not removing anything. Although the release name now has the architecture added it does not appear information within the release has been tagged with the architecture.

Comment 3 W. Trevor King 2020-09-30 22:24:34 UTC
[1] talks about the coming pivot to by-digest pullspecs and the fact that we probably don't need to extract the installer locally anyway.  But extracting locally using a by-digest pullspec will work today.  But you're correct that the current:

$ git --no-pager log -1 --oneline
bea6d6bbfc (HEAD -> master, origin/master, origin/HEAD) Merge pull request #26004 from bobfuru/OSDOCS-1496-v2
$ git --no-pager grep 'release extract' modules/installation-mirror-repository.adoc 
modules/installation-mirror-repository.adoc:$ oc adm -a ${LOCAL_SECRET_JSON} release extract --command=openshift-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}"

does not match the current 'oc' behavior:

$ oc adm release mirror --from=quay.io/openshift-release-dev/ocp-release@sha256:4d048ae1274d11c49f9b7e70713a072315431598b2ddbb512aee4027c422fe3e --to-dir=mirror
$ ls -l mirror/v2/openshift/release/manifests/4.5.11
lrwxrwxrwx. 1 wking wking 71 Sep 30 11:00 mirror/v2/openshift/release/manifests/4.5.11 -> sha256:4d048ae1274d11c49f9b7e70713a072315431598b2ddbb512aee4027c422fe3e

We should update oc (maybe in [2]?) to use 4.5.11-x86_64 for that images and other generated files to align with the docs and to allow folks to mirror multiple architectures to the same repository without having tag-name collisions.

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1878830#c3
[2]: https://github.com/openshift/oc/blob/eadaf8954be1a1bc3a0cdf4db5e89fde3ac7e0f2/pkg/cli/admin/release/mirror.go#L441-L445

Comment 4 Frederic Giloux 2020-10-01 06:13:46 UTC
Thanks Trevor. It sounds good.

Comment 6 Jack Ottofaro 2020-10-23 18:50:52 UTC
I'm working on a higher priority task. Hence I'm adding UpcomingSprint now, and we'll revisit this in the next sprint.

Comment 7 Michal Fojtik 2020-10-31 10:12:07 UTC
This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. Additionally, you can add LifecycleFrozen into Keywords if you think this bug should never be marked as stale. Please consult with bug assignee before you do that.

Comment 9 Michal Fojtik 2020-11-05 14:12:14 UTC
The LifecycleStale keyword was removed because the bug got commented on recently.
The bug assignee was notified.

Comment 10 Jack Ottofaro 2020-11-12 16:09:26 UTC
I'm working on a higher priority task. Hence I'm adding UpcomingSprint now, and we'll revisit this in the next sprint.

Comment 12 Johnny Liu 2020-12-15 06:57:16 UTC
Verified this bug with 4.7.0-0.nightly-2020-12-14-165231, and PASS.

$ oc adm release mirror --from=registry.svc.ci.openshift.org/ocp/release:4.7.0-0.nightly-2020-12-14-165231 --to-dir=/tmp/OCP-30833-S8VrBb/OCP-30833_15002516/data --to=file://test
<--snip-->
12-15 13:25:47     manifests:
12-15 13:25:47       sha256:01e7ba6c28d07376c808042e4e52d3620244d8473e2caa1f62c43d518b89e292 -> 4.7.0-0.nightly-2020-12-14-165231-x86_64-machine-os-content
12-15 13:25:47       sha256:033eea61bb9749be54a2cbd143f17c7d09eb5552fbd12343953bd5db77a2512d -> 4.7.0-0.nightly-2020-12-14-165231-x86_64-prometheus-config-reloader
12-15 13:25:47       sha256:0472078a912eff2e686feeb861d2ce5ecf32c876e0d4ad4d3ee532a1b85fd120 -> 4.7.0-0.nightly-2020-12-14-165231-x86_64-installer-artifacts
12-15 13:25:47       sha256:0cb7208130966c77f7f274bd3b882c13725a8b54d5834ae58564f246190fedbc -> 4.7.0-0.nightly-2020-12-14-165231-x86_64-docker-builder
12-15 13:25:47       sha256:0cd5a1f9bada69434f4958690b7f2b074f8c84efe64bc4b4990f4596f13692bc -> 4.7.0-0.nightly-2020-12-14-165231-x86_64-cluster-kube-controller-manager-operator
12-15 13:25:47       sha256:0e35cfcc3e891bd552386473ae73e7fb50dcdd931bbdb0ffbae08f6dcdce0c97 -> 4.7.0-0.nightly-2020-12-14-165231-x86_64-cluster-openshift-apiserver-operator
12-15 13:25:47       sha256:11505541d093a7fa7ee994d29df6268477a2810d9ca0c6cdd489a19a086550eb -> 4.7.0-0.nightly-2020-12-14-165231-x86_64-cluster-policy-controller
12-15 13:25:47       sha256:11f88809be4a47b22702bd0b098b28b7bba3462bcd0efb81d0aa9fe2f64a5708 -> 4.7.0-0.nightly-2020-12-14-165231-x86_64-cluster-version-operator
12-15 13:25:47       sha256:194093dbe837378a8efcc83c4632a0aaeff66b9cbaa5f0db283182d43ecdfced -> 4.7.0-0.nightly-2020-12-14-165231-x86_64-cluster-etcd-operator
12-15 13:25:47       sha256:1a4bb65684fc02aa020d0a0bcffe4872709116488f40885e62ea19d60b189e03 -> 4.7.0-0.nightly-2020-12-14-165231-x86_64-mdns-publisher
12-15 13:25:47       sha256:1bb15ff72c9c2a0fa008b345c1e4cb37649d40d6454f8871ba0b98eef325eb6d -> 4.7.0-0.nightly-2020-12-14-165231-x86_64-installer
<--snip-->
12-15 13:26:34 Success
12-15 13:26:34 Update image:  test:4.7.0-0.nightly-2020-12-14-165231-x86_64
12-15 13:26:34 Mirror prefix: file://test
12-15 13:26:34 
12-15 13:26:34 To upload local images to a registry, run:
12-15 13:26:34 
12-15 13:26:34     oc image mirror --from-dir=/tmp/OCP-30833-S8VrBb/OCP-30833_15002516/data 'file://test:4.7.0-0.nightly-2020-12-14-165231-x86_64*' REGISTRY/REPOSITORY

Comment 14 W. Trevor King 2020-12-21 17:57:59 UTC
Pulling back to ASSIGNED, so we can address:

  error: unable to push manifest to local-registry.apps.pravind.redhat.com:5000/ocp4/openshift4:4.7.0-0.nightly-ppc64le-2020-12-21-131639=ppc64le-prometheus: invalid tag format

for arches that are not translated.

Comment 16 Johnny Liu 2021-01-06 02:48:30 UTC
Reproduce this issue with ppc64le arch build.

[root@preserve-jialiu-ansible ~]# oc version
Client Version: 4.7.0-0.nightly-2020-12-21-131655
Kubernetes Version: v1.19.0+d59ce34

[root@preserve-jialiu-ansible ~]# oc adm release mirror --from=quay.io/openshift-release-dev/ocp-release:4.7.0-fc.0-ppc64le --to-dir=/mnt/mirror-to-disk/4.7.0-fc.0-ppc64le --to=file://test
info: Mirroring 131 images to file://test ...
<dir>
  test
    blobs:
<--snip-->
    manifests:
      sha256:00b2e87c414d7477b6d97cf1ccab861f28955fe1f8c570a193fa0354bc4e1d2a -> 4.7.0-fc.0=ppc64le-pod
      sha256:01bf3c8cc69a90d057d7a1261b207060be94620a4f47d0e7a84aeef8aa1cdf33 -> 4.7.0-fc.0=ppc64le-kuryr-cni
      sha256:05a25bc0467a94214d6e5559b2d723ca37f21e2dda750086c645a68b2ab1e6a9 -> 4.7.0-fc.0=ppc64le-ovn-kubernetes
<--snip-->
sha256:260ca6b077b84b72c15ecfcfc5b83875a682cbd9a13466ec8aa0b84a25fe3a00 file://test:4.7.0-fc.0=ppc64le-aws-pod-identity-webhook
info: Mirroring completed in 52.94s (124.6MB/s)

Success
Update image:  test:4.7.0-fc.0=ppc64le
Mirror prefix: file://test

To upload local images to a registry, run:

    oc image mirror --from-dir=/mnt/mirror-to-disk/4.7.0-fc.0-ppc64le 'file://test:4.7.0-fc.0=ppc64le*' REGISTRY/REPOSITORY

Configmap signature file /mnt/mirror-to-disk/4.7.0-fc.0-ppc64le/config/signature-sha256-ec78b42fd0fb9a01.yaml created

[root@preserve-jialiu-ansible ~]# oc image mirror --from-dir=/mnt/mirror-to-disk/4.7.0-fc.0-ppc64le 'file://test:4.7.0-fc.0=ppc64le*' upshift-nointernet.mirror-registry.qe.devcluster.openshift.com:5000/jialiu/test1
error: "ppc64le*" is not a valid image reference: invalid reference format


Verified this bug with 4.7.0-0.nightly-2021-01-05-220959, and PASS.

[root@preserve-jialiu-ansible ~]# oc version
Client Version: 4.7.0-0.nightly-2021-01-05-220959
Kubernetes Version: v1.19.0+d59ce34

[root@preserve-jialiu-ansible ~]# oc adm release mirror --from=quay.io/openshift-release-dev/ocp-release:4.7.0-fc.0-ppc64le --to-dir=/mnt/mirror-to-disk/4.7.0-fc.0-ppc64le-1 --to=file://test
info: Mirroring 131 images to file://test ...
<dir>
  test
    blobs:
<--snip-->
    manifests:
      sha256:00b2e87c414d7477b6d97cf1ccab861f28955fe1f8c570a193fa0354bc4e1d2a -> 4.7.0-fc.0-ppc64le-pod
      sha256:01bf3c8cc69a90d057d7a1261b207060be94620a4f47d0e7a84aeef8aa1cdf33 -> 4.7.0-fc.0-ppc64le-kuryr-cni
      sha256:05a25bc0467a94214d6e5559b2d723ca37f21e2dda750086c645a68b2ab1e6a9 -> 4.7.0-fc.0-ppc64le-ovn-kubernetes
      sha256:072f46f858463fb804c3cafbf7110520e9338aa085096dd269d70a9c2d121531 -> 4.7.0-fc.0-ppc64le-oauth-apiserver
<--snip-->
sha256:87d2458ab65f0e80da03011f8935fd6a18c67d8f59e540c550fd79bd595155ee file://test:4.7.0-fc.0-ppc64le-baremetal-machine-controllers
sha256:68cfc5f7f5518b3cabcd95466d2f7a299159f1f71f081adbbca3305374ec6d96 file://test:4.7.0-fc.0-ppc64le-thanos
info: Mirroring completed in 55.02s (119.9MB/s)

Success
Update image:  test:4.7.0-fc.0-ppc64le
Mirror prefix: file://test

To upload local images to a registry, run:

    oc image mirror --from-dir=/mnt/mirror-to-disk/4.7.0-fc.0-ppc64le-1 'file://test:4.7.0-fc.0-ppc64le*' REGISTRY/REPOSITORY

Configmap signature file /mnt/mirror-to-disk/4.7.0-fc.0-ppc64le-1/config/signature-sha256-ec78b42fd0fb9a01.yaml created

[root@preserve-jialiu-ansible ~]# oc image mirror --from-dir=/mnt/mirror-to-disk/4.7.0-fc.0-ppc64le-1 'file://test:4.7.0-fc.0-ppc64le*' upshift-nointernet.mirror-registry.qe.devcluster.openshift.com:5000/jialiu/test2
upshift-nointernet.mirror-registry.qe.devcluster.openshift.com:5000/
  jialiu/test2
    blobs:
<--snip-->
sha256:4cf42d9bc1b269cf7dc3bdc6a231e46b8f6cff7f3d3b8d1b1b3534eb54bf82da upshift-nointernet.mirror-registry.qe.devcluster.openshift.com:5000/jialiu/test2:4.7.0-fc.0-ppc64le-openstack-machine-controllers
sha256:5c50720908ac43ae5156bbf30d21cd1a2e0fcbdce71e35858908163ae1c80bcf upshift-nointernet.mirror-registry.qe.devcluster.openshift.com:5000/jialiu/test2:4.7.0-fc.0-ppc64le-cluster-update-keys
info: Mirroring completed in 54.53s (121MB/s)

Comment 17 W. Trevor King 2021-01-11 18:20:51 UTC
*** Bug 1914712 has been marked as a duplicate of this bug. ***

Comment 19 errata-xmlrpc 2021-02-24 15:18:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.