Bug 2100166

Summary: heterogeneous arch: oc adm extract encodes arch specific release payload pullspec rather than the manifestlisted pullspec
Product: OpenShift Container Platform Reporter: Prashanth Sundararaman <psundara>
Component: ocAssignee: Arda Guclu <aguclu>
oc sub component: oc QA Contact: zhou ying <yinzhou>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: adistefa, mfojtik
Version: 4.11   
Target Milestone: ---   
Target Release: 4.12.0   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-01-17 19:50:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2104282    

Description Prashanth Sundararaman 2022-06-22 15:45:33 UTC
For payloads that are heterogeneous, the `oc adm extract` command encodes the sha of the arch specific paylaod in the openshift-install binary (as observed in binaries here: https://mirror.openshift.com/pub/openshift-v4/multi/clients/ocp-dev-preview/4.11.0-0.nightly-multi-2022-06-15-161850/)

This creates a UX problem as a user who wants to deploy say an arm control plane cluster from an x86 host will have to specify --filter-by-os=aarch64 to extract the installer which encodes the arm64 pullspec. Instead, oc should encode the installer with the pullspec of the top level manifest. this will allow a cluster of any architecture control plane to be installed by the openshift install binary.

Similarly, `oc adm release info` command generates the report with the pullspec of the architecture specific manifest which is not signed (eg: https://mirror.openshift.com/pub/openshift-v4/multi/clients/ocp-dev-preview/4.11.0-0.nightly-multi-2022-06-15-161850/amd64/release.txt). instead it should also display the pullspec of the top level manifest

Comment 1 Maciej Szulik 2022-06-24 12:58:14 UTC
Arda this will most likely require us to expand the ReleaseInfo (https://github.com/openshift/oc/blob/d4d4ab7359ae66dd7826f11519b51be1f4435312/pkg/cli/admin/release/info.go#L694) 
structure such that it has the information that the image is a manifestlist. We currently miss that kind of information there, since we always filter the image contents
to be able to read release information, for example here: https://github.com/openshift/oc/blob/d4d4ab7359ae66dd7826f11519b51be1f4435312/pkg/cli/admin/release/extract_tools.go#L366

That data is then being used here: https://github.com/openshift/oc/blob/master/pkg/cli/admin/release/extract_tools.go#L375 to inject into extracted binaries: https://github.com/openshift/oc/blob/d4d4ab7359ae66dd7826f11519b51be1f4435312/pkg/cli/admin/release/extract_tools.go#L578

Comment 2 Arda Guclu 2022-06-28 12:00:06 UTC
Hi Prashanth Sundararaman;

Just for the clarification, could you please add steps to reproduce the issue and provide more details about what are the actual and expected results?. Thanks.

Comment 3 Arda Guclu 2022-06-28 14:18:15 UTC
quay.io/openshift-release-dev/ocp-release@sha256:539fc5f98ea138395595fc72e9764aff3ad370803745b4d22db7df8e21d530db this is my example release image which is manifestlist.

$ oc adm release extract --from=quay.io/openshift-release-dev/ocp-release@sha256:539fc5f98ea138395595fc72e9764aff3ad370803745b4d22db7df8e21d530db --command=openshift-install
But this command uses arch based sha(sha256:1bff94be38f1f93bca31ce61706d568b1ab06da42fd6862037abb811c82ce0e3) and it is not correct.

If I'm not wrong, if release image is manifestlist image, we need to use "sha256:539fc5f98ea138395595fc72e9764aff3ad370803745b4d22db7df8e21d530db" as sha in this case.

Comment 4 Prashanth Sundararaman 2022-06-28 15:26:03 UTC
Yes the above example summarizes the issue. Similarly, it would also help if the release info encodes the manifestlist sha as well. right now it enodes the arch specific sha:

[psundara@fedora ~]$ oc adm release info quay.io/openshift-release-dev/ocp-release@sha256:1bff94be38f1f93bca31ce61706d568b1ab06da42fd6862037abb811c82ce0e3
W0628 08:17:21.608805 2277266 helpers.go:151] Defaulting of registry auth file to "${HOME}/.docker/config.json" is deprecated. The default will be switched to podman config locations in the future version.
Name:           4.11.0-0.nightly-multi-2022-06-14-102108
Digest:         sha256:1bff94be38f1f93bca31ce61706d568b1ab06da42fd6862037abb811c82ce0e3
Created:        2022-06-14T10:23:24Z
OS/Arch:        linux/amd64
Manifests:      590
Metadata files: 1

Pull From: quay.io/openshift-release-dev/ocp-release@sha256:1bff94be38f1f93bca31ce61706d568b1ab06da42fd6862037abb811c82ce0e3

Release Metadata:
  Version:  4.11.0-0.nightly-multi-2022-06-14-102108
  Upgrades: <none>
  Metadata:
    release.openshift.io/architecture: multi

Component Versions:
  kubernetes 1.24.0                
  machine-os 411.86.202206131434-0 Red Hat Enterprise Linux CoreOS

Comment 6 Prashanth Sundararaman 2022-07-05 16:46:17 UTC
Can we get a backport of this bug for 4.11 please? it is needed for the 4.11 heterogeneous tech preview

Comment 7 zhou ying 2022-07-06 02:14:08 UTC
for older version , when use `oc adm release info` will show the filtered by os image sha:
[root@localhost ~]#  oc version 
W0706 10:05:04.593381  159491 loader.go:221] Config not found: /root/kubeconfig
Client Version: 4.12.0-0.nightly-2022-07-05-010157
Kustomize Version: v4.5.4

[root@localhost ~]# oc adm release info quay.io/openshift-release-dev/ocp-release@sha256:5c7d3cd5cccc8c5874ab0b0b039b141f7c26cfc5dd43560d165976200f11aeaf
Warning: the default reading order of registry auth file will be changed from "${HOME}/.docker/config.json" to podman registry config locations in the future version of oc. "${HOME}/.docker/config.json" is deprecated, but can still be used for storing credentials as a fallback. See https://github.com/containers/image/blob/main/docs/containers-auth.json.5.md for the order of podman registry config locations.
Name:           4.11.0-0.nightly-multi-2022-07-05-234736
Digest:         sha256:f2ec4ce0888eba23d83fb4506dad7e61bf078c665d01fab13e140a0dc4042ca3
Created:        2022-07-05T23:50:03Z
OS/Arch:        linux/amd64
Manifests:      593
Metadata files: 1

Pull From: quay.io/openshift-release-dev/ocp-release@sha256:f2ec4ce0888eba23d83fb4506dad7e61bf078c665d01fab13e140a0dc4042ca3

Release Metadata:
  Version:  4.11.0-0.nightly-multi-2022-07-05-234736
  Upgrades: <none>
  Metadata:
    release.openshift.io/architecture: multi
...

with latest oc , will show the manifestlist image's sha value:
[root@localhost ~]# ./oc version 
W0706 10:05:09.908446  159503 loader.go:221] Config not found: /root/kubeconfig
Client Version: 4.12.0-0.nightly-2022-07-05-225149
Kustomize Version: v4.5.4


 ./oc adm release info quay.io/openshift-release-dev/ocp-release@sha256:5c7d3cd5cccc8c5874ab0b0b039b141f7c26cfc5dd43560d165976200f11aeaf
Warning: the default reading order of registry auth file will be changed from "${HOME}/.docker/config.json" to podman registry config locations in the future version of oc. "${HOME}/.docker/config.json" is deprecated, but can still be used for storing credentials as a fallback. See https://github.com/containers/image/blob/main/docs/containers-auth.json.5.md for the order of podman registry config locations.
Name:           4.11.0-0.nightly-multi-2022-07-05-234736
Digest:         sha256:5c7d3cd5cccc8c5874ab0b0b039b141f7c26cfc5dd43560d165976200f11aeaf
Created:        2022-07-05T23:50:03Z
OS/Arch:        linux/amd64
Manifests:      593
Metadata files: 1

Pull From: quay.io/openshift-release-dev/ocp-release@sha256:5c7d3cd5cccc8c5874ab0b0b039b141f7c26cfc5dd43560d165976200f11aeaf

Release Metadata:
  Version:  4.11.0-0.nightly-multi-2022-07-05-234736
  Upgrades: <none>
  Metadata:
    release.openshift.io/architecture: multi
....

Comment 8 Prashanth Sundararaman 2022-07-06 03:24:15 UTC
also verified that the installer extracted with this oc has the digest of the manifestlist:

[psundara@fedora oc]$ ./openshift-install version
./openshift-install 4.11.0-0.nightly-multi-2022-07-05-234736
built from commit b2e7be726e400022e71ef3b8bd01a2093e53bc5a
release image quay.io/openshift-release-dev/ocp-release@sha256:5c7d3cd5cccc8c5874ab0b0b039b141f7c26cfc5dd43560d165976200f11aeaf
release architecture amd64

Comment 11 errata-xmlrpc 2023-01-17 19:50:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7399