Bug 1837404

Summary: "oc adm release mirror" should create the configmap yaml for image signatures for use by restricted-network clusters
Product: OpenShift Container Platform Reporter: Lalatendu Mohanty <lmohanty>
Component: ocAssignee: Jack Ottofaro <jack.ottofaro>
Status: CLOSED ERRATA QA Contact: Johnny Liu <jialiu>
Severity: high Docs Contact:
Priority: high    
Version: 4.4CC: aos-bugs, jialiu, jokerman, mfojtik, wking, yinzhou
Target Milestone: ---Keywords: Upgrades
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: Enhancement
Doc Text:
Feature: Extends `oc adm release mirror ...` command to also create and apply ConfigMap manifests containing the release image signature which the cluster-version operator can use to verify the mirrored release. Reason: Currently a cluster upgrade can be accomplished on a cluster that does not have an active connection to the internet. However manual steps are required to create a ConfigMap containing the signature data required for update image verification. Result: This enhancement will automatically create the ConfigMap, so the user doesn't have to think about manual steps.
Story Points: ---
Clone Of:
: 1837675 (view as bug list) Environment:
Last Closed: 2020-07-13 17:40:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1837675    

Description Lalatendu Mohanty 2020-05-19 12:51:20 UTC
Description of problem:

We have implemented [2] an enhancement request [1] which creates the configMap manifests containing the release image signature which the cluster-version operator could use to verify the mirrored release for disconnected clusters. 

We have a telecommunication service provider customer is going to be using disconnected OpenShift 4.4 clusters. 

Knowing this we had a discussion about what if anything we could/should backport pre-emptively to short circuit future pain, and avoid introducing bad habits like the use of --force as they do in 4.3 due to the lack of signature mirroring.

Hence we need to backport [2] to 4.4.z

[1] https://github.com/openshift/enhancements/pull/283/
[2] https://github.com/openshift/oc/pull/343 and the blocker bug fix PR which fixes the bug which was introduced in previous PR i.e. https://github.com/openshift/oc/pull/392

Comment 2 W. Trevor King 2020-05-19 15:34:47 UTC
Bug 1825565's verification [1] was just about "is the old usage still broken?", not about "does the new feature work?".  I think we need this feature series to be a completely separate chain of bugs that motivate the feature backport.

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1825565#c6

Comment 3 Scott Dodson 2020-05-19 19:11:32 UTC
We'll just use this to verify the functionality of https://issues.redhat.com/browse/OTA-121 so that it may be backported.

Comment 4 W. Trevor King 2020-05-19 19:22:21 UTC
Updated the GitHub links to drop the 4.4 backport and reference the original 4.5/master PR and the fixup from bug 1825565.

Comment 5 W. Trevor King 2020-05-19 19:23:41 UTC
Moving back to MODIFIED so ART can sweep us into ON_QA and attach us to an errata (both 4.5/master changes have already landed in 4.5, so this is just about catching the errata association up with reality).

Comment 8 Jack Ottofaro 2020-05-20 21:28:05 UTC
Hi jialiu, not sure what's left to do to verify this bug but if you it'd be great if it could be done ASAP and much appreciated. We really only opened this bug to hang a backport bug off if and we're wanting to press ahead with that backport. Thanks.

Comment 9 Johnny Liu 2020-05-21 07:44:44 UTC
Verified this bug with 4.5.0-0.nightly-2020-05-19-011623, and PASS.

In this verification, I did the following scenario testing:
1. mirror release image to private registry and applying signature configmap directly to the target cluster, then update the air-gapped cluster without --force option, succeed.
$ oc adm release mirror --from=registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-05-19-011623 --to=upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release --to-release-image=upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release:4.5.0-0.nightly-2020-05-19-011623 --apply-release-image-signature
configmap/sha256-106ade2fb4f434a4ddfaefa35e0f7e77f211e89ec40a5121a8d4c5e8f34340ac created

2. mirror release image to private registry, and saving signature configmap to a separate directoy, and apply it to the target cluster, then update the air-gapped cluster without --force option, succeed.
$ oc adm release mirror --from=registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-05-19-011623 --to=upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release --to-release-image=upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release:4.5.0-0.nightly-2020-05-19-011623 --release-image-signature-to-dir=/home/installer-auto/workspace/installer-auto-test@3/assets_dir/OCP-27986_145579
To apply signature configmaps use 'oc apply' on files found in /home/installer-auto/workspace/installer-auto-test@3/assets_dir/OCP-27986_145579

Configmap signature file /home/installer-auto/workspace/installer-auto-test@3/assets_dir/OCP-27986_145579/signature-sha256-106ade2fb4f434a4.yaml created

3. mirror release image to local disk together with signature configmap yaml file, move cross firewall, upload disk files to private registry, and apply it to the target cluster, then update the air-gapped cluster without --force option, succeed.
$ oc adm release mirror --from=registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-05-19-011623 --to-dir=/tmp/OCP-30833-8CmRoJ/OCP-30833_21021034/data --to=file://test
To upload local images to a registry, run:

    oc image mirror --from-dir=/tmp/OCP-30833-8CmRoJ/OCP-30833_21021034/data 'file://test:4.5.0-0.nightly-2020-05-19-011623*' REGISTRY/REPOSITORY

info: Mirroring completed in 1m8.13s (88.79MB/s)
Configmap signature file /tmp/OCP-30833-8CmRoJ/OCP-30833_21021034/data/config/signature-sha256-106ade2fb4f434a4.yaml created

<move crosee firewall>

$ /opt/mirror-to-disk/OCP-30833_21021034/oc image mirror --from-dir=/opt/mirror-to-disk/OCP-30833_21021034/data 'file://test:4.5.0-0.nightly-2020-05-19-011623*' upshift-nointernet.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release
sha256:430cf1bc3b7506e935578db1a9a193f21c12ffa23534ece6f5d826d02112d061 upshift-nointernet.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release:4.5.0-0.nightly-2020-05-19-011623-haproxy-router
info: Mirroring completed in 47.95s (126.2MB/s)

$ KUBECONFIG=/opt/mirror-to-disk/OCP-30833_21021034/kubeconfig oc apply -f /opt/mirror-to-disk/OCP-30833_21021034/data/config/signature-*.yaml --overwrite=true
configmap/sha256-106ade2fb4f434a4ddfaefa35e0f7e77f211e89ec40a5121a8d4c5e8f34340ac configured

Comment 10 errata-xmlrpc 2020-07-13 17:40:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.