Bug 1953775 - The --apply-release-image-signature parameter is required for Restricted Network upgrades
Summary: The --apply-release-image-signature parameter is required for Restricted Netw...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Documentation
Version: 4.7
Hardware: s390x
OS: Linux
high
high
Target Milestone: ---
: 4.7.z
Assignee: Chinmayi Chandrasekhar
QA Contact: Johnny Liu
Latha S
URL:
Whiteboard:
Depends On:
Blocks: ocp-48-z-tracker
TreeView+ depends on / blocked
 
Reported: 2021-04-26 22:45 UTC by Philip Chan
Modified: 2022-02-06 06:49 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-01-05 10:34:10 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Philip Chan 2021-04-26 22:45:13 UTC
Document URL: 
https://docs.openshift.com/container-platform/4.6/installing/install_config/installing-restricted-networks-preparations.html

Section Number and Name: 
Mirroring the OpenShift Container Platform image repository - Step 3 - Bullet 2 - "If the local container registry is connected to the mirror host, take the following actions:"

Describe the issue: 
As the current documentation is written for Restricted network installation, the command to mirror the release images to the local registry shows the same command for both an install and upgrade.

The restricted network installation will succeed with the current mirror example:

$ oc adm release mirror -a ${LOCAL_SECRET_JSON}  \
     --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
     --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
     --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}

Using the same command to mirror a newer OCP release, such as 4.7.7 for example, will also succeed.  However, the command to upgrade will fail with "the image may not be safe to use".  

Command to upgrade:
$ oc adm upgrade --allow-upgrade-with-warnings --allow-explicit-upgrade --to-image ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}@sha256:9e0b967e4d50a518e4ffa680095a48c47a641c0e25bdc7799e0e04df2242871f
warning: The requested upgrade image is not one of the available updates.  You have used --allow-explicit-upgrade to the update to proceed anyway
Updating to release image bastion:5000/ocp4/openshift4@sha256:9e0b967e4d50a518e4ffa680095a48c47a641c0e25bdc7799e0e04df2242871f

Failure output:
$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.6     True        True          4m48s   Unable to apply bastion:5000/ocp4/openshift4@sha256:9e0b967e4d50a518e4ffa680095a48c47a641c0e25bdc7799e0e04df2242871f: the image may not be safe to use

The current workaround is to use --force in the upgrade.  But instead, if we were to use the parameter "--apply-release-image-signature" when performing the step for the mirror command, the upgrade will succeed without the need to use --force.

Suggestions for improvement: Add the required parameter "--apply-release-image-signature".

Additional information: We have performed this test on 4.7.x and 4.8.x Restricted Install upgrades and have consistently seen "--apply-release-image-signature" is required.

Comment 1 Philip Chan 2021-04-27 16:28:17 UTC
Changing severity from urgent to high as this is not a blocker bug.  We have the options to workaround the issue.

Comment 2 Silke Niemann 2021-05-28 13:32:10 UTC
@Phil The documentation you are referring to has been removed a few days ago and there's a a brand new description: 

https://docs.openshift.com/container-platform/4.7/installing/installing-mirroring-installation-images.html#installing-mirroring-installation-images

Does that solve the problem?

Comment 3 Philip Chan 2021-06-02 18:48:58 UTC
@Silke I followed the latest 4.7 documentation for mirroring.  As it stands now, the disconnected installation will work using the given steps.  However, if I were to perform an upgrade, for example upgrading from OCP 4.7.12 to 4.7.13, the problem still persists:

[root@bastion ~]# oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.12    True        True          29m     Unable to apply bastion:5000/ocp4/openshift4@sha256:4cf44e68413acad063203e1ee8982fd01d8b9c1f8643a5b31cd7ff341b3199cd: the image may not be safe to use

I had mirrored the version images to my internal registry without "--apply-release-image-signature" parameter.  

In my opinion, there are two ways we can update the documentation to resolve this:
1) We write a separate paragraph that explains how you can upgrade.
2) We add a small bullet or note that states for an upgrade, please add --apply-release-image-signature to the mirror command:

oc adm release mirror -a ${LOCAL_SECRET_JSON} --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE} --apply-release-image-signature

Open to other options too.

Thanks,
Phil

Comment 5 Johnny Liu 2021-12-20 02:25:55 UTC
Can someone paste the doc PR here?

In the present official doc, https://docs.openshift.com/container-platform/4.7/updating/updating-restricted-network-cluster.html, we already mentioned how to apply image signature config map.

"""
If the local container registry and the cluster are connected to the mirror host, directly push the release images to the local registry and apply the config map to the cluster by using following command:

$ oc adm release mirror -a ${LOCAL_SECRET_JSON} --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
  --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} --apply-release-image-signature
"""

"""
Creating the config map for image signature verification by using the oc CLI
"""

"""
Creating an image signature config map manually
"""

So personally I think the official doc looks good to me, no need change, anything missing?

Comment 7 Chinmayi Chandrasekhar 2021-12-31 06:11:46 UTC
PR: https://github.com/openshift/openshift-docs/pull/40174

Comment 8 Johnny Liu 2022-01-04 04:39:04 UTC
After reviewed https://github.com/openshift/openshift-docs/pull/40174, seem like no need any update, the reason mentioned in comment 5, for customers who are mirroring images and upgrading the cluster, the detailed steps about image signature is already described in https://docs.openshift.com/container-platform/4.7/updating/updating-restricted-network-cluster.html. https://docs.openshift.com/container-platform/4.6/installing/install_config/installing-restricted-networks-preparations.html is for a fresh install, but not upgrade.

Comment 9 Chinmayi Chandrasekhar 2022-01-05 10:33:03 UTC
(In reply to Chinmayi Chandrasekhar from comment #7)
> PR: https://github.com/openshift/openshift-docs/pull/40174

Wrong PR. The bug does not require any changes - can be closed.

Comment 10 Chinmayi Chandrasekhar 2022-01-05 10:33:42 UTC
(In reply to Chinmayi Chandrasekhar from comment #6)
> PR:
> https://github.com/openshift/openshift-docs/compare/main...chinmayi-
> chandrasekar:BZ1953775_add_required_parameter_for_network_upgrade?expand=1

Wrong PR. The bug does not require any changes - can be closed.


Note You need to log in before you can comment on or make changes to this bug.