Bug 1828214
| Summary: | Using disconnected network env, "oc adm upgrade" does not succeed without "--force" option | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Hideshi Fukumoto <hfukumot> |
| Component: | Documentation | Assignee: | Lalatendu Mohanty <lmohanty> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Xiaoli Tian <xtian> |
| Severity: | medium | Docs Contact: | Vikram Goyal <vigoyal> |
| Priority: | medium | ||
| Version: | 4.3.z | CC: | aos-bugs, jokerman, lmohanty, scuppett, wking |
| Target Milestone: | --- | Keywords: | Upgrades, ZStream |
| Target Release: | 4.3.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-08-29 03:50:50 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Setting target release to current development version (4.5) for investigation. Where fixes (if any) are required/requested for prior versions, cloned BZs will be created when appropriate. > Message: Unable to retrieve available updates: Get https://api.openshift.com/api/upgrades_info/v1/graph?arch=amd64&channel=stable-4.3&id=aba67c32-b4a6-4a24-8203-d182e2bd98e9&version=4.3.9: dial tcp: lookup api.openshift.com on 10.108.198.100:53: server misbehaving "server misbehaving" sounds like a buggy DNS server, but yeah, you shouldn't expect to be able to hit the external api.openshift.com from a restricted-network cluster unless you have a proxy set up or some such. To tell the cluster-version operator (CVO) to not bother trying, clear ClusterVersion's spec.channel [1]. And unless you have a local Cincinnati service (more on that in bug 1825084), 'oc adm upgrade' is only going to work with --to-image. > == test case-2 == Same as the first case; without a reachable Cincinnati service, the CVO will not have a list of available updates for you to choose from. > == test case-3 == > ... > Unable to apply bastion-host:50000/ocp4/openshift4:4.3.10-x86_64: the image may not be safe to use We don't do a good job of exposing the details in the ClusterVersion status, but this is probably "CVO cannot reach the external signature stores to fetch your release's signature". Bug 1783054 should have you covered there in recent 4.3.z, but we're still working on docs around the procedure. > = test case-4 == Yeah, --force works around the lack of signatures in the meantime, but obviously that's not ideal (it also waives several other checks the CVO applies to decide if the update is safe). Hopefully we'll have docs for bug 1783054 up soon, and we can use this bug to track those docs. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1827378#c4 Docs are in flight with [1], but are still early stages (no feedback from actual docs maintainers yet). [1]: https://github.com/openshift/openshift-docs/pull/21993 All the product-side changes have been released. The docs PR linked from comment 6 is getting a lot of feedback, including within the past 24h. Hopefully it lands soon, but it's had sufficient polish that the folks should be able to work off it now before it lands. Is there something that you feel the current state of that PR leaves unclear? Bug 1783054, with the product fix, links [1], the fixing PR (landed in 4.3.12). With that change, the cluster-version operator (CVO) is able to load a release image signature from a local ConfigMap, which allows the CVO to verify the release when the usual signature mirrors are not available (which is the case for restricted-network clusters). The in-flight docs PR (this bug) walk you through how to create that release image signature ConfigMap, and you should be able to follow those docs to update your 4.3.12 or later cluster. I don't have an estimate for landing the docs; the docs maintainers will land the PR when they are happy with it. But I think the remaining issues are with structure/polish, and the content of the PR should already be sufficient for folks looking to create release image signature ConfigMaps before the docs PR lands. [1]: https://github.com/openshift/cluster-version-operator/pull/340 Lala called for backporting the docs to 4.4 [1]. Not clear to me why we didn't pick them all the way back to 4.3, since the CVO for 4.3.12 and later support signatures from ConfigMaps. I'm fine leaving this bug open until we have clarity on the plan for 4.3 docs. [1]: https://github.com/openshift/openshift-docs/pull/21993#issuecomment-653044417 > Hi, Could you please let us know your comment/answer ? Do you need the docs to be back ported to 4.3 as well? You should be able to use [1] for 4.3.12 or later builds to do upgrade in disconnected clusters. [1] https://docs.openshift.com/container-platform/4.4/updating/updating-restricted-network-cluster.html#update-configuring-image-signature Thanks Hideshi. I am going to close this bug as we do not have anything to fix here. |
Description of problem: When we deployed OCP 4.3 cluster in the disconnected network environment, CLI command "oc adm upgrade" failed. Note: In this case, the OCP cluster can not access to Internet (no proxy), however it can access to the mirror registry on the bastion host, which has access the internet (refer to the doc[1]). So, we needed the following additional options to "oc adm upgrade" to upgrade the cluster: "--to-image", "--allow-explicit-upgrade" and "--force" $ oc adm upgrade --to-image {local_registry}/{local_repository}:{ocp_release} --allow-explicit-upgrade --force ex) $ oc adm upgrade --to-image bastion-host:50000/ocp4/openshift4:4.3.10-x86_64 --allow-explicit-upgrade --force --to-image: Specify an image to upgrade --allow-explicit-upgrade: Upgrade even if the upgrade target is not listed in the available versions list --force: Do upgrade if signature cannot be gotten from internet If this is the limitation on OCP4.3, then Red Hat should provide the manual document explained how to upgrade the cluster. Version-Release number of selected component (if applicable): OCP 4.3.9 How reproducible: Always Steps to Reproduce: (when we confirmed the following upgrade verification test, the upgrading from 4.3.9 to 4.3.10 were available) 1. Preparing the private registries in order to installing OCP under the disconnected network environment. (refer to the manual page[1]) 2. Install OCP 4.3.9 in the disconnected networking env. 3. Upgrade the Cluster using the following CLI command: (refer to github[2]) == test case-1 == $ oc adm upgrade Cluster version is 4.3.9 warning: Cannot display available updates: Reason: RemoteFailed Message: Unable to retrieve available updates: Get https://api.openshift.com/api/upgrades_info/v1/graph?arch=amd64&channel=stable-4.3&id=aba67c32-b4a6-4a24-8203-d182e2bd98e9&version=4.3.9: dial tcp: lookup api.openshift.com on 10.108.198.100:53: server misbehaving == test case-2 == $ oc adm upgrade --to-image bastion-host:50000/ocp4/openshift4:4.3.10-x86_64 error: The requested upgrade image is not one of the available updates, you must pass --allow-explicit-upgrade to continue == test case-3 == $ oc adm upgrade --to-image bastion-host:50000/ocp4/openshift4:4.3.10-x86_64 --allow-explicit-upgrade Updating to release image bastion-host:50000/ocp4/openshift4:4.3.10-x86_64 In this case, we can find out the following error message on the Web Console: "Administration" -> "Cluster Settings" -> "Cluster Operators" Info alert:Cluster update in progress. Unable to apply bastion-host:50000/ocp4/openshift4:4.3.10-x86_64: the image may not be safe to use == test case-4 == $ oc adm upgrade --to-image bastion-host:50000/ocp4/openshift4:4.3.10-x86_64 --allow-explicit-upgrade --force Updating to release image bastion-host:50000/ocp4/openshift4:4.3.10-x86_64 In this case (with "--force" option), the upgrade was successful. Actual results: See above. Expected results: Without any error message Additional info: [1] https://docs.openshift.com/container-platform/4.3/installing/install_config/installing-restricted-networks-preparations.html [2] https://github.com/openshift/openshift-docs/blob/5de3002649cf4bb9b96b170f78bdd514cc322c24/modules/update-restricted-network-cli.adoc