Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2086287

Summary: Unable to upgrade from 4.10.14 to 4.11 nightly build
Product: OpenShift Container Platform Reporter: Polina Rabinovich <prabinov>
Component: Cluster Version OperatorAssignee: Over the Air Updates <aos-team-ota>
Status: CLOSED DUPLICATE QA Contact: liujia <jiajliu>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.10CC: aos-team-ota, mshitrit, msluiter, omichael, wking, yanyang
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-16 21:55:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
automatic clear none

Description Polina Rabinovich 2022-05-15 10:50:52 UTC
Created attachment 1879829 [details]
automatic clear

Description of problem:

Unable to upgrade from 4.10.14 to 4.11: 

I investigate upgrade process and I've got unexpected result:
After running this command:

oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:60a93f1ca1a0e3db1916b201ee2c846f70ba6991cdcdac96b12e740574c9daa1 --allow-explicit-upgrade


I get this message, but nothing happens:

warning: The requested upgrade image is not one of the available updates.You have used --allow-explicit-upgrade for the update to proceed anyway
Updating to release image registry.ci.openshift.org/ocp/release@sha256:60a93f1ca1a0e3db1916b201ee2c846f70ba6991cdcdac96b12e740574c9daa1


After some time I see an automatic clear and it returns back to 4.10.14

NAME	  VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.14   True        True          11s     Working towards 4.10.14: 668 of 771 done (86% complete)

[kni@provisionhost-0-0 ~]$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.14   True        False         41m     Cluster version is 4.10.14

----------------------
Versions:
from: Cluster version is 4.10.14
to: 4.11.0-0.nightly-2022-05-14-193620
----------------------

How reproducible:
100%
----------------------
Steps to Reproduce:
1.oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:60a93f1ca1a0e3db1916b201ee2c846f70ba6991cdcdac96b12e740574c9daa1 --allow-explicit-upgrade

----------------------
Actual results:
Upgrade doesn't work 
----------------------
Expected results:
Cluster successfully upgraded 
----------------------
Additional info:
see comment below

Comment 2 Yang Yang 2022-05-16 01:36:21 UTC
The target image is unsigned. 

# curl --silent --output /dev/null --write-o%{http_code} https://mirror2.openshift.com/pub/openshift-v4/signatures/openshift/release/sha256=60a93f1ca1a0e3db1916b201ee2c846f70ba6991cdcdac96b12e740574c9daa1/signature-1
404

So we have to use force upgrade to skip the precondition validation.

oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:60a93f1ca1a0e3db1916b201ee2c846f70ba6991cdcdac96b12e740574c9daa1 --allow-explicit-upgrade --force

Comment 3 Polina Rabinovich 2022-05-16 06:50:04 UTC
I tried to run upgrade to signed image (I got status: 200):

image - registry.ci.openshift.org/ocp/release:4.11.0-0.nightly-2022-05-11-054135

[kni@provisionhost-0-0 ~]$ curl -s https://mirror2.openshift.com/pub/openshift-v4/signatures/openshift/release/sha256=5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f107/signature-1 -o /dev/null -w "status: %{http_code}\n"
status: 200

However, upgrade doesn't work, nothing happens..

[kni@provisionhost-0-0 ~]$ oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f107 --allow-explicit-upgrade
warning: The requested upgrade image is not one of the available updates.You have used --allow-explicit-upgrade for the update to proceed anyway
Updating to release image registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f107

I got this message:
[kni@provisionhost-0-0 ~]$ oc adm upgrade
Cluster version is 4.10.13

Upstream: https://registry.ci.openshift.org/ocp/
Channel: stable-4.10
warning: Cannot display available updates:
  Reason: ResponseFailed
  Message: Unable to retrieve available updates: unexpected HTTP status: 404 

[kni@provisionhost-0-0 ~]$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.13   True        False         21h     Cluster version is 4.10.13

Any thoughts are much appreciated.

Comment 4 Yang Yang 2022-05-16 07:27:09 UTC
It crosses https://bugzilla.redhat.com/show_bug.cgi?id=2083370. Upgrade from 4.10.0-0.nightly-2022-05-13-065238 to 4.11 would be fine.

Comment 5 Polina Rabinovich 2022-05-16 07:33:05 UTC
ok, I'll check. Thanks!

Comment 6 Polina Rabinovich 2022-05-16 07:35:12 UTC
without force, right?

Comment 7 Yang Yang 2022-05-16 07:52:34 UTC
(In reply to Polina Rabinovich from comment #6)
> without force, right?

Right. In the case that you upgrade without force, the cluster needs to start on 4.10.0-0.nightly-2022-05-13-065238 and later. Upgrade with force would not run into https://bugzilla.redhat.com/show_bug.cgi?id=2083370.

Comment 8 W. Trevor King 2022-05-16 21:55:34 UTC
I'm marking this as a dup of bug 2083370, but if you can reproduce with 4.10.15 or later, feel free to re-open.

*** This bug has been marked as a duplicate of bug 2083370 ***