1706245 – upgrade won't allow user to run upgrade again if a user forgot to use --force=true the first time.

Bug 1706245 - upgrade won't allow user to run upgrade again if a user forgot to use --force=true the first time.

Summary: upgrade won't allow user to run upgrade again if a user forgot to use --force...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Master
Sub Component:
Version:	4.1.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	4.1.0
Assignee:	Mike Dame
QA Contact:	Peter Ruan
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-05-03 21:30 UTC by Peter Ruan
Modified:	2019-06-19 06:45 UTC (History)
CC List:	8 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-06-19 06:45:34 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2019:1382	0	None	None	None	2019-06-19 06:45:44 UTC

Description Peter Ruan 2019-05-03 21:30:43 UTC

Description of problem:
  If a user didn't use --force option initially in the `oc adm upgrade` command, the upgrade will fail and no upgrade is done.  If the user try the same command again with the `--force=true` option added, the client will complain that the cluster is already at the final target version even though the actual upgrade process was never started due to the lack of the `--force` option when run initially.


Version-Release number of the following components:
rpm -q openshift-ansible
rpm -q ansible
ansible --version

How reproducible:
always 

Steps to Reproduce:
1. run upgrade w/o --force
2. run upgrade again with --force
3.

Actual results:
pruan@homer ~ $ oc adm upgrade --v=8 --to-image=registry.svc.ci.openshift.org/ocp/release:4.1.0-0.nightly-2019-05-03-152614                                    [ruby-2.6.0]
I0503 11:25:34.659237   19960 loader.go:359] Config loaded from file /home/pruan/.kube/config
I0503 11:25:34.659765   19960 round_trippers.go:416] GET https://api.qe-pruan-532019.qe.devcluster.openshift.com:6443/apis/config.openshift.io/v1/clusterversions/version
I0503 11:25:34.660630   19960 round_trippers.go:423] Request Headers:
I0503 11:25:34.660661   19960 round_trippers.go:426]     Accept: application/json, */*
I0503 11:25:34.660688   19960 round_trippers.go:426]     User-Agent: oc/v1.13.4+dd90b09 (linux/amd64) kubernetes/dd90b09
I0503 11:25:34.660713   19960 round_trippers.go:426]     Authorization: Bearer tq2SgNwSXEsj7Q5t1YH5j_c5TH8ThpjSri_yc1VwJBM
I0503 11:25:35.020146   19960 round_trippers.go:441] Response Status: 200 OK in 359 milliseconds
I0503 11:25:35.020249   19960 round_trippers.go:444] Response Headers:
I0503 11:25:35.020281   19960 round_trippers.go:447]     Content-Type: application/json
I0503 11:25:35.020311   19960 round_trippers.go:447]     Content-Length: 1675
I0503 11:25:35.020339   19960 round_trippers.go:447]     Date: Fri, 03 May 2019 18:25:34 GMT
I0503 11:25:35.020367   19960 round_trippers.go:447]     Audit-Id: 057244f1-a60b-4997-8d12-6393a3e43b3b
I0503 11:25:35.020396   19960 round_trippers.go:447]     Cache-Control: no-store
I0503 11:25:35.020448   19960 request.go:942] Response Body: {"apiVersion":"config.openshift.io/v1","kind":"ClusterVersion","metadata":{"creationTimestamp":"2019-05-03T16:55:28Z","generation":2,"name":"version","resourceVersion":"30325","selfLink":"/apis/config.openshift.io/v1/clusterversions/version","uid":"4133cde2-6dc4-11e9-b563-0eff287b3e36"},"spec":{"channel":"stable-4.1","clusterID":"908e2825-ce71-4014-aac0-f3fecc63b535","upstream":"https://openshift-release.svc.ci.openshift.org/graph"},"status":{"availableUpdates":[{"force":false,"image":"registry.svc.ci.openshift.org/ocp/release@sha256:e71c3136ce6df1bfac4156a908668cc0d59232d3f14da66f3cf8645e1fa40fdb","version":"4.1.0-0.nightly-2019-05-03-152614"}],"conditions":[{"lastTransitionTime":"2019-05-03T17:08:29Z","message":"Done applying 4.1.0-0.nightly-2019-05-03-121956","status":"True","type":"Available"},{"lastTransitionTime":"2019-05-03T16:59:44Z","status":"False","type":"Failing"},{"lastTransitionTime":"2019-05-03T17:08:29Z","message":"Cluster version is 4.1.0-0.nightly-2019-05-03-121956","status":"False","type":" [truncated 651 chars]
I0503 11:25:35.021557   19960 request.go:942] Request Body: {"kind":"ClusterVersion","apiVersion":"config.openshift.io/v1","metadata":{"name":"version","selfLink":"/apis/config.openshift.io/v1/clusterversions/version","uid":"4133cde2-6dc4-11e9-b563-0eff287b3e36","resourceVersion":"30325","generation":2,"creationTimestamp":"2019-05-03T16:55:28Z"},"spec":{"clusterID":"908e2825-ce71-4014-aac0-f3fecc63b535","desiredUpdate":{"version":"","image":"registry.svc.ci.openshift.org/ocp/release:4.1.0-0.nightly-2019-05-03-152614","force":false},"upstream":"https://openshift-release.svc.ci.openshift.org/graph","channel":"stable-4.1"},"status":{"desired":{"version":"4.1.0-0.nightly-2019-05-03-121956","image":"registry.svc.ci.openshift.org/ocp/release@sha256:2cfbcb0c9e6a4b62f2b915c038cb68d96e039bf869150aeaf12fda1ea554d051","force":false},"history":[{"state":"Completed","startedTime":"2019-05-03T16:55:42Z","completionTime":"2019-05-03T17:08:29Z","version":"4.1.0-0.nightly-2019-05-03-121956","image":"registry.svc.ci.openshift.org/ocp/release@sha256:2cfbcb0c9e6a4b62f2b915c038cb68d96e039 [truncated 782 chars]
I0503 11:25:35.022345   19960 round_trippers.go:416] PUT https://api.qe-pruan-532019.qe.devcluster.openshift.com:6443/apis/config.openshift.io/v1/clusterversions/version
I0503 11:25:35.022383   19960 round_trippers.go:423] Request Headers:
I0503 11:25:35.022413   19960 round_trippers.go:426]     Content-Type: application/json
I0503 11:25:35.022442   19960 round_trippers.go:426]     Accept: application/json, */*
I0503 11:25:35.022471   19960 round_trippers.go:426]     User-Agent: oc/v1.13.4+dd90b09 (linux/amd64) kubernetes/dd90b09
I0503 11:25:35.022499   19960 round_trippers.go:426]     Authorization: Bearer tq2SgNwSXEsj7Q5t1YH5j_c5TH8ThpjSri_yc1VwJBM
I0503 11:25:35.096076   19960 round_trippers.go:441] Response Status: 200 OK in 73 milliseconds
I0503 11:25:35.096187   19960 round_trippers.go:444] Response Headers:
I0503 11:25:35.096219   19960 round_trippers.go:447]     Audit-Id: 23ecac10-2d0c-49fe-bb10-61b38b040d53
I0503 11:25:35.096248   19960 round_trippers.go:447]     Cache-Control: no-store
I0503 11:25:35.096276   19960 round_trippers.go:447]     Content-Type: application/json
I0503 11:25:35.096305   19960 round_trippers.go:447]     Content-Length: 1806
I0503 11:25:35.096333   19960 round_trippers.go:447]     Date: Fri, 03 May 2019 18:25:35 GMT
I0503 11:25:35.096387   19960 request.go:942] Response Body: {"apiVersion":"config.openshift.io/v1","kind":"ClusterVersion","metadata":{"creationTimestamp":"2019-05-03T16:55:28Z","generation":3,"name":"version","resourceVersion":"34473","selfLink":"/apis/config.openshift.io/v1/clusterversions/version","uid":"4133cde2-6dc4-11e9-b563-0eff287b3e36"},"spec":{"channel":"stable-4.1","clusterID":"908e2825-ce71-4014-aac0-f3fecc63b535","desiredUpdate":{"force":false,"image":"registry.svc.ci.openshift.org/ocp/release:4.1.0-0.nightly-2019-05-03-152614","version":""},"upstream":"https://openshift-release.svc.ci.openshift.org/graph"},"status":{"availableUpdates":[{"force":false,"image":"registry.svc.ci.openshift.org/ocp/release@sha256:e71c3136ce6df1bfac4156a908668cc0d59232d3f14da66f3cf8645e1fa40fdb","version":"4.1.0-0.nightly-2019-05-03-152614"}],"conditions":[{"lastTransitionTime":"2019-05-03T17:08:29Z","message":"Done applying 4.1.0-0.nightly-2019-05-03-121956","status":"True","type":"Available"},{"lastTransitionTime":"2019-05-03T16:59:44Z","status":"False","type":"Failing"},{"la [truncated 782 chars]
Updating to release image registry.svc.ci.openshift.org/ocp/release:4.1.0-0.nightly-2019-05-03-152614
pruan@homer ~ $ oc adm upgrade --force=true --v=8 --to-image=registry.svc.ci.openshift.org/ocp/release:4.1.0-0.nightly-2019-05-03-152614                       [ruby-2.6.0]
I0503 11:26:21.037517   20115 loader.go:359] Config loaded from file /home/pruan/.kube/config
I0503 11:26:21.041682   20115 round_trippers.go:416] GET https://api.qe-pruan-532019.qe.devcluster.openshift.com:6443/apis/config.openshift.io/v1/clusterversions/version
I0503 11:26:21.046093   20115 round_trippers.go:423] Request Headers:
I0503 11:26:21.046132   20115 round_trippers.go:426]     Accept: application/json, */*
I0503 11:26:21.046161   20115 round_trippers.go:426]     User-Agent: oc/v1.13.4+dd90b09 (linux/amd64) kubernetes/dd90b09
I0503 11:26:21.046188   20115 round_trippers.go:426]     Authorization: Bearer tq2SgNwSXEsj7Q5t1YH5j_c5TH8ThpjSri_yc1VwJBM
I0503 11:26:21.427121   20115 round_trippers.go:441] Response Status: 200 OK in 380 milliseconds
I0503 11:26:21.427220   20115 round_trippers.go:444] Response Headers:
I0503 11:26:21.427250   20115 round_trippers.go:447]     Content-Length: 2143
I0503 11:26:21.427278   20115 round_trippers.go:447]     Date: Fri, 03 May 2019 18:26:21 GMT
I0503 11:26:21.427304   20115 round_trippers.go:447]     Audit-Id: a34b2e51-b518-4960-ba14-34939ff9ff10
I0503 11:26:21.427331   20115 round_trippers.go:447]     Cache-Control: no-store
I0503 11:26:21.427357   20115 round_trippers.go:447]     Content-Type: application/json
I0503 11:26:21.427411   20115 request.go:942] Response Body: {"apiVersion":"config.openshift.io/v1","kind":"ClusterVersion","metadata":{"creationTimestamp":"2019-05-03T16:55:28Z","generation":3,"name":"version","resourceVersion":"34523","selfLink":"/apis/config.openshift.io/v1/clusterversions/version","uid":"4133cde2-6dc4-11e9-b563-0eff287b3e36"},"spec":{"channel":"stable-4.1","clusterID":"908e2825-ce71-4014-aac0-f3fecc63b535","desiredUpdate":{"force":false,"image":"registry.svc.ci.openshift.org/ocp/release:4.1.0-0.nightly-2019-05-03-152614","version":""},"upstream":"https://openshift-release.svc.ci.openshift.org/graph"},"status":{"availableUpdates":[{"force":false,"image":"registry.svc.ci.openshift.org/ocp/release@sha256:e71c3136ce6df1bfac4156a908668cc0d59232d3f14da66f3cf8645e1fa40fdb","version":"4.1.0-0.nightly-2019-05-03-152614"}],"conditions":[{"lastTransitionTime":"2019-05-03T17:08:29Z","message":"Done applying 4.1.0-0.nightly-2019-05-03-121956","status":"True","type":"Available"},{"lastTransitionTime":"2019-05-03T18:25:44Z","message":"The update cannot be verifie [truncated 1119 chars]
info: Cluster is already using release image registry.svc.ci.openshift.org/ocp/release:4.1.0-0.nightly-2019-05-03-152614
pruan@homer ~ $ oc version                                                                                                                                     [ruby-2.6.0]
Client Version: version.Info{Major:"4", Minor:"1+", GitVersion:"v4.1.0-201905030632+dd90b09-dirty", GitCommit:"dd90b09", GitTreeState:"dirty", BuildDate:"2019-05-03T11:05:42Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+efb1744", GitCommit:"efb1744", GitTreeState:"clean", BuildDate:"2019-05-03T11:06:15Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
pruan@dhcp-91-104 ~/junk $ oc get clusterversion -o yaml                                                                                         [ruby-2.6.0]
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: ClusterVersion
  metadata:
    creationTimestamp: "2019-05-03T16:55:28Z"
    generation: 3
    name: version
    resourceVersion: "34523"
    selfLink: /apis/config.openshift.io/v1/clusterversions/version
    uid: 4133cde2-6dc4-11e9-b563-0eff287b3e36
  spec:
    channel: stable-4.1
    clusterID: 908e2825-ce71-4014-aac0-f3fecc63b535
    desiredUpdate:
      force: false
      image: registry.svc.ci.openshift.org/ocp/release:4.1.0-0.nightly-2019-05-03-152614
      version: ""
    upstream: https://openshift-release.svc.ci.openshift.org/graph
  status:
    availableUpdates:
    - force: false
      image: registry.svc.ci.openshift.org/ocp/release@sha256:e71c3136ce6df1bfac4156a908668cc0d59232d3f14da66f3cf8645e1fa40fdb
      version: 4.1.0-0.nightly-2019-05-03-152614
    conditions:
    - lastTransitionTime: "2019-05-03T17:08:29Z"
      message: Done applying 4.1.0-0.nightly-2019-05-03-121956
      status: "True"
      type: Available
    - lastTransitionTime: "2019-05-03T18:25:44Z"
      message: 'The update cannot be verified: verification is not possible'
      reason: ImageVerificationFailed
      status: "True"
      type: Failing
    - lastTransitionTime: "2019-05-03T18:25:44Z"
      message: 'Unable to apply registry.svc.ci.openshift.org/ocp/release:4.1.0-0.nightly-2019-05-03-152614:
        the image may not be safe to use'
      reason: ImageVerificationFailed
      status: "True"
      type: Progressing
    - lastTransitionTime: "2019-05-03T18:06:06Z"
      status: "True"
      type: RetrievedUpdates
    desired:
      force: false
      image: registry.svc.ci.openshift.org/ocp/release:4.1.0-0.nightly-2019-05-03-152614
      version: ""
    history:
    - completionTime: null
      image: registry.svc.ci.openshift.org/ocp/release:4.1.0-0.nightly-2019-05-03-152614
      startedTime: "2019-05-03T18:25:44Z"
      state: Partial
      verified: false
      version: ""
    - completionTime: "2019-05-03T18:25:44Z"
      image: registry.svc.ci.openshift.org/ocp/release@sha256:2cfbcb0c9e6a4b62f2b915c038cb68d96e039bf869150aeaf12fda1ea554d051
      startedTime: "2019-05-03T16:55:42Z"
      state: Completed
      verified: false
      version: 4.1.0-0.nightly-2019-05-03-121956
    observedGeneration: 3
    versionHash: NEYNW44Z7j8=
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""


pruan@dhcp-91-104 ~/workspace/bushslicer (operator_resource)$ oc get clusteroperator                                                             [ruby-2.6.0]
NAME                                 VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                       4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h1m
cloud-credential                     4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h12m
cluster-autoscaler                   4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h13m
console                              4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h5m
dns                                  4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h12m
image-registry                       4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h6m
ingress                              4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h6m
kube-apiserver                       4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h11m
kube-controller-manager              4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h10m
kube-scheduler                       4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h10m
machine-api                          4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h13m
machine-config                       4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h12m
marketplace                          4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h8m
monitoring                           4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h5m
network                              4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h13m
node-tuning                          4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h9m
openshift-apiserver                  4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h9m
openshift-controller-manager         4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h10m
openshift-samples                    4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h1m
operator-lifecycle-manager           4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h11m
operator-lifecycle-manager-catalog   4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h11m
service-ca                           4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h12m
service-catalog-apiserver            4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h9m
service-catalog-controller-manager   4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h9m
storage                              4.1.0-0.nightly-2019-05-03-121956   True        False         False      4h9m



Expected results:

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Mike Dame 2019-05-06 20:45:15 UTC

I found that I was able to work around this by running `oc adm upgrade --clear` after the first attempt, then trying again with --force. Regardless, I think it makes sense that --force should push through an update even if the image matches, so I opened a PR for this: https://github.com/openshift/origin/pull/22786

Comment 3 Mike Dame 2019-05-09 14:05:12 UTC

PR merged, setting to on_qa

Comment 4 Xingxing Xia 2019-05-17 02:51:02 UTC

Peter, could you continue checking the bug (comment 2)?
As to 4.2, currently I didn't see 4.2 build. Not sure if https://console.svc.ci.openshift.org/k8s/ns/ocp/imagestreams could help. I see there is  e.g. https://console.svc.ci.openshift.org/k8s/ns/ocp/imagestreams/4.2-2019-05-16-065650 which displays registry.svc.ci.openshift.org/ocp/4.2-2019-05-16-065650 .
Thanks.

Comment 5 Xingxing Xia 2019-05-29 02:58:59 UTC

Checked the commit message and below git log, it means the fix landed in 4.1 too. Peter said he will verify it later.
[xxia@xxia-pc local_ose_repo]$ git log --date=local upstream/enterprise-4.1 --pretty="%h %an %cd - %s" | grep "Allow oc adm upgrade with --force even"
56d908bd9d Mike Dame Tue May 7 04:39:52 2019 - Allow oc adm upgrade with --force even if versions match

Comment 6 Peter Ruan 2019-05-29 17:53:16 UTC

verified with 
Client Version: version.Info{Major:"4", Minor:"1+", GitVersion:"v4.1.0-201905191700+82e7b55-dirty", GitCommit:"82e7b55", GitTreeState:"dirty", BuildDate:"2019-05-20T00:17:47Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+838b4fa", GitCommit:"838b4fa", GitTreeState:"clean", BuildDate:"2019-05-19T23:51:04Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

Comment 8 errata-xmlrpc 2019-06-19 06:45:34 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1382

Note You need to log in before you can comment on or make changes to this bug.