Bug 1793051 - 4.4.x -> 4.4.y upgrades are stuck at `Cluster did not complete upgrade: timed out waiting for the condition: Working towards ...`
Summary: 4.4.x -> 4.4.y upgrades are stuck at `Cluster did not complete upgrade: timed...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 4.4
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.4.0
Assignee: Abhinav Dahiya
QA Contact: liujia
URL:
Whiteboard:
: 1792906 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-01-20 15:10 UTC by Vadim Rutkovsky
Modified: 2020-05-04 11:25 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-04 11:25:32 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift cluster-version-operator pull 303 None closed Bug 1793051: cvo: pass limiter when creating a store 2020-08-19 23:31:28 UTC
Red Hat Product Errata RHBA-2020:0581 None None None 2020-05-04 11:25:55 UTC

Description Vadim Rutkovsky 2020-01-20 15:10:24 UTC
Description of problem:
CVO crashes: https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_cluster-api-provider-aws/286/pull-ci-openshift-cluster-api-provider-aws-master-e2e-aws-upgrade/223/artifacts/e2e-aws-upgrade/pods/openshift-cluster-version_cluster-version-operator-5875b9cc65-s8kj9_cluster-version-operator.log

I0120 13:33:01.219756       1 sync_worker.go:471] Running sync registry.svc.ci.openshift.org/ci-op-g6smpmcv/release@sha256:ac73c30e3914b1924bf6f4a74b336a79aafa5a52b78e4ecffa46a5eca9033a71 (force=true) on generation 2 in state Updating at attempt 0
I0120 13:33:01.219869       1 sync_worker.go:477] Loading payload
E0120 13:33:01.220122       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 165 [running]:
github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1581b80, 0x2464bb0)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xaa
github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82
panic(0x1581b80, 0x2464bb0)
	/usr/local/go/src/runtime/panic.go:502 +0x229
github.com/openshift/cluster-version-operator/vendor/golang.org/x/time/rate.(*Limiter).reserveN(0x0, 0xbf8188e34d1cbbee, 0x176b3b3ce6, 0x2480760, 0x1, 0x7fffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/openshift/cluster-version-operator/vendor/golang.org/x/time/rate/rate.go:305 +0x61
github.com/openshift/cluster-version-operator/vendor/golang.org/x/time/rate.(*Limiter).ReserveN(0x0, 0xbf8188e34d1cbbee, 0x176b3b3ce6, 0x2480760, 0x1, 0xc4204e6a28)
	/go/src/github.com/openshift/cluster-version-operator/vendor/golang.org/x/time/rate/rate.go:212 +0x9d
github.com/openshift/cluster-version-operator/vendor/golang.org/x/time/rate.(*Limiter).Reserve(0x0, 0x0)
	/go/src/github.com/openshift/cluster-version-operator/vendor/golang.org/x/time/rate/rate.go:194 +0x56
github.com/openshift/cluster-version-operator/pkg/verify/verifyconfigmap.(*Store).DigestSignatures(0xc4204e6a00, 0x18ee400, 0xc420f0e300, 0xc42067fbb5, 0x47, 0xc4212a9db0, 0x47, 0x3, 0xc4208f1108, 0x410f89)
	/go/src/github.com/openshift/cluster-version-operator/pkg/verify/verifyconfigmap/store.go:74 +0x72
github.com/openshift/cluster-version-operator/pkg/verify.(*ReleaseVerifier).Verify(0xc4204e4780, 0x18ee400, 0xc420f0e300, 0xc42067fbb5, 0x47, 0x7c, 0x1)
	/go/src/github.com/openshift/cluster-version-operator/pkg/verify/verify.go:247 +0x6c8
github.com/openshift/cluster-version-operator/pkg/cvo.(*payloadRetriever).RetrievePayload(0xc4200f7a00, 0x18ee400, 0xc420f0e300, 0x0, 0x0, 0xc42067fb80, 0x7c, 0x1, 0x0, 0x0, ...)
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/updatepayload.go:91 +0x155
github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).syncOnce(0xc42039c840, 0x18ee400, 0xc420f0e300, 0xc420a38120, 0x10, 0x18c9660, 0xc420d14970, 0x0, 0x0)
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:485 +0x5b3
github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).Start.func1.1(0xc420a38120, 0xc42039c840, 0x18ee380, 0xc4208b2400, 0x10, 0x0, 0x0)
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:314 +0x232
github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).Start.func1()
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:315 +0x323
github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc42045df80)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x54
github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc4208f1f80, 0x989680, 0x0, 0x1, 0xc4211ffe00)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xbd
github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc42045df80, 0x989680, 0xc4211ffe00)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).Start(0xc42039c840, 0x18ee380, 0xc4208b2400, 0x10)
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:250 +0x10e
created by github.com/openshift/cluster-version-operator/pkg/cvo.(*Operator).Run
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:345 +0x3c4
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x119cfc1]

Comment 2 Maciej Szulik 2020-01-20 15:59:23 UTC
*** Bug 1792906 has been marked as a duplicate of this bug. ***

Comment 4 liujia 2020-01-23 06:47:07 UTC
Upgrade succeed from 4.4.0-0.nightly-2020-01-23-031941 to 4.4.0-0.nightly-2020-01-23-031941.

# ./oc get clusterversion -o json|jq -r '.items[0].status.history[]|.startedTime + "|" + .completionTime + "|" + .state + "|" + .version'
2020-01-23T05:47:20Z|2020-01-23T06:15:17Z|Completed|4.4.0-0.nightly-2020-01-23-031941
2020-01-23T03:55:30Z|2020-01-23T04:14:50Z|Completed|4.4.0-0.nightly-2020-01-22-221818

Comment 6 errata-xmlrpc 2020-05-04 11:25:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.