Bug 1793051

Summary: 4.4.x -> 4.4.y upgrades are stuck at `Cluster did not complete upgrade: timed out waiting for the condition: Working towards ...`
Product: OpenShift Container Platform Reporter: Vadim Rutkovsky <vrutkovs>
Component: Cluster Version OperatorAssignee: Abhinav Dahiya <adahiya>
Status: CLOSED ERRATA QA Contact: liujia <jiajliu>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 4.4CC: aos-bugs, jokerman, maszulik, sdodson
Target Milestone: ---   
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-04 11:25:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Vadim Rutkovsky 2020-01-20 15:10:24 UTC 
Description of problem:
CVO crashes: https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_cluster-api-provider-aws/286/pull-ci-openshift-cluster-api-provider-aws-master-e2e-aws-upgrade/223/artifacts/e2e-aws-upgrade/pods/openshift-cluster-version_cluster-version-operator-5875b9cc65-s8kj9_cluster-version-operator.log

I0120 13:33:01.219756       1 sync_worker.go:471] Running sync registry.svc.ci.openshift.org/ci-op-g6smpmcv/release@sha256:ac73c30e3914b1924bf6f4a74b336a79aafa5a52b78e4ecffa46a5eca9033a71 (force=true) on generation 2 in state Updating at attempt 0
I0120 13:33:01.219869       1 sync_worker.go:477] Loading payload
E0120 13:33:01.220122       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 165 [running]:
github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1581b80, 0x2464bb0)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xaa
github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82
panic(0x1581b80, 0x2464bb0)
	/usr/local/go/src/runtime/panic.go:502 +0x229
github.com/openshift/cluster-version-operator/vendor/golang.org/x/time/rate.(*Limiter).reserveN(0x0, 0xbf8188e34d1cbbee, 0x176b3b3ce6, 0x2480760, 0x1, 0x7fffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/openshift/cluster-version-operator/vendor/golang.org/x/time/rate/rate.go:305 +0x61
github.com/openshift/cluster-version-operator/vendor/golang.org/x/time/rate.(*Limiter).ReserveN(0x0, 0xbf8188e34d1cbbee, 0x176b3b3ce6, 0x2480760, 0x1, 0xc4204e6a28)
	/go/src/github.com/openshift/cluster-version-operator/vendor/golang.org/x/time/rate/rate.go:212 +0x9d
github.com/openshift/cluster-version-operator/vendor/golang.org/x/time/rate.(*Limiter).Reserve(0x0, 0x0)
	/go/src/github.com/openshift/cluster-version-operator/vendor/golang.org/x/time/rate/rate.go:194 +0x56
github.com/openshift/cluster-version-operator/pkg/verify/verifyconfigmap.(*Store).DigestSignatures(0xc4204e6a00, 0x18ee400, 0xc420f0e300, 0xc42067fbb5, 0x47, 0xc4212a9db0, 0x47, 0x3, 0xc4208f1108, 0x410f89)
	/go/src/github.com/openshift/cluster-version-operator/pkg/verify/verifyconfigmap/store.go:74 +0x72
github.com/openshift/cluster-version-operator/pkg/verify.(*ReleaseVerifier).Verify(0xc4204e4780, 0x18ee400, 0xc420f0e300, 0xc42067fbb5, 0x47, 0x7c, 0x1)
	/go/src/github.com/openshift/cluster-version-operator/pkg/verify/verify.go:247 +0x6c8
github.com/openshift/cluster-version-operator/pkg/cvo.(*payloadRetriever).RetrievePayload(0xc4200f7a00, 0x18ee400, 0xc420f0e300, 0x0, 0x0, 0xc42067fb80, 0x7c, 0x1, 0x0, 0x0, ...)
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/updatepayload.go:91 +0x155
github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).syncOnce(0xc42039c840, 0x18ee400, 0xc420f0e300, 0xc420a38120, 0x10, 0x18c9660, 0xc420d14970, 0x0, 0x0)
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:485 +0x5b3
github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).Start.func1.1(0xc420a38120, 0xc42039c840, 0x18ee380, 0xc4208b2400, 0x10, 0x0, 0x0)
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:314 +0x232
github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).Start.func1()
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:315 +0x323
github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc42045df80)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x54
github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc4208f1f80, 0x989680, 0x0, 0x1, 0xc4211ffe00)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xbd
github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc42045df80, 0x989680, 0xc4211ffe00)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).Start(0xc42039c840, 0x18ee380, 0xc4208b2400, 0x10)
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:250 +0x10e
created by github.com/openshift/cluster-version-operator/pkg/cvo.(*Operator).Run
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:345 +0x3c4
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x119cfc1]
Comment 1 Vadim Rutkovsky 2020-01-20 15:11:15 UTC 
Affects a lot of PRs: https://ci-search-ci-search-next.svc.ci.openshift.org/?search=Cluster+did+not+complete+upgrade%3A+timed+out+waiting+for+the+condition%3A+Working+towards&maxAge=168h&context=-1&type=all (~150 last week, but this list may have false-positives)
Comment 2 Maciej Szulik 2020-01-20 15:59:23 UTC 
*** Bug 1792906 has been marked as a duplicate of this bug. ***
Comment 4 liujia 2020-01-23 06:47:07 UTC 
Upgrade succeed from 4.4.0-0.nightly-2020-01-23-031941 to 4.4.0-0.nightly-2020-01-23-031941.

# ./oc get clusterversion -o json|jq -r '.items[0].status.history[]|.startedTime + "|" + .completionTime + "|" + .state + "|" + .version'
2020-01-23T05:47:20Z|2020-01-23T06:15:17Z|Completed|4.4.0-0.nightly-2020-01-23-031941
2020-01-23T03:55:30Z|2020-01-23T04:14:50Z|Completed|4.4.0-0.nightly-2020-01-22-221818
Comment 6 errata-xmlrpc 2020-05-04 11:25:32 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581