Description of problem: CVO crashes: https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_cluster-api-provider-aws/286/pull-ci-openshift-cluster-api-provider-aws-master-e2e-aws-upgrade/223/artifacts/e2e-aws-upgrade/pods/openshift-cluster-version_cluster-version-operator-5875b9cc65-s8kj9_cluster-version-operator.log I0120 13:33:01.219756 1 sync_worker.go:471] Running sync registry.svc.ci.openshift.org/ci-op-g6smpmcv/release@sha256:ac73c30e3914b1924bf6f4a74b336a79aafa5a52b78e4ecffa46a5eca9033a71 (force=true) on generation 2 in state Updating at attempt 0 I0120 13:33:01.219869 1 sync_worker.go:477] Loading payload E0120 13:33:01.220122 1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) goroutine 165 [running]: github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1581b80, 0x2464bb0) /go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xaa github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82 panic(0x1581b80, 0x2464bb0) /usr/local/go/src/runtime/panic.go:502 +0x229 github.com/openshift/cluster-version-operator/vendor/golang.org/x/time/rate.(*Limiter).reserveN(0x0, 0xbf8188e34d1cbbee, 0x176b3b3ce6, 0x2480760, 0x1, 0x7fffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/openshift/cluster-version-operator/vendor/golang.org/x/time/rate/rate.go:305 +0x61 github.com/openshift/cluster-version-operator/vendor/golang.org/x/time/rate.(*Limiter).ReserveN(0x0, 0xbf8188e34d1cbbee, 0x176b3b3ce6, 0x2480760, 0x1, 0xc4204e6a28) /go/src/github.com/openshift/cluster-version-operator/vendor/golang.org/x/time/rate/rate.go:212 +0x9d github.com/openshift/cluster-version-operator/vendor/golang.org/x/time/rate.(*Limiter).Reserve(0x0, 0x0) /go/src/github.com/openshift/cluster-version-operator/vendor/golang.org/x/time/rate/rate.go:194 +0x56 github.com/openshift/cluster-version-operator/pkg/verify/verifyconfigmap.(*Store).DigestSignatures(0xc4204e6a00, 0x18ee400, 0xc420f0e300, 0xc42067fbb5, 0x47, 0xc4212a9db0, 0x47, 0x3, 0xc4208f1108, 0x410f89) /go/src/github.com/openshift/cluster-version-operator/pkg/verify/verifyconfigmap/store.go:74 +0x72 github.com/openshift/cluster-version-operator/pkg/verify.(*ReleaseVerifier).Verify(0xc4204e4780, 0x18ee400, 0xc420f0e300, 0xc42067fbb5, 0x47, 0x7c, 0x1) /go/src/github.com/openshift/cluster-version-operator/pkg/verify/verify.go:247 +0x6c8 github.com/openshift/cluster-version-operator/pkg/cvo.(*payloadRetriever).RetrievePayload(0xc4200f7a00, 0x18ee400, 0xc420f0e300, 0x0, 0x0, 0xc42067fb80, 0x7c, 0x1, 0x0, 0x0, ...) /go/src/github.com/openshift/cluster-version-operator/pkg/cvo/updatepayload.go:91 +0x155 github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).syncOnce(0xc42039c840, 0x18ee400, 0xc420f0e300, 0xc420a38120, 0x10, 0x18c9660, 0xc420d14970, 0x0, 0x0) /go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:485 +0x5b3 github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).Start.func1.1(0xc420a38120, 0xc42039c840, 0x18ee380, 0xc4208b2400, 0x10, 0x0, 0x0) /go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:314 +0x232 github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).Start.func1() /go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:315 +0x323 github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc42045df80) /go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x54 github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc4208f1f80, 0x989680, 0x0, 0x1, 0xc4211ffe00) /go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xbd github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc42045df80, 0x989680, 0xc4211ffe00) /go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d github.com/openshift/cluster-version-operator/pkg/cvo.(*SyncWorker).Start(0xc42039c840, 0x18ee380, 0xc4208b2400, 0x10) /go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync_worker.go:250 +0x10e created by github.com/openshift/cluster-version-operator/pkg/cvo.(*Operator).Run /go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:345 +0x3c4 panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x119cfc1]
Affects a lot of PRs: https://ci-search-ci-search-next.svc.ci.openshift.org/?search=Cluster+did+not+complete+upgrade%3A+timed+out+waiting+for+the+condition%3A+Working+towards&maxAge=168h&context=-1&type=all (~150 last week, but this list may have false-positives)
*** Bug 1792906 has been marked as a duplicate of this bug. ***
Upgrade succeed from 4.4.0-0.nightly-2020-01-23-031941 to 4.4.0-0.nightly-2020-01-23-031941. # ./oc get clusterversion -o json|jq -r '.items[0].status.history[]|.startedTime + "|" + .completionTime + "|" + .state + "|" + .version' 2020-01-23T05:47:20Z|2020-01-23T06:15:17Z|Completed|4.4.0-0.nightly-2020-01-23-031941 2020-01-23T03:55:30Z|2020-01-23T04:14:50Z|Completed|4.4.0-0.nightly-2020-01-22-221818
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581