Description of problem: seen in CI run: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_openshift-apiserver/21/pull-ci-openshift-openshift-apiserver-master-e2e-aws/65 E0905 19:06:31.421743 1 runtime.go:67] Observed a panic: nil context /go/src/github.com/openshift/openshift-controller-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76 /go/src/github.com/openshift/openshift-controller-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65 /go/src/github.com/openshift/openshift-controller-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51 /usr/local/go/src/runtime/panic.go:522 /usr/local/go/src/net/http/request.go:350 /go/src/github.com/openshift/openshift-controller-manager/vendor/github.com/containers/image/docker/docker_client.go:423 /go/src/github.com/openshift/openshift-controller-manager/vendor/github.com/containers/image/docker/docker_client.go:565 /go/src/github.com/openshift/openshift-controller-manager/vendor/github.com/containers/image/docker/docker_client.go:580 /go/src/github.com/openshift/openshift-controller-manager/vendor/github.com/containers/image/docker/docker_client.go:618 /usr/local/go/src/sync/once.go:44 /go/src/github.com/openshift/openshift-controller-manager/vendor/github.com/containers/image/docker/docker_client.go:618 /go/src/github.com/openshift/openshift-controller-manager/vendor/github.com/containers/image/docker/docker_client.go:406 /go/src/github.com/openshift/openshift-controller-manager/vendor/github.com/containers/image/docker/docker_image_src.go:143 /go/src/github.com/openshift/openshift-controller-manager/vendor/github.com/containers/image/docker/docker_image_src.go:175 /go/src/github.com/openshift/openshift-controller-manager/vendor/github.com/containers/image/docker/docker_image_src.go:87 /go/src/github.com/openshift/openshift-controller-manager/vendor/github.com/containers/image/docker/docker_transport.go:144 /go/src/github.com/openshift/openshift-controller-manager/pkg/image/controller/signature/container_image_downloader.go:39 /go/src/github.com/openshift/openshift-controller-manager/pkg/image/controller/signature/signature_import_controller.go:142 /go/src/github.com/openshift/openshift-controller-manager/pkg/image/controller/signature/signature_import_controller.go:100 /go/src/github.com/openshift/openshift-controller-manager/pkg/image/controller/signature/signature_import_controller.go:86 /go/src/github.com/openshift/openshift-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 /go/src/github.com/openshift/openshift-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 /go/src/github.com/openshift/openshift-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 /usr/local/go/src/runtime/asm_amd64.s:1337 Version-Release number of selected component (if applicable): 4.2 How reproducible: unknown Actual results: nil ptr panic Expected results: should not panic Additional info: there were some imagestream import failures in the run due to registry.redhat.io issues, it's possible that contributed to the state that lead to the nil ptr errors.
Ben / Adam (with Oleg out on PTO) Traversing the stack, the initial call at https://github.com/openshift/openshift-controller-manager/blob/master/pkg/image/controller/signature/container_image_downloader.go#L39 passes a nil context that is propagated down the containers/image methods and flagged by /usr/local/go/src/net/http/request.go:350 You would think this panic would show up frequently if this code path is called with any frequency. Seems like the context creation a few lines down should be moved up and passed into reference.NewImageSource(nil, nil) Thoughts?
Though maybe the recent containers/image bump has inserted more stringent checks.
i think imagesignature import is not exercised regularly, but i would also guess that it's being exercised here by a specific test so you'd think the panic would show up every time the test is run if nothing else. I wonder if this was introduced by a recent dep bump. Anyway in principal i agree w/ the suggested solution... though it might be worth checking how the code used to run to ensure the dep bump didn't introduce a remote call that we weren't even making previously, somehow.
heh. comment collision. i think we're on the same page, gabe.
Yep it did introduce remote calls .... if you look at the old https://github.com/openshift/openshift-controller-manager/blob/f65d698f67c6cc0f38b1977f8cb2a8e5d519217f/vendor/github.com/containers/image/docker/docker_image_src.go and the newImageSource method vs. what is there now ... where there is a testImageSource.ensureManifestIsLoaded(ctx) So I'll get a PR up with the suggested solution But yeah, the bump PRs ran the e2e-aws test suite ... let's keep an eye on it when the PR is up.
[1]. Don't meet the imageimport error in last several e2e-aws ci job history. [2].Also when I try to refer to https://bugzilla.redhat.com/show_bug.cgi?id=1722568#c10 to get the image imagesignatures, just found there is no way to change openshift-controller-manager operator stand-by. $oc describe co openshift-controller-manager Status: Conditions: Last Transition Time: 2019-09-09T01:03:44Z Message: WorkloadDegraded: the controller manager spec was set to Unmanaged state, but that is unsupported, and has no effect on this condition Reason: AsExpected Status: False Type: Degraded So I will verify this bug refer to [1], and will figure out how to make [2] works.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922