Description of problem: - docker-registry keeps restarting due to following gopanic. The customer observed some patterns but all errors seem to be produced by startUploadPurger() eg. PATTERN 1: runtime: unexpected return pc for runtime.rawstringtmp called from 0x0 fatal error: unknown caller pc ... goroutine 12 [running]: sort.Sort(0x218c500, 0xc4204da2c0) /usr/lib/golang/src/sort/sort.go:218 +0x88 fp=0xc4207552c8 sp=0xc4207552c0 pc=0x4dca38 sort.Strings(0xc4203d87e0, 0x7, 0x7) /usr/lib/golang/src/sort/sort.go:353 +0x6d fp=0xc420755310 sp=0xc4207552c8 pc=0x4dd1ed github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/storage.Walk(0xc4204a3500, 0xc42059c460, 0xc420c540f0, 0xc420020000, 0x0, 0x83, 0xc4202cd000, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-dockerregistry-git-390.186ec4f/_output/local/go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/storage/walk.go:47 +0x27e fp=0xc4207553e0 sp=0xc420755310 pc=0x84187e runtime.rawstringtmp(0x0, 0x1517140, 0xc4205a7110, 0xc420c58360, 0xc420c58360, 0x0, 0x0) /usr/lib/golang/src/runtime/string.go:115 +0x6f fp=0xc420755420 sp=0xc4207553e0 pc=0x4493cf created by github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/handlers.startUploadPurger /builddir/build/BUILD/atomic-openshift-dockerregistry-git-390.186ec4f/_output/local/go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/handlers/app.go:1058 +0x2dc e.g PATTERN 2: runtime: unexpected return pc for github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/storage.getOutstandingUploads called from 0x0 fatal error: unknown caller pc ... goroutine 6 [running]: internal/poll.(*FD).ReadDirent(0xc420974f50, 0xc420bbd000, 0x1000, 0x83b0c7, 0xc420972300, 0x1, 0x64) /usr/lib/golang/src/internal/poll/fd_unix.go:361 +0x1ad fp=0xc42091b2f0 sp=0xc42091b2e8 pc=0x49880d os.(*File).readdirnames(0xc4207006b8, 0xfffffffffffffffe, 0x18, 0x89bc83, 0xc42091b3e0, 0xc42091b3e8, 0x0) /usr/lib/golang/src/os/dir_unix.go:67 +0x1bf fp=0xc42091b398 sp=0xc42091b2f0 pc=0x49ab5f os.(*File).Readdirnames(0xc4207006b8, 0x0, 0xc4207006b8, 0x0, 0x0, 0x0, 0x7fb719fee000) /usr/lib/golang/src/os/dir.go:45 +0x3e fp=0xc42091b3e0 sp=0xc42091b398 pc=0x49a52e github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/storage.getOutstandingUploads(0x0, 0x0, 0x85213d, 0xc42091b4e8, 0xc42091b4f0, 0xc4203c5ee0, 0xc42091b538, 0x852186) /builddir/build/BUILD/atomic-openshift-dockerregistry-git-390.186ec4f/_output/local/go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/storage/purgeuploads.go:70 +0x2b7 fp=0xc42091b4a8 sp=0xc42091b3e0 pc=0x83b0c7 created by github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/handlers.startUploadPurger /builddir/build/BUILD/atomic-openshift-dockerregistry-git-390.186ec4f/_output/local/go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/handlers/app.go:1058 +0x2dc Version-Release: - atomic-openshift-3.10.72-1.git.0.3cb2fdc.el7.x86_64 - registry.access.redhat.com/openshift3/ose-docker-registry:v3.10.72 How reproducible: Only customer's env observes this issue. Steps to Reproduce: 1. Not sure about what triggers this issue. Actual results: - registry pod keeps producing gopanic. Expected results: - No gopanic. Additional info: - This issue would be related to docker-distribution or backend storage rather than image-registry. Please feel free to re-assign this ticket to particular component.
I'm going to optimistically close this as a dupe of https://bugzilla.redhat.com/show_bug.cgi?id=1664386 Support, please re-open as needed. *** This bug has been marked as a duplicate of bug 1664386 ***
Sure, no problem. (I'm leaving public comment on this bz for people who found this through internet.) The issue could happen due to dynatrace (or 3rd party tool). All customers who hit this issue enabled dynatrace and the issue stopped after disabling it.