Bug 1661078

Summary: docker-registry keeps restarting due to gopanic with unexpected return pc
Product: OpenShift Container Platform Reporter: Kenjiro Nakayama <knakayam>
Component: Image RegistryAssignee: Alexey Gladkov <agladkov>
Status: CLOSED DUPLICATE QA Contact: Wenjing Zheng <wzheng>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.10.0CC: agladkov, aos-bugs, bparees, deparker, knakayam, mpandey, obulatov
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-18 15:01:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kenjiro Nakayama 2018-12-20 04:15:12 UTC
Description of problem:

- docker-registry keeps restarting due to following gopanic. The customer observed some patterns but all errors seem to be produced by startUploadPurger()

eg. PATTERN 1:
  runtime: unexpected return pc for runtime.rawstringtmp called from 0x0
  fatal error: unknown caller pc
    ...
  goroutine 12 [running]:
  sort.Sort(0x218c500, 0xc4204da2c0)
          /usr/lib/golang/src/sort/sort.go:218 +0x88 fp=0xc4207552c8 sp=0xc4207552c0 pc=0x4dca38
  sort.Strings(0xc4203d87e0, 0x7, 0x7)
          /usr/lib/golang/src/sort/sort.go:353 +0x6d fp=0xc420755310 sp=0xc4207552c8 pc=0x4dd1ed
  github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/storage.Walk(0xc4204a3500, 0xc42059c460, 0xc420c540f0, 0xc420020000, 0x0, 0x83, 0xc4202cd000, 0x0, 0x0)
          /builddir/build/BUILD/atomic-openshift-dockerregistry-git-390.186ec4f/_output/local/go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/storage/walk.go:47 +0x27e fp=0xc4207553e0 sp=0xc420755310 pc=0x84187e
  runtime.rawstringtmp(0x0, 0x1517140, 0xc4205a7110, 0xc420c58360, 0xc420c58360, 0x0, 0x0)
          /usr/lib/golang/src/runtime/string.go:115 +0x6f fp=0xc420755420 sp=0xc4207553e0 pc=0x4493cf
  created by github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/handlers.startUploadPurger
          /builddir/build/BUILD/atomic-openshift-dockerregistry-git-390.186ec4f/_output/local/go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/handlers/app.go:1058 +0x2dc

e.g PATTERN 2:
  runtime: unexpected return pc for github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/storage.getOutstandingUploads called from 0x0
  fatal error: unknown caller pc
    ...
  goroutine 6 [running]:
  internal/poll.(*FD).ReadDirent(0xc420974f50, 0xc420bbd000, 0x1000, 0x83b0c7, 0xc420972300, 0x1, 0x64)
          /usr/lib/golang/src/internal/poll/fd_unix.go:361 +0x1ad fp=0xc42091b2f0 sp=0xc42091b2e8 pc=0x49880d
  os.(*File).readdirnames(0xc4207006b8, 0xfffffffffffffffe, 0x18, 0x89bc83, 0xc42091b3e0, 0xc42091b3e8, 0x0)
          /usr/lib/golang/src/os/dir_unix.go:67 +0x1bf fp=0xc42091b398 sp=0xc42091b2f0 pc=0x49ab5f
  os.(*File).Readdirnames(0xc4207006b8, 0x0, 0xc4207006b8, 0x0, 0x0, 0x0, 0x7fb719fee000)
          /usr/lib/golang/src/os/dir.go:45 +0x3e fp=0xc42091b3e0 sp=0xc42091b398 pc=0x49a52e
  github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/storage.getOutstandingUploads(0x0, 0x0, 0x85213d, 0xc42091b4e8, 0xc42091b4f0, 0xc4203c5ee0, 0xc42091b538, 0x852186)
          /builddir/build/BUILD/atomic-openshift-dockerregistry-git-390.186ec4f/_output/local/go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/storage/purgeuploads.go:70 +0x2b7 fp=0xc42091b4a8 sp=0xc42091b3e0 pc=0x83b0c7
  created by github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/handlers.startUploadPurger
          /builddir/build/BUILD/atomic-openshift-dockerregistry-git-390.186ec4f/_output/local/go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/handlers/app.go:1058 +0x2dc

Version-Release:
- atomic-openshift-3.10.72-1.git.0.3cb2fdc.el7.x86_64
- registry.access.redhat.com/openshift3/ose-docker-registry:v3.10.72

How reproducible: Only customer's env observes this issue.

Steps to Reproduce: 
1. Not sure about what triggers this issue.

Actual results:
- registry pod keeps producing gopanic.

Expected results:
- No gopanic.

Additional info:
- This issue would be related to docker-distribution or backend storage rather than image-registry. Please feel free to re-assign this ticket to particular component.

Comment 19 Ben Parees 2019-01-18 15:01:47 UTC
I'm going to optimistically close this as a dupe of https://bugzilla.redhat.com/show_bug.cgi?id=1664386

Support, please re-open as needed.

*** This bug has been marked as a duplicate of bug 1664386 ***

Comment 20 Kenjiro Nakayama 2019-01-18 23:00:56 UTC
Sure, no problem.

(I'm leaving public comment on this bz for people who found this through internet.) The issue could happen due to dynatrace (or 3rd party tool). All customers who hit this issue enabled dynatrace and the issue stopped after disabling it.