Bug 1976782

Summary: Openshift registry starts to segfault after S3 storage configuration
Product: OpenShift Container Platform Reporter: Simon Belmas-Gauderic <sbelmasg>
Component: Image RegistryAssignee: Oleg Bulatov <obulatov>
Status: CLOSED ERRATA QA Contact: XiuJuan Wang <xiuwang>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.6.zCC: aos-bugs, fgleizes, llopezmo, obulatov, xiuwang
Target Milestone: ---Keywords: Reopened
Target Release: 4.11.0Flags: xiuwang: needinfo-
Hardware: All   
OS: Linux   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: the upstream Distribution doesn't work with Ceph Radosgw as it doesn't provide KeyCount. Consequence: image-registry doesn't work with Ceph Radosgw. Fix: backport the upstream fix for missing KeyCount. Result: image-registry works with Ceph Radosgw.
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-08-10 10:36:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 2068433    
Bug Blocks: 2060362, 2074015    

Description Simon Belmas-Gauderic 2021-06-28 08:37:34 UTC
Description of problem:
On On premise cluster with Scality S3 implementation.

After configuring registry storage to S3, registry logs start to show errors (see below).
Anyway, registry seems to work as expected.

time="{ANONYMIZED}" level=info msg="PurgeUploads starting: olderThan={ANONYMIZED} m=-604739.966203103, actuallyDelete=true"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xe0d1bd]

goroutine 30 [running]:
github.com/docker/distribution/registry/storage/driver/s3-aws.(*driver).doWalk.func1(0xc0006ea200, 0xc000364901, 0x7f82c294f108)
                /go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/storage/driver/s3-aws/s3.go:1014 +0x9d
github.com/aws/aws-sdk-go/service/s3.(*S3).ListObjectsV2PagesWithContext(0xc0002097c0, 0x1f2d1a0, 0xc0009c01c0, 0xc000126280, 0xc000bcf9c0, 0x0, 0x0, 0x0, 0x50f7e5, 0x1a393c0)
                /go/src/github.com/openshift/image-registry/vendor/github.com/aws/aws-sdk-go/service/s3/api.go:6104 +0x16a
github.com/docker/distribution/registry/storage/driver/s3-aws.(*driver).doWalk(0xc000213380, 0x1f2d1a0, 0xc0009c0150, 0xc00082bab0, 0xc000139141, 0x20, 0x1c12ef6, 0x1, 0xc0008ae1e0, 0x0, ...)
                /go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/storage/driver/s3-aws/s3.go:1012 +0x377
github.com/docker/distribution/registry/storage/driver/s3-aws.(*driver).Walk(0xc000213380, 0x1f2d1a0, 0xc0009c0150, 0xc000dae160, 0x20, 0xc0008ae1e0, 0xc000dae160, 0x20)
                /go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/storage/driver/s3-aws/s3.go:960 +0x172
github.com/docker/distribution/registry/storage/driver/base.(*Base).Walk(0xc0005e1ce0, 0x1f2d1a0, 0xc0009c0150, 0xc000dae160, 0x20, 0xc0008ae1e0, 0x0, 0x0)
                /go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/storage/driver/base/base.go:239 +0x28c
github.com/docker/distribution/registry/storage.getOutstandingUploads(0x1f2d2e0, 0xc0007fc3c0, 0x1f47c60, 0xc0005e1ce0, 0xc0006ccde8, 0x2, 0x2, 0xd)
                /go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/storage/purgeuploads.go:70 +0x24c
github.com/docker/distribution/registry/storage.PurgeUploads(0x1f2d2e0, 0xc0007fc3c0, 0x1f47c60, 0xc0005e1ce0, 0xc025580bf750dff3, 0xfffdd9fe01220b21, 0x2aa7fc0, 0x1, 0x0, 0x0, ...)
                /go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/storage/purgeuploads.go:34 +0x15d
github.com/docker/distribution/registry/handlers.startUploadPurger.func1(0x1f50940, 0xc0007ff2d0, 0x1f2d2e0, 0xc0007fc3c0, 0x1f47c60, 0xc0005e1ce0, 0x2260ff9290000, 0x0, 0x4e94914f0000)
                /go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/handlers/app.go:1123 +0x22d
created by github.com/docker/distribution/registry/handlers.startUploadPurger
                /go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/handlers/app.go:1116 +0x2d8

Version-Release number of selected component (if applicable):
OCP 4.6.28

How reproducible:

Steps to Reproduce:
1. Install cluster.
2. Configure registry storage

apiVersion: imageregistry.operator.openshift.io/v1
kind: Config
  - imageregistry.operator.openshift.io/finalizer
  generation: 5
  name: cluster
    managementState: Unmanaged
      bucket: {bucket-name}
      region: {region-name}
      regionEndpoint: {s3-endpoint}
Actual results:
Error in logs

Expected results:
No error in logs

Additional info:
S3 storage is on premise scality solution.

Comment 1 Oleg Bulatov 2021-06-28 10:47:43 UTC
Can you reproduce it on AWS? We don't support 3rd party storage solutions.

Comment 13 Lucas López Montero 2022-02-14 08:01:11 UTC
Is it expected that this will be merged soon?

Comment 14 Oleg Bulatov 2022-02-14 13:16:41 UTC
This issue is scheduled to the next sprint and expected to be fixed by March 11.

Comment 15 Lucas López Montero 2022-02-14 13:45:02 UTC
Thank you very much, Oleg.

Comment 19 XiuJuan Wang 2022-03-08 09:38:53 UTC
Do registry regression test on vsphere 4.11.0-0.nightly-2022-03-04-063157 cluster which configures s3 storage for image registry. 
No issue founds.

Comment 21 XiuJuan Wang 2022-03-14 10:45:25 UTC
After re-check registry with ODF ceph rgw on 4.11.0-0.nightly-2022-03-13-055724, it prompts 403 error when push image.
Assign this bug back.

Comment 24 errata-xmlrpc 2022-08-10 10:36:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.