Bug 1952610

Summary:	[aws] image-registry panics when the cluster is installed in a new region
Product:	OpenShift Container Platform	Reporter:	Matthew Staebler <mstaeble>
Component:	Image Registry	Assignee:	Oleg Bulatov <obulatov>
Status:	CLOSED ERRATA	QA Contact:	Wenjing Zheng <wzheng>
Severity:	urgent	Docs Contact:
Priority:	high
Version:	4.8	CC:	aos-bugs, wzheng, yunjiang
Target Milestone:	---
Target Release:	4.8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Enhancement
Doc Text:	Feature: support new AWS regions Reason: the image registry and its operator should be aware of new AWS regions Result: aws-sdk-go bumped to v1.38.35	Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-07-27 23:02:56 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1977159

Description Matthew Staebler 2021-04-22 16:41:17 UTC

Description of problem:
When the cluster is installed in a new region that is unknown in the AWS SDK version being used, the image-registry pod crashes.


How reproducible:
100%


Steps to Reproduce:
1. Install a cluster in ap-northeast-3. Use custom endpoints for ec2 and elb so that the machine-api-operator can create the worker machines.
2. Note that the image-registry pods crashloop.

Actual results:
time="2021-04-22T16:34:48.852602706Z" level=info msg="start registry" distribution_version=v2.7.1+unknown go.version=go1.16.1 openshift_version=4.8.0-202104162013.p0-1b129a6
time="2021-04-22T16:34:48.853154564Z" level=info msg="caching project quota objects with TTL 1m0s" go.version=go1.16.1
panic: Invalid region provided: ap-northeast-3

goroutine 1 [running]:
github.com/docker/distribution/registry/handlers.NewApp(0x1ef27e8, 0xc000128010, 0xc0002c7880, 0xc0003934d0)
	/go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/handlers/app.go:127 +0x3133
github.com/openshift/image-registry/pkg/dockerregistry/server/supermiddleware.NewApp(0x1ef27e8, 0xc000128010, 0xc0002c7880, 0x1efab98, 0xc000403200, 0x1efe100)
	/go/src/github.com/openshift/image-registry/pkg/dockerregistry/server/supermiddleware/app.go:96 +0x87
github.com/openshift/image-registry/pkg/dockerregistry/server.NewApp(0x1ef27e8, 0xc000128010, 0x1ec6518, 0xc000550bc8, 0xc0002c7880, 0xc0000b20a0, 0x0, 0x0, 0xc0003920f0, 0xc00053f7c0)
	/go/src/github.com/openshift/image-registry/pkg/dockerregistry/server/app.go:138 +0x2c5
github.com/openshift/image-registry/pkg/cmd/dockerregistry.NewServer(0x1ef27e8, 0xc000128010, 0xc0002c7880, 0xc0000b20a0, 0x0, 0x0, 0x1f142b8)
	/go/src/github.com/openshift/image-registry/pkg/cmd/dockerregistry/dockerregistry.go:210 +0x1ce
github.com/openshift/image-registry/pkg/cmd/dockerregistry.Execute(0x1eb9c40, 0xc0005509b8)
	/go/src/github.com/openshift/image-registry/pkg/cmd/dockerregistry/dockerregistry.go:164 +0xa65
main.main()
	/go/src/github.com/openshift/image-registry/cmd/dockerregistry/main.go:93 +0x573


Expected results:
The image-registry will accept the new region as it matches the regular expression for a region in the default AWS partition.

Comment 2 Wenjing Zheng 2021-05-20 09:40:10 UTC

Still met the error when testing with above two prs.

Comment 3 Oleg Bulatov 2021-05-20 09:51:34 UTC

Are you sure? These PRs bump aws-sdk-go to v1.38.35, the region ap-northeast-3 was introduced in v1.37.24.

Which region do you use? What message do you see?

Comment 4 Wenjing Zheng 2021-05-20 10:08:02 UTC

(In reply to Oleg Bulatov from comment #3)
> Are you sure? These PRs bump aws-sdk-go to v1.38.35, the region
> ap-northeast-3 was introduced in v1.37.24.
> 
> Which region do you use? What message do you see?

Sorry, after double confirm, I did a misoperation， will try again.

Comment 5 Wenjing Zheng 2021-05-20 11:19:00 UTC

Installation succeeds with the prs: openshift/image-registry#272,openshift/cluster-image-registry-operator#685

Comment 7 Wenjing Zheng 2021-06-01 08:22:39 UTC

Verified on 4.8.0-0.nightly-2021-06-01-002052.
  storage:
    managementState: Managed
    s3:
      bucket: wzheng-48-gr7js-image-registry-ap-northeast-3-erpntneleqsylxkm
      encrypt: true
      region: ap-northeast-3
      virtualHostedStyle: false
  unsupportedConfigOverrides: null

Comment 10 errata-xmlrpc 2021-07-27 23:02:56 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438