Bug 1952610 - [aws] image-registry panics when the cluster is installed in a new region
Summary: [aws] image-registry panics when the cluster is installed in a new region
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 4.8
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ---
: 4.8.0
Assignee: Oleg Bulatov
QA Contact: Wenjing Zheng
URL:
Whiteboard:
Depends On:
Blocks: 1977159
TreeView+ depends on / blocked
 
Reported: 2021-04-22 16:41 UTC by Matthew Staebler
Modified: 2021-07-27 23:03 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Feature: support new AWS regions Reason: the image registry and its operator should be aware of new AWS regions Result: aws-sdk-go bumped to v1.38.35
Clone Of:
Environment:
Last Closed: 2021-07-27 23:02:56 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-image-registry-operator pull 685 0 None open Bug 1952610: Bump dependencies 2021-05-07 12:32:53 UTC
Github openshift image-registry pull 272 0 None open Bug 1952610: Bump dependencies 2021-05-07 12:52:23 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 23:03:12 UTC

Description Matthew Staebler 2021-04-22 16:41:17 UTC
Description of problem:
When the cluster is installed in a new region that is unknown in the AWS SDK version being used, the image-registry pod crashes.


How reproducible:
100%


Steps to Reproduce:
1. Install a cluster in ap-northeast-3. Use custom endpoints for ec2 and elb so that the machine-api-operator can create the worker machines.
2. Note that the image-registry pods crashloop.

Actual results:
time="2021-04-22T16:34:48.852602706Z" level=info msg="start registry" distribution_version=v2.7.1+unknown go.version=go1.16.1 openshift_version=4.8.0-202104162013.p0-1b129a6
time="2021-04-22T16:34:48.853154564Z" level=info msg="caching project quota objects with TTL 1m0s" go.version=go1.16.1
panic: Invalid region provided: ap-northeast-3

goroutine 1 [running]:
github.com/docker/distribution/registry/handlers.NewApp(0x1ef27e8, 0xc000128010, 0xc0002c7880, 0xc0003934d0)
	/go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/handlers/app.go:127 +0x3133
github.com/openshift/image-registry/pkg/dockerregistry/server/supermiddleware.NewApp(0x1ef27e8, 0xc000128010, 0xc0002c7880, 0x1efab98, 0xc000403200, 0x1efe100)
	/go/src/github.com/openshift/image-registry/pkg/dockerregistry/server/supermiddleware/app.go:96 +0x87
github.com/openshift/image-registry/pkg/dockerregistry/server.NewApp(0x1ef27e8, 0xc000128010, 0x1ec6518, 0xc000550bc8, 0xc0002c7880, 0xc0000b20a0, 0x0, 0x0, 0xc0003920f0, 0xc00053f7c0)
	/go/src/github.com/openshift/image-registry/pkg/dockerregistry/server/app.go:138 +0x2c5
github.com/openshift/image-registry/pkg/cmd/dockerregistry.NewServer(0x1ef27e8, 0xc000128010, 0xc0002c7880, 0xc0000b20a0, 0x0, 0x0, 0x1f142b8)
	/go/src/github.com/openshift/image-registry/pkg/cmd/dockerregistry/dockerregistry.go:210 +0x1ce
github.com/openshift/image-registry/pkg/cmd/dockerregistry.Execute(0x1eb9c40, 0xc0005509b8)
	/go/src/github.com/openshift/image-registry/pkg/cmd/dockerregistry/dockerregistry.go:164 +0xa65
main.main()
	/go/src/github.com/openshift/image-registry/cmd/dockerregistry/main.go:93 +0x573


Expected results:
The image-registry will accept the new region as it matches the regular expression for a region in the default AWS partition.

Comment 2 Wenjing Zheng 2021-05-20 09:40:10 UTC
Still met the error when testing with above two prs.

Comment 3 Oleg Bulatov 2021-05-20 09:51:34 UTC
Are you sure? These PRs bump aws-sdk-go to v1.38.35, the region ap-northeast-3 was introduced in v1.37.24.

Which region do you use? What message do you see?

Comment 4 Wenjing Zheng 2021-05-20 10:08:02 UTC
(In reply to Oleg Bulatov from comment #3)
> Are you sure? These PRs bump aws-sdk-go to v1.38.35, the region
> ap-northeast-3 was introduced in v1.37.24.
> 
> Which region do you use? What message do you see?

Sorry, after double confirm, I did a misoperation, will try again.

Comment 5 Wenjing Zheng 2021-05-20 11:19:00 UTC
Installation succeeds with the prs: openshift/image-registry#272,openshift/cluster-image-registry-operator#685

Comment 7 Wenjing Zheng 2021-06-01 08:22:39 UTC
Verified on 4.8.0-0.nightly-2021-06-01-002052.
  storage:
    managementState: Managed
    s3:
      bucket: wzheng-48-gr7js-image-registry-ap-northeast-3-erpntneleqsylxkm
      encrypt: true
      region: ap-northeast-3
      virtualHostedStyle: false
  unsupportedConfigOverrides: null

Comment 10 errata-xmlrpc 2021-07-27 23:02:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.