Install an IPI cluster on new AWS region ap-southeast-3, imgege registry got a panic error: oc logs -n openshift-image-registry image-registry-7745c97c84-w2fm6 time="2022-03-18T07:04:04.329871661Z" level=info msg="start registry" distribution_version=v2.7.1+unknown go.version=go1.17.5 openshift_version=4.10.0-202203111548.p0.g9fb7451.assembly.stream-9fb7451 time="2022-03-18T07:04:04.330259828Z" level=info msg="caching project quota objects with TTL 1m0s" go.version=go1.17.5 panic: Invalid region provided: ap-southeast-3 goroutine 1 [running]: github.com/docker/distribution/registry/handlers.NewApp({0x20252a8, 0xc00012c000}, 0xc000535880) /go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/handlers/app.go:127 +0x2829 github.com/openshift/image-registry/pkg/dockerregistry/server/supermiddleware.NewApp({0x20252a8, 0xc00012c000}, 0x0, {0x2041e98, 0xc000579dd0}) /go/src/github.com/openshift/image-registry/pkg/dockerregistry/server/supermiddleware/app.go:96 +0xb9 github.com/openshift/image-registry/pkg/dockerregistry/server.NewApp({0x20252a8, 0xc00012c000}, {0x2009750, 0xc00031c1a0}, 0xc000535880, 0xc0001b8dc0, {0x0, 0x0}) /go/src/github.com/openshift/image-registry/pkg/dockerregistry/server/app.go:138 +0x466 github.com/openshift/image-registry/pkg/cmd/dockerregistry.NewServer({0x20252a8, 0xc00012c000}, 0xc000535880, 0xc0001b8dc0) /go/src/github.com/openshift/image-registry/pkg/cmd/dockerregistry/dockerregistry.go:210 +0x36a github.com/openshift/image-registry/pkg/cmd/dockerregistry.Execute({0x1ff2660, 0xc00031c010}) /go/src/github.com/openshift/image-registry/pkg/cmd/dockerregistry/dockerregistry.go:164 +0x889 main.main() /go/src/github.com/openshift/image-registry/cmd/dockerregistry/main.go:93 +0x496 Version-Release number of the following components: 4.10.5 How reproducible: Always Steps to Reproduce: 1. Create install-config.yaml and update it as: platform: aws: region: ap-southeast-3 serviceEndpoints: - name: ec2 url: https://ec2.ap-southeast-3.amazonaws.com 2. create an IPI cluster Actual results: panic: Invalid region provided: ap-southeast-3 Expected results: image-registry operator is available and cluster can be installed successfully Additional info: 1. this issue is similar as Bug 1882199 2. specify serviceEndpoints is due to Bug 2065510, this is a workaround.
Hi, I have passed the image registry panic error by specifying the serviceEndpoints for S3 also. This is probably due to openshift-installer (via its vendor aws-sdk) does not have the visibility of ap-southeast-3 region, because it was not GA yet during the development of openshift-install 4.10. I have put the details of my install-config.yaml in [1] [1] https://bugzilla.redhat.com/show_bug.cgi?id=2065510
The workaround to add serviceEndpoints for ap-southeast-3 region in install-config.yaml works for image-registry. If not add `regionEndpoint: https://s3.ap-southeast-3.amazonaws.com` in image registry configure, will meet comment #0 panic.
Works well in ap-southeast-3 redhat@jitli:~/work/src/test/2074050/test$ oc get node -l node-role.kubernetes.io/worker -o=jsonpath='{.items[*].metadata.labels.topology\.kubernetes\.io\/zone}' ap-southeast-3a ap-southeast-3b ap-southeast-3c redhat@jitli:~/work/src/test/2074050/test$ oc get pods -n openshift-image-registry NAME READY STATUS RESTARTS AGE cluster-image-registry-operator-975868bd5-mzftg 1/1 Running 0 34m image-registry-5d7bc4499c-6srvt 1/1 Running 0 21m image-registry-5d7bc4499c-r9l62 1/1 Running 0 21m node-ca-5tbsp 1/1 Running 0 21m node-ca-69xw9 1/1 Running 0 21m node-ca-b24tn 1/1 Running 0 17m node-ca-bnvmt 1/1 Running 0 21m node-ca-n4v7z 1/1 Running 0 17m node-ca-sfkzr 1/1 Running 0 17m https://mastern-jenkins-csb-openshift-qe.apps.ocp-c1.prod.psi.redhat.com/job/ocp-common/job/ginkgo-test-vm/19619/testReport/ lgtm Done
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069