Bug 1798879

Created attachment 1658119 [details]
Inventory file

Description of problem:

Internal image registry pods don't start on a newly installed cluster on AWS, the log from the pods show the probable cause:

panic: Invalid region provided: eu-north-1

Apparently the issue comes from the storage backend S3 bucket used for the registry, that is created in the eu-north-1 (Stockholm) region, as the rest of the components of the cluster.

Issue could be related to https://github.com/docker/distribution/issues/2870

The workaround is simple enough, create an S3 bucket in another supported region and modify the registry-config secret to point to that bucket, then rollout a new deployment of docker-registry deployment config.

Version-Release number of selected component (if applicable):

OCP 3.11.161

How reproducible:

Deploy an OCP 3.11.161 cluster in AWS region eu-north-1 and use an S3 bucket as backend storage for the registry.  Deployment completes without errors but docker-registry pods fail to start


Actual results:

$ oc get pods -n default
NAME                       READY     STATUS             RESTARTS   AGE
docker-registry-1-deploy   0/1       Error              0          18h
docker-registry-2-deploy   0/1       Error              0          16h
docker-registry-3-deploy   0/1       Error              0          15h
docker-registry-4-deploy   1/1       Running            0          7m
docker-registry-4-l9xfw    0/1       CrashLoopBackOff   6          6m
docker-registry-4-lx8jd    0/1       CrashLoopBackOff   6          6m
docker-registry-4-vdhpp    0/1       CrashLoopBackOff   6          6m


Additional info:

Inventory file used to deploy the cluster is pretty simple, attached to the bugzilla.


registry-config secret contents:

version: 0.1
log:
  level: info
http:
  addr: :5000
storage:
  delete:
    enabled: true
  cache:
    blobdescriptor: inmemory
  s3:
    accesskey: XXX
    secretkey: XXX
    region: eu-north-1
    bucket: XXX
    encrypt: False
    secure: true
    v4auth: true
    rootdirectory: /registry
    chunksize: "26214400"
auth:
  openshift:
    realm: openshift
middleware:
  registry:
  - name: openshift
  repository:
  - name: openshift
    options:
      pullthrough: true
      acceptschema2: true
      enforcequota: true
  storage:
  - name: openshift