Here is image-registry pod log: [root@preserve-jialiu-ansible 20190307]# oc logs image-registry-79cfddbcc7-r2dgq -n openshift-image-registry time="2019-03-07T10:20:45.91601366Z" level=info msg="start registry" distribution_version=v2.6.0+unknown go.version=go1.10.8 openshift_version=v4.0.16-1+b616663-dirty time="2019-03-07T10:20:45.916526347Z" level=info msg="caching project quota objects with TTL 1m0s" go.version=go1.10.8 panic: Invalid region provided: eu-west-3 goroutine 1 [running]: github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/handlers.NewApp(0x180a060, 0xc420046038, 0xc4200d4700, 0xc4205c4e00) /go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/handlers/app.go:127 +0x339c github.com/openshift/image-registry/pkg/dockerregistry/server/supermiddleware.NewApp(0x180a060, 0xc420046038, 0xc4200d4700, 0x180fde0, 0xc420506d80, 0x1811cc0) /go/src/github.com/openshift/image-registry/pkg/dockerregistry/server/supermiddleware/app.go:96 +0x8a github.com/openshift/image-registry/pkg/dockerregistry/server.NewApp(0x180a060, 0xc420046038, 0x17f0f80, 0xc4201902a8, 0xc4200d4700, 0xc420450000, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/openshift/image-registry/pkg/dockerregistry/server/app.go:138 +0x2ae github.com/openshift/image-registry/pkg/cmd/dockerregistry.NewServer(0x180a060, 0xc420046038, 0xc4200d4700, 0xc420450000, 0x0, 0x0, 0x1829160) /go/src/github.com/openshift/image-registry/pkg/cmd/dockerregistry/dockerregistry.go:187 +0x19c github.com/openshift/image-registry/pkg/cmd/dockerregistry.Execute(0x17ed860, 0xc420190080) /go/src/github.com/openshift/image-registry/pkg/cmd/dockerregistry/dockerregistry.go:162 +0x9c3 main.main() /go/src/github.com/openshift/image-registry/cmd/dockerregistry/main.go:56 +0x403
This has been fixed by [1], but for some reason it isn't in the release. [1]: https://github.com/openshift/image-registry/pull/162
The first release with the fix is registry.svc.ci.openshift.org/ocp/release:4.0.0-0.nightly-2019-03-06-095230
> This has been fixed by [1], but for some reason it isn't in the release. > > [1]: https://github.com/openshift/image-registry/pull/162 Because that PR landed 2019-03-05T15:44Z [1] and the release was cut from a 2019-03-05T18:33:12Z build [2]. [1]: https://github.com/openshift/image-registry/pull/162#event-2181367399 [2]: https://github.com/openshift/installer/releases/tag/v0.14.0
> Because that PR landed 2019-03-05T15:44Z [1] and the release was cut from a 2019-03-05T18:33:12Z build [2]. And obviously I can't do math, since 15 < 18 ;). I dunno, maybe ART just didn't fast-forward their tag? Or maybe the build timestamps are generated at the end of the build cycle instead of the beginning?
Verified this bug with 4.0.0-0.nightly-2019-03-14-212114, and PASS. [root@preserve-jialiu-ansible 20190315]# oc get po -n openshift-image-registry NAME READY STATUS RESTARTS AGE cluster-image-registry-operator-5bd768db7c-xthfs 1/1 Running 0 28m image-registry-5646c8d48-zkbgl 1/1 Running 0 27m node-ca-bn4gd 1/1 Running 0 27m node-ca-hwxt9 1/1 Running 0 27m node-ca-mkxnd 1/1 Running 0 27m node-ca-v9fjk 1/1 Running 0 26m [root@preserve-jialiu-ansible 20190315]# oc get machinesets.machine.openshift.io --all-namespaces NAMESPACE NAME DESIRED CURRENT READY AVAILABLE AGE openshift-machine-api qe-jialiu1-ghd86-worker-eu-west-3a 1 1 1 1 33m openshift-machine-api qe-jialiu1-ghd86-worker-eu-west-3b 0 0 33m openshift-machine-api qe-jialiu1-ghd86-worker-eu-west-3c 0 0 33m
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758