Bug 1686361

Summary: "eu-west-3" is treated as an invalid region for image registry
Product: OpenShift Container Platform Reporter: Johnny Liu <jialiu>
Component: Image RegistryAssignee: Oleg Bulatov <obulatov>
Status: CLOSED ERRATA QA Contact: Wenjing Zheng <wzheng>
Severity: high Docs Contact:
Priority: high    
Version: 4.1.0CC: aos-bugs, emahoney, erich, mifiedle, wking
Target Milestone: ---Keywords: NeedsTestCase
Target Release: 4.1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: 1674034 Environment:
Last Closed: 2019-06-04 10:45:17 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1674034    
Bug Blocks: 1664187    

Comment 1 Johnny Liu 2019-03-07 10:31:15 UTC 
Here is image-registry pod log:
[root@preserve-jialiu-ansible 20190307]# oc logs image-registry-79cfddbcc7-r2dgq -n openshift-image-registry
time="2019-03-07T10:20:45.91601366Z" level=info msg="start registry" distribution_version=v2.6.0+unknown go.version=go1.10.8 openshift_version=v4.0.16-1+b616663-dirty
time="2019-03-07T10:20:45.916526347Z" level=info msg="caching project quota objects with TTL 1m0s" go.version=go1.10.8
panic: Invalid region provided: eu-west-3

goroutine 1 [running]:
github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/handlers.NewApp(0x180a060, 0xc420046038, 0xc4200d4700, 0xc4205c4e00)
	/go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/handlers/app.go:127 +0x339c
github.com/openshift/image-registry/pkg/dockerregistry/server/supermiddleware.NewApp(0x180a060, 0xc420046038, 0xc4200d4700, 0x180fde0, 0xc420506d80, 0x1811cc0)
	/go/src/github.com/openshift/image-registry/pkg/dockerregistry/server/supermiddleware/app.go:96 +0x8a
github.com/openshift/image-registry/pkg/dockerregistry/server.NewApp(0x180a060, 0xc420046038, 0x17f0f80, 0xc4201902a8, 0xc4200d4700, 0xc420450000, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/openshift/image-registry/pkg/dockerregistry/server/app.go:138 +0x2ae
github.com/openshift/image-registry/pkg/cmd/dockerregistry.NewServer(0x180a060, 0xc420046038, 0xc4200d4700, 0xc420450000, 0x0, 0x0, 0x1829160)
	/go/src/github.com/openshift/image-registry/pkg/cmd/dockerregistry/dockerregistry.go:187 +0x19c
github.com/openshift/image-registry/pkg/cmd/dockerregistry.Execute(0x17ed860, 0xc420190080)
	/go/src/github.com/openshift/image-registry/pkg/cmd/dockerregistry/dockerregistry.go:162 +0x9c3
main.main()
	/go/src/github.com/openshift/image-registry/cmd/dockerregistry/main.go:56 +0x403
Comment 2 Oleg Bulatov 2019-03-07 11:24:57 UTC 
This has been fixed by [1], but for some reason it isn't in the release.

[1]: https://github.com/openshift/image-registry/pull/162
Comment 3 Oleg Bulatov 2019-03-07 11:27:11 UTC 
The first release with the fix is registry.svc.ci.openshift.org/ocp/release:4.0.0-0.nightly-2019-03-06-095230
Comment 4 W. Trevor King 2019-03-07 19:24:14 UTC 
> This has been fixed by [1], but for some reason it isn't in the release.
>
> [1]: https://github.com/openshift/image-registry/pull/162

Because that PR landed 2019-03-05T15:44Z [1] and the release was cut from a 2019-03-05T18:33:12Z build [2].

[1]: https://github.com/openshift/image-registry/pull/162#event-2181367399
[2]: https://github.com/openshift/installer/releases/tag/v0.14.0
Comment 5 W. Trevor King 2019-03-07 19:25:33 UTC 
> Because that PR landed 2019-03-05T15:44Z [1] and the release was cut from a 2019-03-05T18:33:12Z build [2].

And obviously I can't do math, since 15 < 18 ;).  I dunno, maybe ART just didn't fast-forward their tag?  Or maybe the build timestamps are generated at the end of the build cycle instead of the beginning?
Comment 8 Johnny Liu 2019-03-15 07:25:48 UTC 
Verified this bug with 4.0.0-0.nightly-2019-03-14-212114, and PASS.


[root@preserve-jialiu-ansible 20190315]# oc get po -n openshift-image-registry
NAME                                               READY   STATUS    RESTARTS   AGE
cluster-image-registry-operator-5bd768db7c-xthfs   1/1     Running   0          28m
image-registry-5646c8d48-zkbgl                     1/1     Running   0          27m
node-ca-bn4gd                                      1/1     Running   0          27m
node-ca-hwxt9                                      1/1     Running   0          27m
node-ca-mkxnd                                      1/1     Running   0          27m
node-ca-v9fjk                                      1/1     Running   0          26m

[root@preserve-jialiu-ansible 20190315]# oc get machinesets.machine.openshift.io --all-namespaces
NAMESPACE               NAME                                 DESIRED   CURRENT   READY   AVAILABLE   AGE
openshift-machine-api   qe-jialiu1-ghd86-worker-eu-west-3a   1         1         1       1           33m
openshift-machine-api   qe-jialiu1-ghd86-worker-eu-west-3b   0         0                             33m
openshift-machine-api   qe-jialiu1-ghd86-worker-eu-west-3c   0         0                             33m
Comment 10 errata-xmlrpc 2019-06-04 10:45:17 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758