Description of problem: The OpenShift 3.2 registry is missing the Google Cloud Storage (GCS) backend. I spoke with clayton and this was a simple oversight. Version-Release number of selected component (if applicable): The registry currently deployed in dev-preview-int is: registry.qe.openshift.com/openshift3/ose-docker-registry v3.2.0.16 9340a6bd7ef2 How reproducible: Very, it's missing. Steps to Reproduce: 1. Ask Clayton. He knows the details. Actual results: It's missing. Expected results: It should be there.
storage driver enabled here: https://github.com/openshift/origin/pull/8692
tag v3.2.0.41 and higher
More info: https://github.com/openshift/origin/issues/8826
Fixed in https://github.com/openshift/origin/pull/9211
Fixed in master by https://github.com/openshift/origin/pull/9211 and backported to 3.2.1 with https://github.com/openshift/ose/pull/271/commits.
Re-test this bug with atomic-openshift-3.2.1.4-1.git.0.9fe156c.el7.x86_64 and ose-docker-registry:v3.2.1.4 (df8d5ff42e01), failed with the same error as comment 9. # oc logs docker-registry-5-x8sp0 time="2016-07-04T03:34:27.769208666-04:00" level=info msg="version=v2.1.0+unknown" panic: StorageDriver not registered: gcs goroutine 1 [running]: github.com/docker/distribution/registry/handlers.NewApp(0x7f6037fb78d8, 0x2101a40, 0xc2082e0280, 0x7f6037fb78d8) /builddir/build/BUILD/atomic-openshift-git-0.9fe156c/_thirdpartyhacks/src/github.com/docker/distribution/registry/handlers/app.go:105 +0x3e1 github.com/openshift/origin/pkg/cmd/dockerregistry.Execute(0x7f6037faa798, 0xc20802c6e8) /builddir/build/BUILD/atomic-openshift-git-0.9fe156c/_build/src/github.com/openshift/origin/pkg/cmd/dockerregistry/dockerregistry.go:60 +0x4d0 main.main() /builddir/build/BUILD/atomic-openshift-git-0.9fe156c/_build/src/github.com/openshift/origin/cmd/dockerregistry/main.go:51 +0x3ea goroutine 5 [chan receive]: github.com/golang/glog.(*loggingT).flushDaemon(0x2102460) /builddir/build/BUILD/atomic-openshift-git-0.9fe156c/_thirdpartyhacks/src/github.com/golang/glog/glog.go:879 +0x78 created by github.com/golang/glog.init·1 /builddir/build/BUILD/atomic-openshift-git-0.9fe156c/_thirdpartyhacks/src/github.com/golang/glog/glog.go:410 +0x2a7 goroutine 17 [syscall, locked to thread]: runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:2232 +0x1 goroutine 13 [syscall]: os/signal.loop() /usr/lib/golang/src/os/signal/signal_unix.go:21 +0x1f created by os/signal.init·1 /usr/lib/golang/src/os/signal/signal_unix.go:27 +0x35
The dockerregistry binary that is put into the image needs to be built with the Go build tag. build-cross.sh expects readonly OS_IMAGE_COMPILE_GOFLAGS="-tags include_gcs" If that is not being used when the necessary binaries are brew built, GCS will not be enabled in the final image.
Origin: https://github.com/openshift/origin/pull/10000 OSE 3.2: https://github.com/openshift/ose/issues/315
I tested this build today and was able to get passed the go panic error. I attempted to test the registry by doing a simple build and push and received the following error: I0727 17:52:32.343262 1 sti.go:334] Successfully built kwoodsontest/ruby-hello-world-4:ffd94ac6 I0727 17:52:32.368288 1 cleanup.go:23] Removing temporary directory /tmp/s2i-build129614529 I0727 17:52:32.368327 1 fs.go:156] Removing directory '/tmp/s2i-build129614529' I0727 17:52:32.375616 1 sti.go:268] Using provided push secret for pushing 172.30.56.234:5000/kwoodsontest/ruby-hello-world:latest image I0727 17:52:32.375635 1 sti.go:272] Pushing 172.30.56.234:5000/kwoodsontest/ruby-hello-world:latest image ... I0727 17:52:38.139585 1 sti.go:277] Registry server Address: I0727 17:52:38.139747 1 sti.go:278] Registry server User Name: serviceaccount I0727 17:52:38.139758 1 sti.go:279] Registry server Email: serviceaccount I0727 17:52:38.139765 1 sti.go:284] Registry server Password: <<non-empty>> F0727 17:52:38.139774 1 builder.go:204] Error: build error: Failed to push image. Response from registry is: Received unexpected HTTP status: 500 Internal Server Error Here is my registry-config: version: 0.1 log: level: debug http: addr: :5000 storage: cache: layerinfo: inmemory delete: enabled: true gcs: bucket: <redacted> keyfile: /etc/registry_creds/creds.json rootdirectory: /registry auth: openshift: realm: openshift middleware: repository: - name: openshift options: pullthrough: true Any info regarding this would be appreciated.
Kenny, could you please provide registry log from the time of failed push?
Test aganist # openshift version openshift v3.2.1.12 kubernetes v1.2.0-36-g4a3f9c5 etcd 2.2.5 Met the same error as comment 15. Retrying in 1 seconds Retrying in 1 seconds Retrying in 1 seconds Retrying in 1 seconds I0728 03:32:10.942189 1 sti.go:277] Registry server Address: I0728 03:32:10.942209 1 sti.go:278] Registry server User Name: serviceaccount I0728 03:32:10.942217 1 sti.go:279] Registry server Email: serviceaccount I0728 03:32:10.942224 1 sti.go:284] Registry server Password: <<non-empty>> F0728 03:32:10.942273 1 builder.go:204] Error: build error: Failed to push image. Response from registry is: Received unexpected HTTP status: 500 Internal Server Error
Copy&pasting error from Gan's registry log: time="2016-07-28T03:32:11.107829057-04:00" level=error msg="response completed with error" err.code=UNKNOWN err.detail="gcs: googleapi: Error 403: Forbidden, forbidden" Related operation is a simple stat on the uploaded data blob: time="2016-07-28T03:32:11.10764685-04:00" level=debug msg="gcs.Stat(\"/docker/registry/v2/repositories/install-test/cakephp-example/_uploads/274054d6-f306-44eb-b9a4-834e27e65ab9/data\")" It seems that the registry is either badly configured or the user is not authorized to store anything there? Is it owned by someone else? Does the owner of the bucket need to set read-write permissions on /docker/registry to the pusher?
Thanks Michal for you reply. I pinged Michal on IRC and we were able to get this working. As Michal stated in comment #19 this was caused by a permission issue. After running docker logs <containerid> I was able to find these logs: time="2016-07-28T09:04:27.722237404-04:00" level=error msg="error canceling upload after error: gcs: googleapi: Error 403: Forbidden, forbidden" go.version=go1.4.2 http.request.host="172.30.56.234:5000" In the gce console under the section IAM, select your service account/user. Select the drop down on the right and grant the sa/user the Storage Object Admin. (Don't forget to click save!) From the docs https://cloud.google.com/iam/docs/managing-policies - description: Full control of GCS objects. name: roles/storage.objectAdmin title: Storage Object Admin I0728 09:36:08.287538 1 sti.go:268] Using provided push secret for pushing 172.30.56.234:5000/kwoodsontest/ruby-hello-world:latest image I0728 09:36:08.287552 1 sti.go:272] Pushing 172.30.56.234:5000/kwoodsontest/ruby-hello-world:latest image ... I0728 09:36:17.134119 1 sti.go:288] Successfully pushed 172.30.56.234:5000/kwoodsontest/ruby-hello-world:latest The only take away is that it would be nice if the build logs included the errors from the docker-registry so that we could see the issue but the answer here is sufficient after minimal digging. Internal server error is somewhat difficult to decipher. Thanks again Michal, Gan, Johnny, and other qe team members.
I also successed to push to the registry after using another service account which has been granted to access the storage. Thanks Michal and Kenny. Move to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1608