Bug 1804160 - The server responds with "500 Internal Server Error" when trying to push an image to a non-existing namespace
Summary: The server responds with "500 Internal Server Error" when trying to push an i...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 4.3.z
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.5.0
Assignee: Ricardo Maraschini
QA Contact: Wenjing Zheng
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-02-18 11:19 UTC by Udi
Modified: 2020-07-13 17:16 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Pushing an image to a namespace that does not exist. Consequence: Image registry was returning a 500 error code. Fix: Changed the return code to indicate the lack of permissions. Result: When pushing images to a namespace that does not exist a permission denied error is returned.
Clone Of:
Environment:
Last Closed: 2020-07-13 17:15:41 UTC
Target Upstream Version:


Attachments (Terms of Use)
registry pod's logs (601.24 KB, text/plain)
2020-02-18 20:44 UTC, Udi
no flags Details


Links
System ID Priority Status Summary Last Updated
Github openshift cluster-image-registry-operator pull 521 None closed Bug 1804160: Allow default service account to Get() namespaces 2020-08-31 05:59:05 UTC
Github openshift image-registry pull 234 None closed Bug 1804160: Checking for namespace existence 2020-08-31 05:59:05 UTC
Github openshift image-registry pull 235 None closed Bug 1804160: Permission denied if namespace does not exist 2020-08-31 05:59:04 UTC
Red Hat Product Errata RHBA-2020:2409 None None None 2020-07-13 17:16:10 UTC

Description Udi 2020-02-18 11:19:56 UTC
Description of problem:
I built a container and tagged it, then tried to push it with the command:
podman push image-registry.openshift-image-registry.svc:5000/keepgoing/centos

I got back a 500 Internal Server Error which didn't give any clue for what the root cause is:

Error: Error copying image to the remote destination: Uploading manifest failed, attempted the following formats: application/vnd.oci.image.manifest.v1+json(Error writing manifest: Error uploading manifest latest to image-registry.openshift-image-registry.svc:5000/keepgoing/centos: manifest invalid: manifest invalid), application/vnd.docker.distribution.manifest.v2+json(Error writing manifest: Error uploading manifest latest to image-registry.openshift-image-registry.svc:5000/keepgoing/centos: received unexpected HTTP status: 500 Internal Server Error) ....

Only after carefully looking in the registry pod's logs, I saw that the namespace I was trying to push to doesn't exist. The server should not respond with a 500 Internal Error in this case.


Version-Release number of selected component (if applicable):
4.3.1


How reproducible:
100%


Steps to Reproduce:
1. I worked on one of the master nodes, to have access to push to the internal registry. My cluster is installed on libvirt to simulate bare metals.
2. I created a simple container based on centos.
3. I tagged the container appropriately to push to the internal registry.
4. The push command: podman push image-registry.openshift-image-registry.svc:5000/keepgoing/centos


Actual results:
500 Internal Server Error


Expected results:
The server should respond with a more specific error message of what the root cause of the failure is, and not result in such a "500 Internal Server Error".

Comment 2 Oleg Bulatov 2020-02-18 20:28:41 UTC
Please attach the registry pod's logs.

Comment 3 Udi 2020-02-18 20:44:11 UTC
Created attachment 1663890 [details]
registry pod's logs

Attaching the logs

Comment 7 Wenjing Zheng 2020-05-12 08:50:23 UTC
Verified on 4.5.0-0.nightly-2020-05-10-180138:
$ docker push default-route-openshift-image-registry.apps.jima-ipishared.qe.devcluster.openshift.com/invalid/myimage
The push refers to a repository [default-route-openshift-image-registry.apps.jima-ipishared.qe.devcluster.openshift.com/invalid/myimage]
5b0d2d635df8: Layer already exists 
denied
$ oc logs pods/image-registry-55c76b59b9-6cmxk | grep denied
time="2020-05-12T08:46:08.689475393Z" level=error msg="manifestService.Put: imagestreammapping got access denied for image invalid/myimage@sha256:a2490cec4484ee6c1068ba3a05f89934010c85242f736280b35343483b2264b6: ImageStream:Forbidden: CreateImageStreamMapping: error creating invalid/myimage ImageStreamMapping: namespaces \"invalid\" not found" go.version=go1.13.4 http.request.contenttype=application/vnd.docker.distribution.manifest.v2+json http.request.host=default-route-openshift-image-registry.apps.jima-ipishared.qe.devcluster.openshift.com http.request.id=ab8d502e-5ca4-40f7-a0a1-eae274643e8c http.request.method=PUT http.request.remoteaddr=66.187.233.202 http.request.uri=/v2/invalid/myimage/manifests/latest http.request.useragent="docker/1.13.1 go/go1.10.3 kernel/3.10.0-1060.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))" openshift.auth.user="system:serviceaccount:wzheng1:registry" vars.name=invalid/myimage vars.reference=latest
time="2020-05-12T08:46:08.68958593Z" level=error msg="response completed with error" err.code=denied err.message="requested access to the resource is denied" go.version=go1.13.4 http.request.contenttype=application/vnd.docker.distribution.manifest.v2+json http.request.host=default-route-openshift-image-registry.apps.jima-ipishared.qe.devcluster.openshift.com http.request.id=ab8d502e-5ca4-40f7-a0a1-eae274643e8c http.request.method=PUT http.request.remoteaddr=66.187.233.202 http.request.uri=/v2/invalid/myimage/manifests/latest http.request.useragent="docker/1.13.1 go/go1.10.3 kernel/3.10.0-1060.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=73.337921ms http.response.status=403 http.response.written=86 openshift.auth.user="system:serviceaccount:wzheng1:registry" vars.name=invalid/myimage vars.reference=latest
time="2020-05-12T08:46:12.533040173Z" level=error msg="manifestService.Put: imagestreammapping got access denied for image invalid/myimage@sha256:f0fdd92f1dbc78a8f113cf251ef1962e7cb864234f0e67e921ae4fa3390f6f04: ImageStream:Forbidden: CreateImageStreamMapping: error creating invalid/myimage ImageStreamMapping: namespaces \"invalid\" not found" go.version=go1.13.4 http.request.contenttype=application/vnd.docker.distribution.manifest.v1+prettyjws http.request.host=default-route-openshift-image-registry.apps.jima-ipishared.qe.devcluster.openshift.com http.request.id=829d67f7-5d09-43a7-913b-b139eb883c93 http.request.method=PUT http.request.remoteaddr=66.187.233.202 http.request.uri=/v2/invalid/myimage/manifests/latest http.request.useragent="docker/1.13.1 go/go1.10.3 kernel/3.10.0-1060.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))" openshift.auth.user="system:serviceaccount:wzheng1:registry" vars.name=invalid/myimage vars.reference=latest
time="2020-05-12T08:46:12.533180662Z" level=error msg="response completed with error" err.code=denied err.message="requested access to the resource is denied" go.version=go1.13.4 http.request.contenttype=application/vnd.docker.distribution.manifest.v1+prettyjws http.request.host=default-route-openshift-image-registry.apps.jima-ipishared.qe.devcluster.openshift.com http.request.id=829d67f7-5d09-43a7-913b-b139eb883c93 http.request.method=PUT http.request.remoteaddr=66.187.233.202 http.request.uri=/v2/invalid/myimage/manifests/latest http.request.useragent="docker/1.13.1 go/go1.10.3 kernel/3.10.0-1060.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=77.261243ms http.response.status=403 http.response.written=86 openshift.auth.user="system:serviceaccount:wzheng1:registry" vars.name=invalid/myimage vars.reference=latest

Comment 9 errata-xmlrpc 2020-07-13 17:15:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.