Bug 1804160

Summary: The server responds with "500 Internal Server Error" when trying to push an image to a non-existing namespace
Product: OpenShift Container Platform Reporter: Udi Kalifon <ukalifon>
Component: Image RegistryAssignee: Ricardo Maraschini <rmarasch>
Status: CLOSED ERRATA QA Contact: Wenjing Zheng <wzheng>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.3.zCC: adam.kaplan, aos-bugs, obulatov, rmarasch
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Pushing an image to a namespace that does not exist. Consequence: Image registry was returning a 500 error code. Fix: Changed the return code to indicate the lack of permissions. Result: When pushing images to a namespace that does not exist a permission denied error is returned.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2020-07-13 17:15:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
registry pod's logs none

Description Udi Kalifon 2020-02-18 11:19:56 UTC 
Description of problem:
I built a container and tagged it, then tried to push it with the command:
podman push image-registry.openshift-image-registry.svc:5000/keepgoing/centos

I got back a 500 Internal Server Error which didn't give any clue for what the root cause is:

Error: Error copying image to the remote destination: Uploading manifest failed, attempted the following formats: application/vnd.oci.image.manifest.v1+json(Error writing manifest: Error uploading manifest latest to image-registry.openshift-image-registry.svc:5000/keepgoing/centos: manifest invalid: manifest invalid), application/vnd.docker.distribution.manifest.v2+json(Error writing manifest: Error uploading manifest latest to image-registry.openshift-image-registry.svc:5000/keepgoing/centos: received unexpected HTTP status: 500 Internal Server Error) ....

Only after carefully looking in the registry pod's logs, I saw that the namespace I was trying to push to doesn't exist. The server should not respond with a 500 Internal Error in this case.


Version-Release number of selected component (if applicable):
4.3.1


How reproducible:
100%


Steps to Reproduce:
1. I worked on one of the master nodes, to have access to push to the internal registry. My cluster is installed on libvirt to simulate bare metals.
2. I created a simple container based on centos.
3. I tagged the container appropriately to push to the internal registry.
4. The push command: podman push image-registry.openshift-image-registry.svc:5000/keepgoing/centos


Actual results:
500 Internal Server Error


Expected results:
The server should respond with a more specific error message of what the root cause of the failure is, and not result in such a "500 Internal Server Error".
Comment 2 Oleg Bulatov 2020-02-18 20:28:41 UTC 
Please attach the registry pod's logs.
Comment 3 Udi Kalifon 2020-02-18 20:44:11 UTC 
Created attachment 1663890 [details]
registry pod's logs

Attaching the logs
Comment 7 Wenjing Zheng 2020-05-12 08:50:23 UTC 
Verified on 4.5.0-0.nightly-2020-05-10-180138:
$ docker push default-route-openshift-image-registry.apps.jima-ipishared.qe.devcluster.openshift.com/invalid/myimage
The push refers to a repository [default-route-openshift-image-registry.apps.jima-ipishared.qe.devcluster.openshift.com/invalid/myimage]
5b0d2d635df8: Layer already exists 
denied
$ oc logs pods/image-registry-55c76b59b9-6cmxk | grep denied
time="2020-05-12T08:46:08.689475393Z" level=error msg="manifestService.Put: imagestreammapping got access denied for image invalid/myimage@sha256:a2490cec4484ee6c1068ba3a05f89934010c85242f736280b35343483b2264b6: ImageStream:Forbidden: CreateImageStreamMapping: error creating invalid/myimage ImageStreamMapping: namespaces \"invalid\" not found" go.version=go1.13.4 http.request.contenttype=application/vnd.docker.distribution.manifest.v2+json http.request.host=default-route-openshift-image-registry.apps.jima-ipishared.qe.devcluster.openshift.com http.request.id=ab8d502e-5ca4-40f7-a0a1-eae274643e8c http.request.method=PUT http.request.remoteaddr=66.187.233.202 http.request.uri=/v2/invalid/myimage/manifests/latest http.request.useragent="docker/1.13.1 go/go1.10.3 kernel/3.10.0-1060.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))" openshift.auth.user="system:serviceaccount:wzheng1:registry" vars.name=invalid/myimage vars.reference=latest
time="2020-05-12T08:46:08.68958593Z" level=error msg="response completed with error" err.code=denied err.message="requested access to the resource is denied" go.version=go1.13.4 http.request.contenttype=application/vnd.docker.distribution.manifest.v2+json http.request.host=default-route-openshift-image-registry.apps.jima-ipishared.qe.devcluster.openshift.com http.request.id=ab8d502e-5ca4-40f7-a0a1-eae274643e8c http.request.method=PUT http.request.remoteaddr=66.187.233.202 http.request.uri=/v2/invalid/myimage/manifests/latest http.request.useragent="docker/1.13.1 go/go1.10.3 kernel/3.10.0-1060.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=73.337921ms http.response.status=403 http.response.written=86 openshift.auth.user="system:serviceaccount:wzheng1:registry" vars.name=invalid/myimage vars.reference=latest
time="2020-05-12T08:46:12.533040173Z" level=error msg="manifestService.Put: imagestreammapping got access denied for image invalid/myimage@sha256:f0fdd92f1dbc78a8f113cf251ef1962e7cb864234f0e67e921ae4fa3390f6f04: ImageStream:Forbidden: CreateImageStreamMapping: error creating invalid/myimage ImageStreamMapping: namespaces \"invalid\" not found" go.version=go1.13.4 http.request.contenttype=application/vnd.docker.distribution.manifest.v1+prettyjws http.request.host=default-route-openshift-image-registry.apps.jima-ipishared.qe.devcluster.openshift.com http.request.id=829d67f7-5d09-43a7-913b-b139eb883c93 http.request.method=PUT http.request.remoteaddr=66.187.233.202 http.request.uri=/v2/invalid/myimage/manifests/latest http.request.useragent="docker/1.13.1 go/go1.10.3 kernel/3.10.0-1060.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))" openshift.auth.user="system:serviceaccount:wzheng1:registry" vars.name=invalid/myimage vars.reference=latest
time="2020-05-12T08:46:12.533180662Z" level=error msg="response completed with error" err.code=denied err.message="requested access to the resource is denied" go.version=go1.13.4 http.request.contenttype=application/vnd.docker.distribution.manifest.v1+prettyjws http.request.host=default-route-openshift-image-registry.apps.jima-ipishared.qe.devcluster.openshift.com http.request.id=829d67f7-5d09-43a7-913b-b139eb883c93 http.request.method=PUT http.request.remoteaddr=66.187.233.202 http.request.uri=/v2/invalid/myimage/manifests/latest http.request.useragent="docker/1.13.1 go/go1.10.3 kernel/3.10.0-1060.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=77.261243ms http.response.status=403 http.response.written=86 openshift.auth.user="system:serviceaccount:wzheng1:registry" vars.name=invalid/myimage vars.reference=latest
Comment 9 errata-xmlrpc 2020-07-13 17:15:41 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409