Description of problem: I built a container and tagged it, then tried to push it with the command: podman push image-registry.openshift-image-registry.svc:5000/keepgoing/centos I got back a 500 Internal Server Error which didn't give any clue for what the root cause is: Error: Error copying image to the remote destination: Uploading manifest failed, attempted the following formats: application/vnd.oci.image.manifest.v1+json(Error writing manifest: Error uploading manifest latest to image-registry.openshift-image-registry.svc:5000/keepgoing/centos: manifest invalid: manifest invalid), application/vnd.docker.distribution.manifest.v2+json(Error writing manifest: Error uploading manifest latest to image-registry.openshift-image-registry.svc:5000/keepgoing/centos: received unexpected HTTP status: 500 Internal Server Error) .... Only after carefully looking in the registry pod's logs, I saw that the namespace I was trying to push to doesn't exist. The server should not respond with a 500 Internal Error in this case. Version-Release number of selected component (if applicable): 4.3.1 How reproducible: 100% Steps to Reproduce: 1. I worked on one of the master nodes, to have access to push to the internal registry. My cluster is installed on libvirt to simulate bare metals. 2. I created a simple container based on centos. 3. I tagged the container appropriately to push to the internal registry. 4. The push command: podman push image-registry.openshift-image-registry.svc:5000/keepgoing/centos Actual results: 500 Internal Server Error Expected results: The server should respond with a more specific error message of what the root cause of the failure is, and not result in such a "500 Internal Server Error".
Please attach the registry pod's logs.
Created attachment 1663890 [details] registry pod's logs Attaching the logs
Verified on 4.5.0-0.nightly-2020-05-10-180138: $ docker push default-route-openshift-image-registry.apps.jima-ipishared.qe.devcluster.openshift.com/invalid/myimage The push refers to a repository [default-route-openshift-image-registry.apps.jima-ipishared.qe.devcluster.openshift.com/invalid/myimage] 5b0d2d635df8: Layer already exists denied $ oc logs pods/image-registry-55c76b59b9-6cmxk | grep denied time="2020-05-12T08:46:08.689475393Z" level=error msg="manifestService.Put: imagestreammapping got access denied for image invalid/myimage@sha256:a2490cec4484ee6c1068ba3a05f89934010c85242f736280b35343483b2264b6: ImageStream:Forbidden: CreateImageStreamMapping: error creating invalid/myimage ImageStreamMapping: namespaces \"invalid\" not found" go.version=go1.13.4 http.request.contenttype=application/vnd.docker.distribution.manifest.v2+json http.request.host=default-route-openshift-image-registry.apps.jima-ipishared.qe.devcluster.openshift.com http.request.id=ab8d502e-5ca4-40f7-a0a1-eae274643e8c http.request.method=PUT http.request.remoteaddr=66.187.233.202 http.request.uri=/v2/invalid/myimage/manifests/latest http.request.useragent="docker/1.13.1 go/go1.10.3 kernel/3.10.0-1060.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))" openshift.auth.user="system:serviceaccount:wzheng1:registry" vars.name=invalid/myimage vars.reference=latest time="2020-05-12T08:46:08.68958593Z" level=error msg="response completed with error" err.code=denied err.message="requested access to the resource is denied" go.version=go1.13.4 http.request.contenttype=application/vnd.docker.distribution.manifest.v2+json http.request.host=default-route-openshift-image-registry.apps.jima-ipishared.qe.devcluster.openshift.com http.request.id=ab8d502e-5ca4-40f7-a0a1-eae274643e8c http.request.method=PUT http.request.remoteaddr=66.187.233.202 http.request.uri=/v2/invalid/myimage/manifests/latest http.request.useragent="docker/1.13.1 go/go1.10.3 kernel/3.10.0-1060.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=73.337921ms http.response.status=403 http.response.written=86 openshift.auth.user="system:serviceaccount:wzheng1:registry" vars.name=invalid/myimage vars.reference=latest time="2020-05-12T08:46:12.533040173Z" level=error msg="manifestService.Put: imagestreammapping got access denied for image invalid/myimage@sha256:f0fdd92f1dbc78a8f113cf251ef1962e7cb864234f0e67e921ae4fa3390f6f04: ImageStream:Forbidden: CreateImageStreamMapping: error creating invalid/myimage ImageStreamMapping: namespaces \"invalid\" not found" go.version=go1.13.4 http.request.contenttype=application/vnd.docker.distribution.manifest.v1+prettyjws http.request.host=default-route-openshift-image-registry.apps.jima-ipishared.qe.devcluster.openshift.com http.request.id=829d67f7-5d09-43a7-913b-b139eb883c93 http.request.method=PUT http.request.remoteaddr=66.187.233.202 http.request.uri=/v2/invalid/myimage/manifests/latest http.request.useragent="docker/1.13.1 go/go1.10.3 kernel/3.10.0-1060.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))" openshift.auth.user="system:serviceaccount:wzheng1:registry" vars.name=invalid/myimage vars.reference=latest time="2020-05-12T08:46:12.533180662Z" level=error msg="response completed with error" err.code=denied err.message="requested access to the resource is denied" go.version=go1.13.4 http.request.contenttype=application/vnd.docker.distribution.manifest.v1+prettyjws http.request.host=default-route-openshift-image-registry.apps.jima-ipishared.qe.devcluster.openshift.com http.request.id=829d67f7-5d09-43a7-913b-b139eb883c93 http.request.method=PUT http.request.remoteaddr=66.187.233.202 http.request.uri=/v2/invalid/myimage/manifests/latest http.request.useragent="docker/1.13.1 go/go1.10.3 kernel/3.10.0-1060.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=77.261243ms http.response.status=403 http.response.written=86 openshift.auth.user="system:serviceaccount:wzheng1:registry" vars.name=invalid/myimage vars.reference=latest
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409