Created attachment 1450345 [details] Misc information relevant to the case Description of problem: I am trying to deploy the Prometheus pod. I have created ImageStreams for all the containers, setting referencePolicy type to local. When trying to deploy the pod the oauth-proxy image pull fails: Failed to pull image "docker-registry.default.svc:5000/prom4/oauth-proxy:v3.7.46": rpc error: code = Unknown desc = error pulling image configuration: unknown blob In the internal registry I can se the following error message: time="2018-06-12T07:00:50.676641423Z" level=error msg="Error statting blob sha256:323f53d5853ed14ff5613a9955b56ad9ceb1a870a17383e316180054cb6a695f in remote repository \"registry.access.redhat.com/openshift3/oauth-proxy\": missing content-length header for request: https://registry.access.redhat.com/v2/openshift3/oauth-proxy/blobs/sha256:323f53d5853ed14ff5613a9955b56ad9ceb1a870a17383e316180054cb6a695f" go.version=go1.9.4 http.request.host="docker-registry.default.svc:5000" http.request.id=b4a779b0-592f-4243-bb21-4a786d560f2c http.request.method=GET http.request.remoteaddr="10.131.0.1:45574" http.request.uri="/v2/prom4/oauth-proxy/blobs/sha256:323f53d5853ed14ff5613a9955b56ad9ceb1a870a17383e316180054cb6a695f" http.request.useragent="docker/1.13.1 go/go1.9.2 kernel/3.10.0-862.3.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" instance.id=2422639a-5e3a-47f2-9396-dc004a681b51 openshift.auth.user="system:serviceaccount:prom4:prometheus" vars.digest="sha256:323f53d5853ed14ff5613a9955b56ad9ceb1a870a17383e316180054cb6a695f" vars.name=prom4/oauth-proxy I can do a docker pull of oauth-proxy. I can deploy oauth-proxy if I reference the image directly. And all other container images in the pod successfully pulls using pull through. Seems to only be an issue with oauth-proxy. Version-Release number of selected component (if applicable): I have tried with oauth-proxy tags: latest, 3.9.30, 3.7.52, 3.7.47 How reproducible: Create ImageStream for oauth-proxy with the referencePolicy Local. Steps to Reproduce: 1. 2. 3. Actual results: Image is not pulled Expected results: The oauth-proxy Image should be pulled and deployed like the other container images. Additional info: I have attached the IS and a description of it.
Jonas, do you have exact commands to reproduce this? $ oc import-image registry.access.redhat.com/openshift3/oauth-proxy:v3.9.30 --confirm $ docker pull 172.30.1.1:5000/myproject/oauth-proxy:v3.9.30 These commands succeed on 3.10. Also check `curl -IL https://registry.access.redhat.com/v2/openshift3/oauth-proxy/blobs/sha256:323f53d5853ed14ff5613a9955b56ad9ceb1a870a17383e316180054cb6a695f`. Do you get the "Content-Length" header in a response?
I added the IS in the template. So I did not use the import-image command. Did you try to use the oc import-image command with --reference-policy="local" ? Referencing the image directly worked.
I pulled through the internal registry by pulling directly from the internal registry. The option --reference-policy=local means to pull from the internal registry instead of the original registry, nothing more.
I would much appreciate getting steps to reproduce or any additional information that can help reproducing it.
Sorry for the delay. I was testing the oauth-proxy:v.3.9.30 image again to see if I could reproduce the issue and all the sudden it worked. So I am not sure what is happening. I was able to recreate it with a different tag. 1. oc import-image oauth-proxy:v3.7.52 --from=registry.access.redhat.com/openshift3/oauth-proxy:v3.7.52 --confirm --reference-policy='local' 2. oc run oauth-proxy --image=docker-registry.default.svc:5000/test-oauth/oauth-proxy:v3.7.52 --command -- /bin/sh -c "sleep 3600" gives the error Failed to pull image "docker-registry.default.svc:5000/test-oauth/oauth-proxy:v3.7.52": rpc error: code = Unknown desc = error pulling image configuration: unknown blob I have attached logs from docker-registry
Created attachment 1454789 [details] docker-registry logs
The same problem: Error statting blob sha256:bd4e6795a6dc1c42eb54756d5b2f9f1caede46b44a32fd84a6e6fe7af605e2b5 in remote repository "registry.access.redhat.com/openshift3/oauth-proxy": missing content-length header for request: https://registry.access.redhat.com/v2/openshift3/oauth-proxy/blobs/sha256:bd4e6795a6dc1c42eb54756d5b2f9f1caede46b44a32fd84a6e6fe7af605e2b5 But it has the Content-Length header. $ curl -IL https://registry.access.redhat.com/v2/openshift3/oauth-proxy/blobs/sha256:bd4e6795a6dc1c42eb54756d5b2f9f1caede46b44a32fd84a6e6fe7af605e2b5 HTTP/1.1 200 OK ... Content-Length: 3364 ... Looks like flakes on the registry.access.redhat.com side.
Jonas, is this behavior consistent for you or intermittent? Since we cannot reproduce it and the logs/analysis imply an upstream issue in the redhat registry, I would like to close this out.
Jonas is on vacation at the moment but I am able to reproduce this issue consistently in both Minishift, customer env, and a home lab. I am unsure if this is a upstream issue in the redhat registry as it works very well to download via reference-policy set as "source". Changing it to a pullthrough style via "local" is the only time we see this issue. So I think its a problem with the internal openshift registry?
Is it always reproducible? If so, please provide us with the steps to reproduce it. Otherwise it's a bug on the Red Hat registry side: one of the servers behind the load balancer responds to HEAD requests with incomplete set of headers. Docker doesn't send HEAD requests, it explains why only the integrated registry hits this problem.