Bug 1419984

The information in this report seems very sparse, not really sure what to do with this.

Can we have some actionable information such as logs etc.?

Thanks

Siva, it built ok for me using the imagestream and templates from:

https://raw.githubusercontent.com/openshift/openshift-ansible/master/roles/openshift_examples/files/examples/v1.5/xpaas-templates/jws30-tomcat8-mongodb-persistent-s2i.json

https://raw.githubusercontent.com/openshift/openshift-ansible/master/roles/openshift_examples/files/examples/v1.5/xpaas-streams/jboss-image-streams.json

Can you show the build log so we can see exactly what image it was trying to pull?

Also the buildconfig yaml would be helpful.

for reference, here is my successful pull:
Pulling image "registry.access.redhat.com/jboss-webserver-3/webserver30-tomcat8-openshift@sha256:af61377003ae10156bab89724cd16947565ce4a4f50386a2e5d57ab9f50caa22"

Created attachment 1248436 [details]
The json file for creating the tomcat pods

Ben,

  the pods are created by:
oc process -f tomcat8-mongodb-pv.json | oc create -f -

and the logs are:
# oc get pods\
> 
NAME                      READY     STATUS             RESTARTS   AGE
jws-app-1-0kvxf           0/1       ImagePullBackOff   0          2m
jws-app-mongodb-1-cfjdj   1/1       Running            0          2d
# oc logs -f pod/jws-app-1-0kvxf
Error from server (BadRequest): container "jws-app" in pod "jws-app-1-0kvxf" is waiting to start: trying and failing to pull image
# oc get is
NAME      DOCKER REPO                                    TAGS      UPDATED
jws-app   172.25.217.134:5000/tomcat8-mongodb0/jws-app   latest    2 days ago
root@ip-172-31-27-63: ~/schitukugit/svt/openshift_scalability # docker pull is/jws-app
Using default tag: latest
Trying to pull repository registry.ops.openshift.com/is/jws-app ... 
Trying to pull repository registry.access.redhat.com/is/jws-app ... 
Trying to pull repository docker.io/is/jws-app ... 
unauthorized: authentication required
# oc describe pod
assigned jws-app-1-0kvxf to ip-172-31-12-44.us-west-2.compute.internal
  1h            4m              20      {kubelet ip-172-31-12-44.us-west-2.compute.internal}    spec.containers{jws-app}        Normal          Pulling         pulling image "172.25.217.134:5000/tomcat8-mongodb0/jws-app@sha256:9266003f3fd82fc6c6de0a6700d3b3e131cc41e580b941c9123a82587d6959a5"
  1h            4m              20      {kubelet ip-172-31-12-44.us-west-2.compute.internal}    spec.containers{jws-app}        Warning         Failed          Failed to pull image "172.25.217.134:5000/tomcat8-mongodb0/jws-app@sha256:9266003f3fd82fc6c6de0a6700d3b3e131cc41e580b941c9123a82587d6959a5": manifest unknown: manifest unknown
  1h            4m              20      {kubelet ip-172-31-12-44.us-west-2.compute.internal}                                    Warning         FailedSync      Error syncing pod, skipping: failed to "StartContainer" for "jws-app" with ErrImagePull: "manifest unknown: manifest unknown"

  1h    11s     357     {kubelet ip-172-31-12-44.us-west-2.compute.internal}    spec.containers{jws-app}        Normal  BackOff         Back-off pulling image "172.25.217.134:5000/tomcat8-mongodb0/jws-app@sha256:9266003f3fd82fc6c6de0a6700d3b3e131cc41e580b941c9123a82587d6959a5"
  1h    11s     357     {kubelet ip-172-31-12-44.us-west-2.compute.internal}                                    Warning FailedSync      Error syncing pod, skipping: failed to "StartContainer" for "jws-app" with ImagePullBackOff: "Back-off pulling image \"172.25.217.134:5000/tomcat8-mongodb0/jws-app@sha256:9266003f3fd82fc6c6de0a6700d3b3e131cc41e580b941c9123a82587d6959a5\""

If you are pulling jws-app then this should be the result of the S2I process rather than the product image.

Has something changed in this area in the OCP 3.5 codebase?

What version of Docker you use?

Docker version:

 # docker version
Client:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-common-1.12.6-1.el7.x86_64
 Go version:      go1.7.4
 Git commit:      037a2f5/1.12.6
 Built:           Wed Jan 18 10:37:44 2017
 OS/Arch:         linux/amd64

Server:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-common-1.12.6-1.el7.x86_64
 Go version:      go1.7.4
 Git commit:      037a2f5/1.12.6
 Built:           Wed Jan 18 10:37:44 2017
 OS/Arch:         linux/amd64

Thanks, that's one possibility down, would it be also possible to get logs from internal registry?

This is a wild guess, but do you have the registry backed by persistent storage (which is a must)?

Can you please also `os describe is -n tomcat8-mongodb0 jws-app`?

# oc describe is -n tomcat8-mongodb0 jws-app                                        Name:                   jws-app
Namespace:              tomcat8-mongodb0
Created:                2 days ago
Labels:                 application=jws-app
                        template=jws30-tomcat8-mongodb-persistent-s2i
                        xpaas=1.2.0
Annotations:            <none>
Docker Pull Spec:       172.25.217.134:5000/tomcat8-mongodb0/jws-app
Unique Images:          2
Tags:                   1

latest
  pushed image

    172.25.217.134:5000/tomcat8-mongodb0/jws-app@sha256:9266003f3fd82fc6c6de0a6700d3b3e131cc41e580b941c9123a82587d6959a5
      2 days ago

The registry is backed by S3 this being an AWS instance.

Hi Michal & Michal,

   Figured out the root cause and now I'm closing the bug. The S3 storage that backs the registry get purged everyday to conserve space on this expensive storage unit by a cron job. So the images got purged and hence not able to pull.

Thanks,
Siva.