Bug 1419984 - Tomcat Quick Start App image pull error
Summary: Tomcat Quick Start App image pull error
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 3.5.0
Hardware: All
OS: All
medium
medium
Target Milestone: ---
: ---
Assignee: Alexey Gladkov
QA Contact: Wei Sun
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-02-07 14:54 UTC by Siva Reddy
Modified: 2017-02-07 16:37 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-02-07 16:37:46 UTC
Target Upstream Version:


Attachments (Terms of Use)
The json file for creating the tomcat pods (30.37 KB, text/plain)
2017-02-07 15:38 UTC, Siva Reddy
no flags Details

Description Siva Reddy 2017-02-07 14:54:20 UTC
Description of problem:
   The tomcat quick start app image pull results in error.

Version-Release number of selected component (if applicable):
openshift v3.5.0.14+20b49d0
kubernetes v1.5.2+43a9be4
etcd 3.1.0

How reproducible:
always

Steps to Reproduce:
1. create an app using the openshift quick start template
   jws30-tomcat8-mongodb-persistent-s2i
2. Note the error on  the pod creation

Actual results:
   the build process fails with "ErrImagePull". It looks like image is missing from the image registries

Expected results:
   The build process should succeed


Additional info:

Comment 1 kconner 2017-02-07 15:05:42 UTC
The information in this report seems very sparse, not really sure what to do with this.

Can we have some actionable information such as logs etc.?

Thanks

Comment 2 Ben Parees 2017-02-07 15:08:36 UTC
Siva, it built ok for me using the imagestream and templates from:

https://raw.githubusercontent.com/openshift/openshift-ansible/master/roles/openshift_examples/files/examples/v1.5/xpaas-templates/jws30-tomcat8-mongodb-persistent-s2i.json

https://raw.githubusercontent.com/openshift/openshift-ansible/master/roles/openshift_examples/files/examples/v1.5/xpaas-streams/jboss-image-streams.json

Can you show the build log so we can see exactly what image it was trying to pull?

Also the buildconfig yaml would be helpful.

for reference, here is my successful pull:
Pulling image "registry.access.redhat.com/jboss-webserver-3/webserver30-tomcat8-openshift@sha256:af61377003ae10156bab89724cd16947565ce4a4f50386a2e5d57ab9f50caa22"

Comment 3 Siva Reddy 2017-02-07 15:38:33 UTC
Created attachment 1248436 [details]
The json file for creating the tomcat pods

Comment 4 Siva Reddy 2017-02-07 15:41:46 UTC
Ben,

  the pods are created by:
oc process -f tomcat8-mongodb-pv.json | oc create -f -

and the logs are:
# oc get pods\
> 
NAME                      READY     STATUS             RESTARTS   AGE
jws-app-1-0kvxf           0/1       ImagePullBackOff   0          2m
jws-app-mongodb-1-cfjdj   1/1       Running            0          2d
# oc logs -f pod/jws-app-1-0kvxf
Error from server (BadRequest): container "jws-app" in pod "jws-app-1-0kvxf" is waiting to start: trying and failing to pull image
# oc get is
NAME      DOCKER REPO                                    TAGS      UPDATED
jws-app   172.25.217.134:5000/tomcat8-mongodb0/jws-app   latest    2 days ago
root@ip-172-31-27-63: ~/schitukugit/svt/openshift_scalability # docker pull is/jws-app
Using default tag: latest
Trying to pull repository registry.ops.openshift.com/is/jws-app ... 
Trying to pull repository registry.access.redhat.com/is/jws-app ... 
Trying to pull repository docker.io/is/jws-app ... 
unauthorized: authentication required
# oc describe pod
assigned jws-app-1-0kvxf to ip-172-31-12-44.us-west-2.compute.internal
  1h            4m              20      {kubelet ip-172-31-12-44.us-west-2.compute.internal}    spec.containers{jws-app}        Normal          Pulling         pulling image "172.25.217.134:5000/tomcat8-mongodb0/jws-app@sha256:9266003f3fd82fc6c6de0a6700d3b3e131cc41e580b941c9123a82587d6959a5"
  1h            4m              20      {kubelet ip-172-31-12-44.us-west-2.compute.internal}    spec.containers{jws-app}        Warning         Failed          Failed to pull image "172.25.217.134:5000/tomcat8-mongodb0/jws-app@sha256:9266003f3fd82fc6c6de0a6700d3b3e131cc41e580b941c9123a82587d6959a5": manifest unknown: manifest unknown
  1h            4m              20      {kubelet ip-172-31-12-44.us-west-2.compute.internal}                                    Warning         FailedSync      Error syncing pod, skipping: failed to "StartContainer" for "jws-app" with ErrImagePull: "manifest unknown: manifest unknown"

  1h    11s     357     {kubelet ip-172-31-12-44.us-west-2.compute.internal}    spec.containers{jws-app}        Normal  BackOff         Back-off pulling image "172.25.217.134:5000/tomcat8-mongodb0/jws-app@sha256:9266003f3fd82fc6c6de0a6700d3b3e131cc41e580b941c9123a82587d6959a5"
  1h    11s     357     {kubelet ip-172-31-12-44.us-west-2.compute.internal}                                    Warning FailedSync      Error syncing pod, skipping: failed to "StartContainer" for "jws-app" with ImagePullBackOff: "Back-off pulling image \"172.25.217.134:5000/tomcat8-mongodb0/jws-app@sha256:9266003f3fd82fc6c6de0a6700d3b3e131cc41e580b941c9123a82587d6959a5\""

Comment 5 kconner 2017-02-07 15:46:30 UTC
If you are pulling jws-app then this should be the result of the S2I process rather than the product image.

Has something changed in this area in the OCP 3.5 codebase?

Comment 6 Michal Fojtik 2017-02-07 15:52:38 UTC
What version of Docker you use?

Comment 7 Siva Reddy 2017-02-07 15:54:15 UTC
Docker version:

 # docker version
Client:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-common-1.12.6-1.el7.x86_64
 Go version:      go1.7.4
 Git commit:      037a2f5/1.12.6
 Built:           Wed Jan 18 10:37:44 2017
 OS/Arch:         linux/amd64

Server:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-common-1.12.6-1.el7.x86_64
 Go version:      go1.7.4
 Git commit:      037a2f5/1.12.6
 Built:           Wed Jan 18 10:37:44 2017
 OS/Arch:         linux/amd64

Comment 8 Michal Minar 2017-02-07 16:16:57 UTC
Thanks, that's one possibility down, would it be also possible to get logs from internal registry?

This is a wild guess, but do you have the registry backed by persistent storage (which is a must)?

Can you please also `os describe is -n tomcat8-mongodb0 jws-app`?

Comment 9 Siva Reddy 2017-02-07 16:22:03 UTC
# oc describe is -n tomcat8-mongodb0 jws-app                                        Name:                   jws-app
Namespace:              tomcat8-mongodb0
Created:                2 days ago
Labels:                 application=jws-app
                        template=jws30-tomcat8-mongodb-persistent-s2i
                        xpaas=1.2.0
Annotations:            <none>
Docker Pull Spec:       172.25.217.134:5000/tomcat8-mongodb0/jws-app
Unique Images:          2
Tags:                   1

latest
  pushed image

    172.25.217.134:5000/tomcat8-mongodb0/jws-app@sha256:9266003f3fd82fc6c6de0a6700d3b3e131cc41e580b941c9123a82587d6959a5
      2 days ago

Comment 10 Siva Reddy 2017-02-07 16:22:56 UTC
The registry is backed by S3 this being an AWS instance.

Comment 11 Siva Reddy 2017-02-07 16:37:46 UTC
Hi Michal & Michal,

   Figured out the root cause and now I'm closing the bug. The S3 storage that backs the registry get purged everyday to conserve space on this expensive storage unit by a cron job. So the images got purged and hence not able to pull.

Thanks,
Siva.


Note You need to log in before you can comment on or make changes to this bug.