Bug 1483845

Summary: ocp 3.7 Pod status is 'ContainerCreating' because 'ose-pod' image version is wrong
Product: OpenShift Container Platform Reporter: ge liu <geliu>
Component: ReleaseAssignee: Justin Pierce <jupierce>
Status: CLOSED CURRENTRELEASE QA Contact: Johnny Liu <jialiu>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.7.0CC: aos-bugs, bmeng, dakini, jialiu, jokerman, jupierce, mifiedle, mmccomas, smunilla, vlaad, wmeng, wzheng, xiuwang, xxia
Target Milestone: ---   
Target Release: 3.7.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
 Story Points: ---
Clone Of: Environment:
Last Closed: 2018-10-08 13:25:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1470003    

Description ge liu 2017-08-22 06:40:43 UTC 
Description of problem:
Install ocp 3.7 successfully, 'oc get pods' with user:'system:admin', and found all pods is in 'ContainerCreating' status,
check the node sevice log on node, got error msg:
****************************************************************************************************
Aug 22 02:09:19 ip-172-18-10-143.ec2.internal atomic-openshift-node[7898]: E0822 02:09:19.073920    7898 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "registry-console-1-deploy_default(2e7fefd6-86f0-11e7-8223-0e42b37815da)" failed: rpc error: code = 2 desc = unable to pull sandbox image "registry.ops.openshift.com/openshift3/ose-pod:v3.7.0-0.104.0": manifest unknown: manifest unknown
****************************************************************************************************
$ # curl https://registry.ops.openshift.com/v2/openshift3/ose-pod/tags/list  | python -m json.tool | grep v3.7
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10048    0 10048    0     0  43275      0 --:--:-- --:--:-- --:--:-- 43310
        "v3.7.0-0.104.0.0",
        "v3.7.0-0.27.0.0",
        "v3.7.0",
        "v3.7"

we could find that the exact image version is: "v3.7.0-0.104.0.0", but pull sandbox image version is: "v3.7.0-0.104.0",  so the version dismatch result this error. 

openshift v3.7.0-0.104.0
kubernetes v1.7.0+695f48a16f
etcd 3.2.1

How reproducible:
Always

Steps to Reproduce:
1. install ocp 3.7
2. oc get pods:
# oc get pods
NAME                        READY     STATUS              RESTARTS   AGE
docker-registry-1-deploy    0/1       ContainerCreating   0          2h
registry-console-1-deploy   0/1       ContainerCreating   0          2h
router-1-deploy             0/1       ContainerCreating   0          2h
3. login node srv and got the error msg as description above: 

unable to pull sandbox image "registry.ops.openshift.com/openshift3/ose-pod:v3.7.0-0.104.0"

4. run curl command and find the image version dismatch
# curl https://registry.ops.openshift.com/v2/openshift3/ose-pod/tags/list  | python -m json.tool | grep v3.7
"v3.7.0-0.104.0.0"

Actual results:
registry pods in abormal status
Expected results:
registry pods in running status
Comment 2 Justin Pierce 2017-08-28 21:16:27 UTC 
This should be addressed as of build v3.7.0-0.117.0 . Thanks!
Comment 20 ge liu 2017-10-09 05:37:57 UTC 
Tried installation on GCE env with ocp build below, this problem fixed already.

openshift v3.7.0-0.143.1
kubernetes v1.7.0+80709908fd
etcd 3.2.1