As far as I can tell something broke between hello-openshift:v0.5 and hello-openshift:v0.5.1 however there's no diff in hello-openshift so it's not the image itself.

Docker logs :

May 21 11:40:26 ose3-master.example.com docker[2124]: time="2015-05-21T11:40:26-04:00" level=info msg="POST /containers/d503d79a4f5b1b1423707ce41652168535ba629ef13593c1d97c84da75251f44/start"
May 21 11:40:26 ose3-master.example.com docker[2124]: time="2015-05-21T11:40:26-04:00" level=info msg="+job start(d503d79a4f5b1b1423707ce41652168535ba629ef13593c1d97c84da75251f44)"
May 21 11:40:26 ose3-master.example.com docker[2124]: time="2015-05-21T11:40:26-04:00" level=info msg="+job log(start, d503d79a4f5b1b1423707ce41652168535ba629ef13593c1d97c84da75251f44, docker.io/openshift/hello-openshift:v0.5.1)"
May 21 11:40:26 ose3-master.example.com docker[2124]: time="2015-05-21T11:40:26-04:00" level=info msg="-job log(start, d503d79a4f5b1b1423707ce41652168535ba629ef13593c1d97c84da75251f44, docker.io/openshift/hello-openshift:v0.5.1) = OK (0)"
May 21 11:40:26 ose3-master.example.com docker[2124]: time="2015-05-21T11:40:26-04:00" level=info msg="GET /containers/d503d79a4f5b1b1423707ce41652168535ba629ef13593c1d97c84da75251f44/json"
May 21 11:40:26 ose3-master.example.com docker[2124]: time="2015-05-21T11:40:26-04:00" level=info msg="+job container_inspect(d503d79a4f5b1b1423707ce41652168535ba629ef13593c1d97c84da75251f44)"
May 21 11:40:26 ose3-master.example.com docker[2124]: time="2015-05-21T11:40:26-04:00" level=warning msg="exit status 1"
May 21 11:40:27 ose3-master.example.com docker[2124]: time="2015-05-21T11:40:27-04:00" level=error msg="Error unmounting device d503d79a4f5b1b1423707ce41652168535ba629ef13593c1d97c84da75251f44: UnmountDevice: device not-mounted id d503d79a4f5b1b1423707ce41652168535ba629ef
13593c1d97c84da75251f44"
May 21 11:40:27 ose3-master.example.com docker[2124]: time="2015-05-21T11:40:27-04:00" level=info msg="+job log(die, d503d79a4f5b1b1423707ce41652168535ba629ef13593c1d97c84da75251f44, docker.io/openshift/hello-openshift:v0.5.1)"
May 21 11:40:27 ose3-master.example.com docker[2124]: time="2015-05-21T11:40:27-04:00" level=info msg="-job log(die, d503d79a4f5b1b1423707ce41652168535ba629ef13593c1d97c84da75251f44, docker.io/openshift/hello-openshift:v0.5.1) = OK (0)"
May 21 11:40:27 ose3-master.example.com docker[2124]: Cannot start container d503d79a4f5b1b1423707ce41652168535ba629ef13593c1d97c84da75251f44: [8] System error: exec: "/hello-openshift": permission denied
May 21 11:40:27 ose3-master.example.com docker[2124]: time="2015-05-21T11:40:27-04:00" level=info msg="-job start(d503d79a4f5b1b1423707ce41652168535ba629ef13593c1d97c84da75251f44) = ERR (1)"
May 21 11:40:27 ose3-master.example.com docker[2124]: time="2015-05-21T11:40:27-04:00" level=error msg="Handler for POST /containers/{name:.*}/start returned error: Cannot start container d503d79a4f5b1b1423707ce41652168535ba629ef13593c1d97c84da75251f44: [8] System error: 
exec: \"/hello-openshift\": permission denied"
May 21 11:40:27 ose3-master.example.com docker[2124]: time="2015-05-21T11:40:27-04:00" level=error msg="HTTP Error: statusCode=500 Cannot start container d503d79a4f5b1b1423707ce41652168535ba629ef13593c1d97c84da75251f44: [8] System error: exec: \"/hello-openshift\": permis
sion denied"
May 21 11:40:27 ose3-master.example.com docker[2124]: time="2015-05-21T11:40:27-04:00" level=info msg="-job container_inspect(d503d79a4f5b1b1423707ce41652168535ba629ef13593c1d97c84da75251f44) = OK (0)"
May 21 11:40:27 ose3-master.example.com docker[2124]: time="2015-05-21T11:40:27-04:00" level=info msg="GET /containers/json?all=1"
May 21 11:40:27 ose3-master.example.com docker[2124]: time="2015-05-21T11:40:27-04:00" level=info msg="+job containers()"
May 21 11:40:27 ose3-master.example.com docker[2124]: time="2015-05-21T11:40:27-04:00" level=info msg="-job containers() = OK (0)"
May 21 11:40:27 ose3-master.example.com docker[2124]: time="2015-05-21T11:40:27-04:00" level=info msg="GET /containers/d503d79a4f5b1b1423707ce41652168535ba629ef13593c1d97c84da75251f44/json"
May 21 11:40:27 ose3-master.example.com docker[2124]: time="2015-05-21T11:40:27-04:00" level=info msg="+job container_inspect(d503d79a4f5b1b1423707ce41652168535ba629ef13593c1d97c84da75251f44)"
May 21 11:40:27 ose3-master.example.com docker[2124]: time="2015-05-21T11:40:27-04:00" level=info msg="-job container_inspect(d503d79a4f5b1b1423707ce41652168535ba629ef13593c1d97c84da75251f44) = OK (0)"

pod json
$ cat hello-pod.json 
{
  "apiVersion":"v1beta3",
  "kind": "Pod",
  "metadata": {
    "name": "hello-openshift"
  },
  "labels": {
    "name": "hello-openshift"
  },
  "spec": {
    "containers": [{
      "name": "hello-openshift",
      "image": "openshift/hello-openshift:v0.5.1",
        "resources": {
          "limits": {
            "cpu": "100m",
            "memory": "16Mi"
        },
        "ports": [{
          "name": "hello-openshift",
          "hostPort": 6061,
          "containerPort": 8080,
          "protocol": "TCP"
        }]
      }
    }]
  }
}

the image is broken, it can't even be docker run:
$ docker run openshift/hello-openshift:latest

Timestamp: 2015-05-21 13:07:23.554348602 -0400 EDT
Code: System error

Message: [/usr/bin/tar -cf /var/lib/docker/tmp/3e5aab92457bfe5d00c316335734fa2ba9a190ee360eb23e8c6c49cbd7360a36061423886/_tmp.tar -C /var/lib/docker/devicemapper/mnt/3e5aab92457bfe5d00c316335734fa2ba9a190ee360eb23e8c6c49cbd7360a36/rootfs/tmp .] failed: /usr/bin/tar: /var/lib/docker/devicemapper/mnt/3e5aab92457bfe5d00c316335734fa2ba9a190ee360eb23e8c6c49cbd7360a36/rootfs/tmp: Cannot open: No such file or directory
/usr/bin/tar: Error is not recoverable: exiting now
: exit status 2

Frames:
---
0: setupRootfs
Package: github.com/docker/libcontainer
File: rootfs_linux.go@30
---
1: Init
Package: github.com/docker/libcontainer.(*linuxStandardInit)
File: standard_init_linux.go@52
---
2: StartInitialization
Package: github.com/docker/libcontainer.(*LinuxFactory)
File: factory_linux.go@223
---
3: initializer
Package: github.com/docker/docker/daemon/execdriver/native
File: init.go@35
---
4: Init
Package: github.com/docker/docker/pkg/reexec
File: reexec.go@26
---
5: main
Package: main
File: docker.go@29
---
6: main
Package: runtime
File: proc.go@63
---
7: goexit
Package: runtime
File: asm_amd64.s@2232
FATA[0002] Error response from daemon: : exit status 2

appears to actually be an issue in docker that prevents the image from running.

The only thing that makes me wonder about that diagnosis is that I get the exact same system error when I `docker run openshift/hello-openshift:v0.5` however it runs fine as a pod, perhaps there's multiple issues here.

The hello-openshift binary in openshift/hello-openshift:v0.5 is not dynamically linked. The binary in the latest image is dynamically linked, which is why running fails. I'm not immediately sure why the latest build is dynamic off the top of my head.

https://github.com/openshift/origin/pull/2700

looks like docker 1.6.2 will fix the /tmp requirement issue but i'm going to update the pod to always mount /tmp in anyway so people on older versions will be fixed.

Re-test this bug with the following version, FAIL.

openshift v0.5.2.2-19-g8dc4a9a
kubernetes v0.17.1-804-g496be63
openshift/hello-openshift:latest (0a10016e3e27)
docker-1.6.2-4.el7.x86_64


$ cat hello-pod.json
{
  "id": "hello-openshift",
  "kind": "Pod",
  "apiVersion":"v1beta2",
  "labels": {
    "name": "hello-openshift"
  },
  "desiredState": {
    "manifest": {
      "version": "v1beta1",
      "id": "hello-openshift",
      "containers": [{
        "name": "hello-openshift",
        "image": "openshift/hello-openshift",
        "ports": [{
          "hostPort": 6061,
          "containerPort": 8080
        }]
      }]
    }
  }
}


$ osc create -f ~/training/beta2/hello-pod.json
pods/hello-openshift

$ osc get po
hello-openshift    10.1.0.14                                                                                          minion1.cluster.local/10.14.6.143   name=hello-openshift                                            Pending   4 seconds   
                               hello-openshift            openshift/hello-openshift                                                                                                                                       Waiting               API error (404): Cannot start container 4062de370898b490825c753fdb58f294a9c8e1d00932fc6cf27c7bb2086cd1d0: [8] System error: no such file or directory

I also tried the latest hello-openshift json file, still fail with the same error.


$ cat hello-pod.json 
{
  "kind": "Pod",
  "apiVersion": "v1beta3",
  "metadata": {
    "name": "hello-openshift",
    "creationTimestamp": null,
    "labels": {
      "name": "hello-openshift"
    }
  },
  "spec": {
    "containers": [
      {
        "name": "hello-openshift",
        "image": "openshift/hello-openshift",
        "ports": [
          {
            "hostPort": 6061,
            "containerPort": 8080,
            "protocol": "TCP"
          }
        ],
        "resources": {},
        "volumeMounts": [
          {
            "name":"tmp",
            "mountPath":"/tmp"
          }
        ],
        "terminationMessagePath": "/dev/termination-log",
        "imagePullPolicy": "IfNotPresent",
        "capabilities": {},
        "securityContext": {
          "capabilities": {},
          "privileged": false
        }
      }
    ],
    "volumes": [
      {
        "name":"tmp",
        "emptyDir": {}
      }
    ],
    "restartPolicy": "Always",
    "dnsPolicy": "ClusterFirst",
    "serviceAccount": ""
  },
  "status": {}
}

$ osc get po
hello-openshift    10.1.0.16                                                                                          minion1.cluster.local/10.14.6.143   name=hello-openshift                                            Pending   5 seconds   
                               hello-openshift            openshift/hello-openshift                                                                                                                                       Waiting               API error (404): Cannot start container ccafb95064ce5038b4c8a9a3da9401ec57b86d01827c13494c8e77500e060475: [8] System error: no such file or directory

it definitely won't work with the beta2 json, you need the updated json that defines the tmp volume.

however it looks like what happened here is i didn't realize we are rebuilding the hello-openshift image as part of the build and it's not being built correctly when that happens, so the regular build overwrite my working version of the image.

i need to update our build process to build it correctly.

Hi Ben,

According to comment 7, my understanding is the following:
1. old hello-openshift json + docker (>= 1.6.2) + new hello-openshift image - PASS
2. new hello-openshift json + docker (< 1.6.2) +  new hello-openshift image - PASS

That means beta2 json would also work in #1, my understanding is right?

your understanding is correct...with docker 1.6.2 you only need the new image, not the new pod json.

https://github.com/openshift/origin/pull/2904

still requires either docker 1.6.2 or that you use the latest hello-pod.json that mounts a /tmp volume.

Verified this bug with openshift-0.5.2.2-0.git.26.701be15.el7ose and latest openshift/hello-openshift image (ae47c85b9f17), PASS.

1. new hello-openshift json + docker-1.6.2-6.el7 + new hello-openshift image - PASS
2. old hello-openshift json + docker-1.6.2-6.el7 + new hello-openshift image - PASS
3. new hello-openshift json + docker-1.6.0-11.el7 + new hello-openshift image - PASS
4. old hello-openshift json + docker-1.6.0-11.el7 + new hello-openshift image - PASS