Nothing wrong with image origin-custom-docker-builder, since if I use dockerimage to build directly, build succeeds, only failed when use imagestream.

Guessing that is another image that's not being built with docker1.9 (as none of the origin repo built images are), so this will be fixed when we get the docker1.10 patch which pushes images w/ the right metadata.

Also met this with `oc new-app openshift/hello-openshift:latest`, but `oc run hello --image=openshift/hello-openshift:latest` work well.

    [root@dhcp-128-75 ~]# oc run hello --image=openshift/hello-openshift:latest
    deploymentconfig "hello" created
    [root@dhcp-128-75 ~]# oc get pod
    NAME             READY     STATUS    RESTARTS   AGE
    hello-1-deploy   1/1       Running   0          15s
    hello-1-tze37    1/1       Running   0          12s
    [root@dhcp-128-75 ~]# oc new-app openshift/hello-openshift:latest
    --> Found Docker image 9600aad (2 hours old) from Docker Hub for "openshift/hello-openshift:latest"
     
        * An image stream will be created as "hello-openshift:latest" that will track this image
        * This image will be deployed in deployment config "hello-openshift"
        * Ports 8080/tcp, 8888/tcp will be load balanced by service "hello-openshift"
          * Other containers can access this service through the hostname "hello-openshift"
        * WARNING: Image "hello-openshift" runs as the 'root' user which may not be permitted by your cluster administrator    [root@dhcp-128-75 ~]# oc run hello --image=openshift/hello-openshift:latest
    deploymentconfig "hello" created
    [root@dhcp-128-75 ~]# oc get pod
    NAME             READY     STATUS    RESTARTS   AGE
    hello-1-deploy   1/1       Running   0          15s
    hello-1-tze37    1/1       Running   0          12s
    [root@dhcp-128-75 ~]# oc new-app openshift/hello-openshift:latest
    --> Found Docker image 9600aad (2 hours old) from Docker Hub for "openshift/hello-openshift:latest"
     
        * An image stream will be created as "hello-openshift:latest" that will track this image
        * This image will be deployed in deployment config "hello-openshift"
        * Ports 8080/tcp, 8888/tcp will be load balanced by service "hello-openshift"
          * Other containers can access this service through the hostname "hello-openshift"
        * WARNING: Image "hello-openshift" runs as the 'root' user which may not be permitted by your cluster administrator
     
    --> Creating resources with label app=hello-openshift ...
        imagestream "hello-openshift" created
        deploymentconfig "hello-openshift" created
        service "hello-openshift" created
    --> Success
        Run 'oc status' to view your app.
    [root@dhcp-128-75 ~]# oc get pod
    NAME                       READY     STATUS              RESTARTS   AGE
    hello-1-tze37              1/1       Running             0          1m
    hello-openshift-1-deploy   1/1       Running             0          9s
    hello-openshift-1-tjtqm    0/1       ContainerCreating   0          6s
    [root@dhcp-128-75 ~]# oc get pod
    NAME                       READY     STATUS         RESTARTS   AGE
    hello-1-tze37              1/1       Running        0          1m
    hello-openshift-1-deploy   1/1       Running        0          17s
    hello-openshift-1-tjtqm    0/1       ErrImagePull   0          14s

     
    --> Creating resources with label app=hello-openshift ...
        imagestream "hello-openshift" created
        deploymentconfig "hello-openshift" created
        service "hello-openshift" created
    --> Success
        Run 'oc status' to view your app.
    [root@dhcp-128-75 ~]# oc get pod
    NAME                       READY     STATUS              RESTARTS   AGE
    hello-1-tze37              1/1       Running             0          1m
    hello-openshift-1-deploy   1/1       Running             0          9s
    hello-openshift-1-tjtqm    0/1       ContainerCreating   0          6s
    [root@dhcp-128-75 ~]# oc get pod
    NAME                       READY     STATUS         RESTARTS   AGE
    hello-1-tze37              1/1       Running        0          1m
    hello-openshift-1-deploy   1/1       Running        0          17s
    hello-openshift-1-tjtqm    0/1       ErrImagePull   0          14s
# oc describe pod hello-openshift-2-dhtjo
Name:           hello-openshift-2-dhtjo
Namespace:      wjiang
Node:           openshift-143.lab.sjc.redhat.com/192.168.1.20
Start Time:     Fri, 17 Jun 2016 11:08:39 +0800
Labels:         app=hello-openshift,deployment=hello-openshift-2,deploymentconfig=hello-openshift
Status:         Pending
IP:             10.1.1.10
Controllers:    ReplicationController/hello-openshift-2
Containers:
  hello-openshift:
    Container ID:
    Image:              openshift/hello-openshift@sha256:3c55d5d43668e3bddc6d322ad3d2ea74f27a99ac91f509e815b10284e3ac5309
    Image ID:
    Ports:              8080/TCP, 8888/TCP
    QoS Tier:
      cpu:              BestEffort
      memory:           BestEffort
    State:              Waiting
      Reason:           ErrImagePull
    Ready:              False
    Restart Count:      0
    Environment Variables:
Conditions:
  Type          Status
  Ready         False 
Volumes:
  default-token-mxk9d:
    Type:       Secret (a volume populated by a Secret)
    SecretName: default-token-mxk9d
Events:
  FirstSeen     LastSeen        Count   From                                            SubobjectPath                           Type            Reason          Message
  ---------     --------        -----   ----                                            -------------                           --------        ------          -------
  16s           16s             1       {default-scheduler }                                                                    Normal          Scheduled       Successfully assigned hello-openshift-2-dhtjo to openshift-143.lab.sjc.redhat.com
  15s           15s             1       {kubelet openshift-143.lab.sjc.redhat.com}      spec.containers{hello-openshift}        Normal          Pulling         pulling image "openshift/hello-openshift@sha256:3c55d5d43668e3bddc6d322ad3d2ea74f27a99ac91f509e815b10284e3ac5309"
  11s           11s             1       {kubelet openshift-143.lab.sjc.redhat.com}      spec.containers{hello-openshift}        Warning         Failed          Failed to pull image "openshift/hello-openshift@sha256:3c55d5d43668e3bddc6d322ad3d2ea74f27a99ac91f509e815b10284e3ac5309": manifest unknown: manifest unknown
  11s           11s             1       {kubelet openshift-143.lab.sjc.redhat.com}                                              Warning         FailedSync      Error syncing pod, skipping: failed to "StartContainer" for "hello-openshift" with ErrImagePull: "manifest unknown: manifest unknown"

  10s   10s     1       {kubelet openshift-143.lab.sjc.redhat.com}      spec.containers{hello-openshift}        Normal  BackOff         Back-off pulling image "openshift/hello-openshift@sha256:3c55d5d43668e3bddc6d322ad3d2ea74f27a99ac91f509e815b10284e3ac5309"
  10s   10s     1       {kubelet openshift-143.lab.sjc.redhat.com}                                              Warning FailedSync      Error syncing pod, skipping: failed to "StartContainer" for "hello-openshift" with ImagePullBackOff: "Back-off pulling image \"openshift/hello-openshift@sha256:3c55d5d43668e3bddc6d322ad3d2ea74f27a99ac91f509e815b10284e3ac5309\""

are there any possible workaround for this issue?

Right now I get:

$ oc describe builds ruby-sample-build-1
Name:		ruby-sample-build-1
Namespace:	myproject
Created:	35 seconds ago
Labels:		app=ruby-sample-build
		buildconfig=ruby-sample-build
		name=ruby-sample-build
		openshift.io/build-config.name=ruby-sample-build
		openshift.io/build.start-policy=Serial
		template=application-template-custombuild
Annotations:	openshift.io/build-config.name=ruby-sample-build
		openshift.io/build.number=1
		openshift.io/build.pod-name=ruby-sample-build-1-build

Status:		Running
Started:	Tue, 12 Jul 2016 14:09:32 CEST
Duration:	running for 24s
Build Config:	ruby-sample-build
Build Pod:	ruby-sample-build-1-build

Strategy:		Custom
URL:			git://github.com/openshift/ruby-hello-world.git
Image Reference:	DockerImage openshift/origin-custom-docker-builder@sha256:3214f2c94c7c52b37cc6f99cfc496246dd32b5193dca9cdde4da06bd23677d59
Expose Docker Socket:	yes
Environment:		OPENSHIFT_CUSTOM_BUILD_BASE_IMAGE=openshift/origin-custom-docker-builder@sha256:3214f2c94c7c52b37cc6f99cfc496246dd32b5193dca9cdde4da06bd23677d59
Output to:		ImageStreamTag origin-ruby-sample:latest
Push Secret:		builder-dockercfg-brvw7

Build trigger cause:	Image change
Image ID:		openshift/origin-custom-docker-builder@sha256:3214f2c94c7c52b37cc6f99cfc496246dd32b5193dca9cdde4da06bd23677d59
Image Name/Kind:	origin-custom-docker-builder:latest / ImageStreamTag

Events:
  FirstSeen	LastSeen	Count	From					SubobjectPath			Type		Reason		Message
  ---------	--------	-----	----					-------------			--------	------		-------
  35s		35s		1	{default-scheduler }							Normal		Scheduled	Successfully assigned ruby-sample-build-1-build to dhcp129-55.brq.redhat.com
  35s		35s		1	{kubelet dhcp129-55.brq.redhat.com}	spec.containers{custom-build}	Normal		Pulling		pulling image "openshift/origin-custom-docker-builder@sha256:3214f2c94c7c52b37cc6f99cfc496246dd32b5193dca9cdde4da06bd23677d59"
  25s		25s		1	{kubelet dhcp129-55.brq.redhat.com}	spec.containers{custom-build}	Normal		Pulled		Successfully pulled image "openshift/origin-custom-docker-builder@sha256:3214f2c94c7c52b37cc6f99cfc496246dd32b5193dca9cdde4da06bd23677d59"
  25s		25s		1	{kubelet dhcp129-55.brq.redhat.com}	spec.containers{custom-build}	Normal		Created		Created container with docker id 6a5aff3c3a8d
  24s		24s		1	{kubelet dhcp129-55.brq.redhat.com}	spec.containers{custom-build}	Normal		Started		Started container with docker id 6a5aff3c3a8d


$ oc get pod
NAME                        READY     STATUS      RESTARTS   AGE
database-1-m7hug            1/1       Running     0          2m
frontend-1-egcy6            1/1       Running     0          29s
frontend-1-hgln8            1/1       Running     0          29s
ruby-sample-build-1-build   0/1       Completed   0          2m

I reproduce this bug on 
[openshift v3.2.1.7
 kubernetes v1.2.0-36-g4a3f9c5
 etcd 2.2.5]

Step:
1. do docker build
 $ oc new-build -D $'FROM centos:7\nRUN yum install -y httpd' --to=mytest
2. check build log
FROM centos@sha256:ee06bb52b1fc453a72cee57b4ee4afb94d8e0aca479cd9dae5999f96e1e4f7a4
Trying to pull repository docker.io/library/centos ... 
manifest unknown: manifest unknown
F0715 03:09:07.944906       1 builder.go:204] Error: build error: manifest unknown: manifest unknown


but it works well on
[openshift v3.3.0.4
 kubernetes v1.3.0+57fb9ac
 etcd 2.3.0+git]

buildlog:
FROM centos@sha256:81e4f2f663eaa1bf46ff9be348396dd7053734b257ef4147d7133d6f25bbf7cf
Trying to pull repository docker.io/library/centos ... 
Pulling from docker.io/library/centos
Digest: sha256:81e4f2f663eaa1bf46ff9be348396dd7053734b257ef4147d7133d6f25bbf7cf
Status: Downloaded newer image for docker.io/centos@sha256:81e4f2f663eaa1bf46ff9be348396dd7053734b257ef4147d7133d6f25bbf7cf
 ---> 05188b417f30

the image digests are different, why ose3.3.0.4 can get correct image digest, but ose3.2.1.7 cannot

I think a fix was put into 3.3 to make it get the correct manifest so it can work around this docker1.9/1.10 issue, but this is really a question for Alexey i think.

This is because the ose3.2.1 does not support the v2 scheme. We added v2 schema support only in 3.3.

ose3.1 also have the problem
openshift v3.1.1.6-64-g80b61da
kubernetes v1.1.0-origin-1107-g4c8e6f4
etcd 2.1.2

reproduced against Ded-Int-Gcp
Version
OpenShift Master: v3.2.1.9-1-g2265530
Kubernetes Master: v1.2.0-36-g4a3f9c5

Test with
fork_ami_openshift3_miminar_295 (ami-66c0af71)

openshift v3.2.0.42-230-g1e262d6
kubernetes v1.2.0-36-g4a3f9c5
etcd 2.2.5

Docker version 1.10.3, build 8491c12-unsupported


cannot reproduce this issue

step:
$ oc new-build -D $'FROM centos:7\nRUN yum install -y httpd' --to=mytest

(In reply to Dongbo Yan from comment #14)
> Test with
> fork_ami_openshift3_miminar_295 (ami-66c0af71)
> 
> openshift v3.2.0.42-230-g1e262d6
> kubernetes v1.2.0-36-g4a3f9c5
> etcd 2.2.5
> 
> Docker version 1.10.3, build 8491c12-unsupported
> 
> 
> cannot reproduce this issue
> 
> step:
> $ oc new-build -D $'FROM centos:7\nRUN yum install -y httpd' --to=mytest

Are you testing with an old enought version of OpenShift? 

(In reply to mdong from comment #11)
> reproduced against Ded-Int-Gcp
> Version
> OpenShift Master: v3.2.1.9-1-g2265530
> Kubernetes Master: v1.2.0-36-g4a3f9c5

Could you please update me on whether the fix has been backported to 3.2 and what exact version has this fix included?

Verified with version:
openshift v3.2.1.16
kubernetes v1.2.0-36-g4a3f9c5
etcd 2.2.5