This should have been fixed by this PR (merged two days ago):
https://github.com/openshift/origin/pull/7713

Looking at the logs, seems that the version of openshift used here doesn't have that patch.

Wang Haoran, could you please try with a more recent version of origin?
Thanks

verified with the following version, and fell free to move the bug to ON_QA
openshift v1.1.3-553-g19dbf2a
kubernetes v1.2.0-alpha.7-703-gbc4550d
etcd 2.2.5
openshift/origin-docker-builder               latest              0d57ad7c9cc1

this is the incremental build log:
I0310 02:31:48.727816       1 util.go:186] Pulling image "172.30.214.88:5000/haowang/ruby-hello-world:latest" ...
I0310 02:31:48.727825       1 docker.go:308] Pulling Docker image 172.30.214.88:5000/haowang/ruby-hello-world:latest ...
I0310 02:31:49.302541       1 sti.go:146] Existing image for tag haowang/ruby-hello-world-6:a9d2d308 detected for incremental build
I0310 02:31:49.302616       1 sti.go:151] Performing source build from file:///tmp/s2i-build547406922/upload/src
I0310 02:31:49.302697       1 sti.go:377] Saving build artifacts from image 172.30.214.88:5000/haowang/ruby-hello-world:latest to path /tmp/s2i-build547406922/upload/artifacts
I0310 02:31:49.308702       1 sti.go:388] The assemble user is not set, defaulting to "1001" user
I0310 02:31:49.322838       1 docker.go:410] Image contains io.openshift.s2i.scripts-url set to 'image:///usr/libexec/s2i'
I0310 02:31:49.322853       1 docker.go:465] Base directory for STI scripts is '/usr/libexec/s2i'. Untarring destination is '/tmp'.
I0310 02:31:49.322861       1 docker.go:479] Running "/usr/libexec/s2i/save-artifacts" command in container ...
I0310 02:31:49.322871       1 docker.go:614] Creating container  using config: &{Hostname: Domainname: User:1001 Memory:0 MemorySwap:0 MemoryReservation:0 KernelMemory:0 CPUShares:0 CPUSet: AttachStdin:false AttachStdout:true AttachStderr:false PortSpecs:[] ExposedPorts:map[] StopSignal: Tty:false OpenStdin:false StdinOnce:false Env:[] Cmd:[/usr/libexec/s2i/save-artifacts] DNS:[] Image:172.30.214.88:5000/haowang/ruby-hello-world:latest Volumes:map[] VolumeDriver: VolumesFrom: WorkingDir: MacAddress: Entrypoint:[] NetworkDisabled:false SecurityOpts:[] OnBuild:[] Mounts:[] Labels:map[]}, hostconfig: &{Binds:[] CapAdd:[] CapDrop:[] GroupAdd:[] ContainerIDFile: LxcConf:[] Privileged:false PortBindings:map[] Links:[] PublishAllPorts:false DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] VolumesFrom:[] NetworkMode:container:263f40554cc16c48874c46549b54af4a14bf9d9911c4b3eeeadfe57a129e623a IpcMode: PidMode: UTSMode: RestartPolicy:{Name: MaximumRetryCount:0} Devices:[] LogConfig:{Type: Config:map[]} ReadonlyRootfs:false SecurityOpt:[] CgroupParent: Memory:9223372036854775807 MemorySwap:9223372036854775807 MemorySwappiness:0 OOMKillDisable:false CPUShares:2 CPUSet: CPUSetCPUs: CPUSetMEMs: CPUQuota:-1 CPUPeriod:100000 BlkioWeight:0 Ulimits:[] VolumeDriver: OomScoreAdj:0}

Verified with :
openshift v3.2.0.1
kubernetes v1.2.0-alpha.7-703-gbc4550d
etcd 2.2.5

reopen this bug, since currently can pull the previous image successfully when do incremental build, but if test with a repo contains the save-artifacts script, build will hang ...
Steps:
1. create app using repo contains save-artifacts script
  oc new-app https://github.com/wanghaoran1988/nodejs-ex
2. update the bc to enable incremental build,wait the first build complete
3. start a new build, and watch the build logs, the build will stop at "Container wait returns with 0 and <nil>"

[root@ip-172-18-3-72 ~]# oc logs -f build/nodejs-ex-3
I0314 05:07:23.532627       1 builder.go:46] $BUILD env var is {"kind":"Build","apiVersion":"v1","metadata":{"name":"nodejs-ex-3","namespace":"haowang","selfLink":"/oapi/v1/namespaces/haowang/builds/nodejs-ex-3","uid":"a010f17a-e9a2-11e5-a7fd-0eb9d8036749","resourceVersion":"1205","creationTimestamp":"2016-03-14T05:07:18Z","labels":{"app":"nodejs-ex","buildconfig":"nodejs-ex","openshift.io/build-config.name":"nodejs-ex"},"annotations":{"openshift.io/build.number":"3"}},"spec":{"serviceAccount":"builder","source":{"type":"Git","git":{"uri":"https://github.com/wanghaoran1988/nodejs-ex"},"secrets":[]},"strategy":{"type":"Source","sourceStrategy":{"from":{"kind":"DockerImage","name":"registry.access.redhat.com/openshift3/nodejs-010-rhel7:latest"},"incremental":true}},"output":{"to":{"kind":"DockerImage","name":"172.30.207.39:5000/haowang/nodejs-ex:latest"},"pushSecret":{"name":"builder-dockercfg-giut4"}},"resources":{},"postCommit":{}},"status":{"phase":"New","outputDockerImageReference":"172.30.207.39:5000/haowang/nodejs-ex:latest","config":{"kind":"BuildConfig","namespace":"haowang","name":"nodejs-ex"}}}
 
I0314 05:07:23.536673       1 builder.go:57] Master version "v1.1.3-631-g2d2d659", Builder version "v1.1.3-631-g2d2d659"
I0314 05:07:23.537547       1 builder.go:145] Running build with cgroup limits: api.CGroupLimits{MemoryLimitBytes:9223372036854775807, CPUShares:2, CPUPeriod:100000, CPUQuota:-1, MemorySwap:9223372036854775807}
I0314 05:07:23.544657       1 sti.go:189] With force pull false, setting policies to if-not-present
I0314 05:07:23.544672       1 sti.go:195] The value of ALLOWED_UIDS is [1-]
I0314 05:07:23.544685       1 sti.go:203] The value of DROP_CAPS is [KILL,MKNOD,SETGID,SETUID,SYS_CHROOT]
I0314 05:07:23.544697       1 cfg.go:45] Locating docker auth for image registry.access.redhat.com/openshift3/nodejs-010-rhel7:latest and type PULL_DOCKERCFG_PATH
I0314 05:07:23.544752       1 cfg.go:111] Using Docker authentication configuration in '/root/.docker/config.json'
I0314 05:07:23.545224       1 cfg.go:57] Problem accessing /root/.docker/config.json: stat /root/.docker/config.json: no such file or directory
I0314 05:07:23.545241       1 cfg.go:45] Locating docker auth for image 172.30.207.39:5000/haowang/nodejs-ex:latest and type PUSH_DOCKERCFG_PATH
I0314 05:07:23.545262       1 cfg.go:111] Using Docker authentication configuration in '/var/run/secrets/openshift.io/push/.dockercfg'
I0314 05:07:23.545428       1 cfg.go:83] Using serviceaccount user for Docker authentication for image 172.30.207.39:5000/haowang/nodejs-ex:latest
I0314 05:07:23.551690       1 docker.go:291] Using locally available image "registry.access.redhat.com/openshift3/nodejs-010-rhel7:latest"
I0314 05:07:23.552980       1 sti.go:222] Creating a new S2I builder with build config: "Builder Name:\t\t\tNode.js 0.10\nBuilder Image:\t\t\tregistry.access.redhat.com/openshift3/nodejs-010-rhel7:latest\nSource:\t\t\t\tfile:///tmp/s2i-build580693189/upload/src\nOutput Image Tag:\t\thaowang/nodejs-ex-3:116b14c7\nEnvironment:\t\t\tOPENSHIFT_BUILD_NAME=nodejs-ex-3,OPENSHIFT_BUILD_NAMESPACE=haowang,OPENSHIFT_BUILD_SOURCE=https://github.com/wanghaoran1988/nodejs-ex\nIncremental Build:\t\t\x1b[1menabled\x1b[0m\nIncremental Image Pull User:\tserviceaccount\nRemove Old Build:\t\tdisabled\nBuilder Pull Policy:\t\tif-not-present\nPrevious Image Pull Policy:\talways\nQuiet:\t\t\t\tdisabled\nLayered Build:\t\t\tdisabled\nWorkdir:\t\t\t/tmp/s2i-build580693189\nDocker NetworkMode:\t\tcontainer:aabad3e0f88a6d7446605012fd6f8a63754a647a232f0b0de4a496e29e76924d\nDocker Endpoint:\t\tunix:///var/run/docker.sock\n"
I0314 05:07:23.554381       1 docker.go:291] Using locally available image "registry.access.redhat.com/openshift3/nodejs-010-rhel7:latest"
I0314 05:07:23.557916       1 docker.go:291] Using locally available image "registry.access.redhat.com/openshift3/nodejs-010-rhel7:latest"
I0314 05:07:23.557935       1 docker.go:411] Image contains io.openshift.s2i.scripts-url set to 'image:///usr/libexec/s2i'
I0314 05:07:23.557970       1 sti.go:228] Starting S2I build from haowang/nodejs-ex-3 BuildConfig ...
I0314 05:07:23.557978       1 sti.go:140] Preparing to build haowang/nodejs-ex-3:116b14c7
I0314 05:07:23.571536       1 source.go:197] Downloading "https://github.com/wanghaoran1988/nodejs-ex" ...
I0314 05:07:23.571567       1 source.go:109] git ls-remote https://github.com/wanghaoran1988/nodejs-ex --heads
I0314 05:07:23.571612       1 repository.go:290] Executing git ls-remote https://github.com/wanghaoran1988/nodejs-ex --heads
I0314 05:07:24.054552       1 source.go:208] Cloning source from https://github.com/wanghaoran1988/nodejs-ex
I0314 05:07:24.054834       1 repository.go:290] Executing git clone --recursive https://github.com/wanghaoran1988/nodejs-ex /tmp/s2i-build580693189/upload/src
I0314 05:07:24.486666       1 repository.go:315] Out: Cloning into '/tmp/s2i-build580693189/upload/src'...
I0314 05:07:24.486703       1 repository.go:290] Executing git config --get remote.origin.url
I0314 05:07:24.487908       1 repository.go:315] Out: https://github.com/wanghaoran1988/nodejs-ex
I0314 05:07:24.487927       1 repository.go:290] Executing git rev-parse --abbrev-ref HEAD
I0314 05:07:24.489089       1 repository.go:315] Out: master
I0314 05:07:24.489140       1 repository.go:290] Executing git rev-parse --verify HEAD
I0314 05:07:24.490230       1 repository.go:315] Out: 8edcd691bce0fa62870b6927a532509a4f9ddb46
I0314 05:07:24.490261       1 repository.go:290] Executing git --no-pager show -s --format=%an HEAD
I0314 05:07:24.491769       1 repository.go:315] Out: Haoran Wang
I0314 05:07:24.491789       1 repository.go:290] Executing git --no-pager show -s --format=%ae HEAD
I0314 05:07:24.493125       1 repository.go:315] Out: haowang
I0314 05:07:24.493146       1 repository.go:290] Executing git --no-pager show -s --format=%cn HEAD
I0314 05:07:24.506763       1 repository.go:315] Out: Haoran Wang
I0314 05:07:24.506785       1 repository.go:290] Executing git --no-pager show -s --format=%ce HEAD
I0314 05:07:24.508102       1 repository.go:315] Out: haowang
I0314 05:07:24.508155       1 repository.go:290] Executing git --no-pager show -s --format=%ad HEAD
I0314 05:07:24.509567       1 repository.go:315] Out: Thu Dec 3 11:13:02 2015 +0800
I0314 05:07:24.509588       1 repository.go:290] Executing git --no-pager show -s --format=%<(80,trunc)%s HEAD
I0314 05:07:24.510914       1 repository.go:315] Out: add test env
I0314 05:07:24.510940       1 common.go:86] Setting build revision to &api.GitSourceRevision{Commit:"8edcd691bce0fa62870b6927a532509a4f9ddb46", Author:api.SourceControlUser{Name:"Haoran Wang", Email:"haowang"}, Committer:api.SourceControlUser{Name:"Haoran Wang", Email:"haowang"}, Message:"add test env"}
I0314 05:07:24.569908       1 install.go:111] DEPRECATED: Use .s2i/bin instead of .sti/bin
I0314 05:07:24.569965       1 install.go:236] Using "assemble" installed from "<source-dir>/.sti/bin/assemble"
I0314 05:07:24.570000       1 install.go:236] Using "run" installed from "image:///usr/libexec/s2i/run"
I0314 05:07:24.570014       1 install.go:111] DEPRECATED: Use .s2i/bin instead of .sti/bin
I0314 05:07:24.570038       1 install.go:236] Using "save-artifacts" installed from "<source-dir>/.sti/bin/save-artifacts"
I0314 05:07:24.570154       1 ignore.go:58] .s2iignore file does not exist
I0314 05:07:24.570168       1 util.go:186] Pulling image "172.30.207.39:5000/haowang/nodejs-ex:latest" ...
I0314 05:07:24.570178       1 docker.go:309] Pulling Docker image 172.30.207.39:5000/haowang/nodejs-ex:latest ...
I0314 05:07:24.761334       1 sti.go:150] Existing image for tag 172.30.207.39:5000/haowang/nodejs-ex:latest detected for incremental build
I0314 05:07:24.761353       1 sti.go:155] Performing source build from file:///tmp/s2i-build580693189/upload/src
I0314 05:07:24.761436       1 sti.go:381] Saving build artifacts from image 172.30.207.39:5000/haowang/nodejs-ex:latest to path /tmp/s2i-build580693189/upload/artifacts
I0314 05:07:24.763309       1 sti.go:392] The assemble user is not set, defaulting to "1001" user
I0314 05:07:24.765105       1 docker.go:457] Both scripts and untarred source will be placed in '/tmp'
I0314 05:07:24.765124       1 docker.go:480] Running "/tmp/scripts/save-artifacts" command in container ...
I0314 05:07:24.765136       1 docker.go:616] Creating container  using config: &{Hostname: Domainname: User:1001 Memory:0 MemorySwap:0 MemoryReservation:0 KernelMemory:0 CPUShares:0 CPUSet: AttachStdin:false AttachStdout:true AttachStderr:false PortSpecs:[] ExposedPorts:map[] StopSignal: Tty:false OpenStdin:false StdinOnce:false Env:[] Cmd:[/tmp/scripts/save-artifacts] DNS:[] Image:172.30.207.39:5000/haowang/nodejs-ex:latest Volumes:map[] VolumeDriver: VolumesFrom: WorkingDir: MacAddress: Entrypoint:[] NetworkDisabled:false SecurityOpts:[] OnBuild:[] Mounts:[] Labels:map[]}, hostconfig: &{Binds:[] CapAdd:[] CapDrop:[KILL MKNOD SETGID SETUID SYS_CHROOT] GroupAdd:[] ContainerIDFile: LxcConf:[] Privileged:false PortBindings:map[] Links:[] PublishAllPorts:false DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] VolumesFrom:[] NetworkMode:container:aabad3e0f88a6d7446605012fd6f8a63754a647a232f0b0de4a496e29e76924d IpcMode: PidMode: UTSMode: RestartPolicy:{Name: MaximumRetryCount:0} Devices:[] LogConfig:{Type: Config:map[]} ReadonlyRootfs:false SecurityOpt:[] CgroupParent: Memory:9223372036854775807 MemorySwap:9223372036854775807 MemorySwappiness:0 OOMKillDisable:false CPUShares:2 CPUSet: CPUSetCPUs: CPUSetMEMs: CPUQuota:-1 CPUPeriod:100000 BlkioWeight:0 Ulimits:[] VolumeDriver: OomScoreAdj:0}
I0314 05:07:25.231548       1 docker.go:623] Attaching to container
I0314 05:07:25.233182       1 docker.go:629] Starting container
I0314 05:07:25.615653       1 tar.go:296] Creating directory /tmp/s2i-build580693189/upload/artifacts/.npm
.......
I0314 05:07:26.253225       1 tar.go:366] Extracting/writing /tmp/s2i-build580693189/upload/artifacts/.npm/vary/1.0.1/package/package.json
I0314 05:07:26.253287       1 tar.go:305] Creating directory /tmp/s2i-build580693189/upload/artifacts/.npm/vary/1.0.1
I0314 05:07:26.253297       1 tar.go:356] Creating /tmp/s2i-build580693189/upload/artifacts/.npm/vary/1.0.1/package.tgz
I0314 05:07:26.253752       1 tar.go:366] Extracting/writing /tmp/s2i-build580693189/upload/artifacts/.npm/vary/1.0.1/package.tgz
I0314 05:07:26.253810       1 tar.go:335] Done extracting tar stream
I0314 05:07:26.253821       1 docker.go:549] Waiting for container
I0314 05:07:26.319620       1 docker.go:551] Container wait returns with 0 and <nil>

Confirmed this is broken in s2i itself as well, incremental builds hang on attaching to the container.

Rodolfo, I wasn't able to chase it much further than that, but it's very easy to recreate w/ an incremental build using s2i.

ok I chased it a little further, the problem was introduced by this commit:
https://github.com/openshift/source-to-image/commit/c0b4d96f419747b19f61d1cc04a7e1e773b0fee1

prior to that commit, we were getting an error when attaching to the container (because the pipe was closed), but things were able to still proceed (I think it just means we get no output from the save-artifacts script when the attach fails).

after that commit, the attach doesn't fail, it hangs.

Which is not to say i think the fix in that commit was wrong, but it has exposed a deeper issue w/ attach.  So the quick fix is to stop attaching.  The better fix is to figure out why attach is hanging.

(this is the attach call that's hanging: https://github.com/openshift/source-to-image/blob/master/pkg/docker/docker.go#L540)

Hi, Ben , I found when the pod hang their , the language container with the save-artifect already stopped, so attach hanging there without receive the attach sucucessful signal, we should found out why failed to attach to the container when the container is running.

the more i look at it, the less I understand why we're attaching stdout for the save-artifacts path.  We already want to send the stdout to the tar function we set as the OnStart, so i don't know that we should be attaching to stdout again.  it's possible this second attach is what's causing the hang.

Ben, perhaps the time has come to review that code.

When I wrote the helper in origin to start a container I could not reuse the one in S2I, and the end result I got was much simpler than what's in S2I.

Possibly we don't even need to attach, all we need is to stream the logs (which is a different API call).

I'll look into it.

Tentative to reproduce:


First run (image built from scratch):

[vagrant@localhost ~]$ s2i build https://github.com/bparees/incremental-app centos/ruby-22-centos7 incr-test --incremental
I0317 09:05:37.455744 17841 docker.go:287] Image "centos/ruby-22-centos7:latest" not available locally, pulling ...
I0317 09:06:20.359652 17841 clone.go:32] Downloading "https://github.com/bparees/incremental-app" ...
I0317 09:06:21.861743 17841 install.go:111] DEPRECATED: Use .s2i/bin instead of .sti/bin
I0317 09:06:21.862772 17841 install.go:236] Using "assemble" installed from "<source-dir>/.sti/bin/assemble"
I0317 09:06:21.863531 17841 install.go:236] Using "run" installed from "image:///usr/libexec/s2i/run"
I0317 09:06:21.863597 17841 install.go:111] DEPRECATED: Use .s2i/bin instead of .sti/bin
I0317 09:06:21.863631 17841 install.go:236] Using "save-artifacts" installed from "<source-dir>/.sti/bin/save-artifacts"
I0317 09:06:21.866941 17841 docker.go:287] Image "incr-test:latest" not available locally, pulling ...
I0317 09:06:27.210619 17841 environment.go:31] DEPRECATED: Use .s2i/environment instead of .sti/environment
I0317 09:06:27.210780 17841 environment.go:60] Setting 1 environment variables provided by environment file in sources
---> Installing application source ...
---> Building your Ruby application from source ...
---> Running 'bundle install --deployment' ...
Fetching gem metadata from https://rubygems.org/..........
Installing rack 1.6.4
Installing rack-protection 1.5.3
Installing tilt 2.0.1
Installing sinatra 1.4.6
Using bundler 1.7.8
Your bundle is complete!
It was installed into ./bundle
---> Cleaning up unused ruby gems ...
I0317 09:06:43.398408 17841 environment.go:31] DEPRECATED: Use .s2i/environment instead of .sti/environment
I0317 09:06:43.398468 17841 environment.go:60] Setting 1 environment variables provided by environment file in sources


OK.
Second run (incremental build):

[vagrant@localhost ~]$ s2i build https://github.com/bparees/incremental-app centos/ruby-22-centos7 incr-test --incremental 
I0317 09:09:05.923786 18837 clone.go:32] Downloading "https://github.com/bparees/incremental-app" ...
I0317 09:09:07.102825 18837 install.go:111] DEPRECATED: Use .s2i/bin instead of .sti/bin
I0317 09:09:07.102882 18837 install.go:236] Using "assemble" installed from "<source-dir>/.sti/bin/assemble"
I0317 09:09:07.103292 18837 install.go:236] Using "run" installed from "image:///usr/libexec/s2i/run"
I0317 09:09:07.103326 18837 install.go:111] DEPRECATED: Use .s2i/bin instead of .sti/bin
I0317 09:09:07.103341 18837 install.go:236] Using "save-artifacts" installed from "<source-dir>/.sti/bin/save-artifacts"
(hung up)


Reproduced successfully.

Sending SIGKILL to the process above I got:


^\SIGQUIT: quit
PC=0x441e71

goroutine 6 [syscall, 1 minutes]:
runtime.notetsleepg(0xb4f8c0, 0xffffffffffffffff, 0x1)
        /usr/lib/golang/src/runtime/lock_futex.go:201 +0x52 fp=0xc208019768 sp=0xc208019740
runtime.signal_recv(0x0)
        /usr/lib/golang/src/runtime/sigqueue.go:109 +0x135 fp=0xc2080197a0 sp=0xc208019768
os/signal.loop()
        /usr/lib/golang/src/os/signal/signal_unix.go:21 +0x1f fp=0xc2080197e0 sp=0xc2080197a0
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc2080197e8 sp=0xc2080197e0
created by os/signal.init·1
        /usr/lib/golang/src/os/signal/signal_unix.go:27 +0x35

goroutine 1 [semacquire]:
sync.(*WaitGroup).Wait(0xc20811dc40)
        /usr/lib/golang/src/sync/waitgroup.go:132 +0x169
github.com/openshift/source-to-image/pkg/docker.runContainerWait(0xc20811dc40, 0xc20805aaa0, 0xc208041380, 0x0, 0x0)
        /home/rodolfo/openshift/src/github.com/openshift/source-to-image/_output/local/go/src/github.com/openshift/source-to-image/pkg/docker/docker.go:553 +0x249
github.com/openshift/source-to-image/pkg/docker.(*stiDocker).RunContainer(0xc20805aaa0, 0x7ffd48124581, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/rodolfo/openshift/src/github.com/openshift/source-to-image/_output/local/go/src/github.com/openshift/source-to-image/pkg/docker/docker.go:644 +0xc2b
github.com/openshift/source-to-image/pkg/build/strategies/sti.(*STI).Save(0xc2080a4000, 0xc20806d900, 0x0, 0x0)
        /home/rodolfo/openshift/src/github.com/openshift/source-to-image/_output/local/go/src/github.com/openshift/source-to-image/pkg/build/strategies/sti/sti.go:414 +0xa22
github.com/openshift/source-to-image/pkg/build/strategies/sti.(*STI).Build(0xc2080a4000, 0xc20806d900, 0x0, 0x0, 0x0)
        /home/rodolfo/openshift/src/github.com/openshift/source-to-image/_output/local/go/src/github.com/openshift/source-to-image/pkg/build/strategies/sti/sti.go:157 +0x583
main.func·002(0xc2080ca3c0, 0xc208057040, 0x3, 0x4)
        /home/rodolfo/openshift/src/github.com/openshift/source-to-image/_output/local/go/src/github.com/openshift/source-to-image/cmd/s2i/main.go:148 +0x11fa
github.com/spf13/cobra.(*Command).execute(0xc2080ca3c0, 0xc208056f80, 0x4, 0x4, 0x0, 0x0)
        /home/rodolfo/openshift/src/github.com/openshift/source-to-image/Godeps/_workspace/src/github.com/spf13/cobra/command.go:547 +0x82f
github.com/spf13/cobra.(*Command).Execute(0xc2080ca000, 0x0, 0x0)
        /home/rodolfo/openshift/src/github.com/openshift/source-to-image/Godeps/_workspace/src/github.com/spf13/cobra/command.go:630 +0x414
main.main()
        /home/rodolfo/openshift/src/github.com/openshift/source-to-image/_output/local/go/src/github.com/openshift/source-to-image/cmd/s2i/main.go:375 +0x781
...



As I suspected, we have a problem in `runContainerWait`.

I'm going to build s2i with -race enabled and let the runtime give us more hints...

From the stacktrace above, this is the line where we're waiting forever:

https://github.com/openshift/source-to-image/blob/9fe0bf74dd7658364e202ecce2a28f8cb7f83f66/pkg/docker/docker.go#L553

(sti.go) Build -> (sti.go) Save -> (docker.go) RunContainer -> (docker.go) runContainerWait


Didn't find any data races for this flow with the runtime race detector.

Working on a fix to the RunContainer logic.

This PR should fix the problem in S2I:
https://github.com/openshift/source-to-image/pull/430

Once it gets merged we'll need to bump the version in origin.

This is the origin PR we're waiting for:
https://github.com/openshift/origin/pull/8115

I make a test again with latest origin AMI,the hang problem disappeared ,but find a new problem.
When I use openshift/nodejs-010-centos7, the incremental build failed with logs:
I0321 02:06:41.904260       1 tar.go:260] Adding to tar: /tmp/s2i-build849450176/upload/src/package.json as src/package.json
I0321 02:06:41.904325       1 tar.go:260] Adding to tar: /tmp/s2i-build849450176/upload/src/server.js as src/server.js
I0321 02:06:41.904449       1 tar.go:260] Adding to tar: /tmp/s2i-build849450176/upload/src/views/index.html as src/views/index.html
I0321 02:06:41.998286       1 sti.go:563] Restoring build artifacts
E0321 02:06:42.027486       1 util.go:91] mv: cannot move '/tmp/artifacts/.npm' to '/opt/app-root/src/.npm': Directory not empty
I0321 02:06:42.465807       1 docker.go:614] Removing container "41ad3e7beb1adca5158fe86fcc5dc3d1099320b67f2cf6649f434e4e24da0d35" ...
I0321 02:06:42.596145       1 docker.go:618] Removed container "41ad3e7beb1adca5158fe86fcc5dc3d1099320b67f2cf6649f434e4e24da0d35"
I0321 02:06:42.600316       1 cleanup.go:23] Removing temporary directory /tmp/s2i-build849450176
I0321 02:06:42.600333       1 fs.go:156] Removing directory '/tmp/s2i-build849450176'
F0321 02:06:43.009280       1 builder.go:204] Error: build error: non-zero (13) exit code from openshift/nodejs-010-centos7@sha256:3379dca65270afbbb4a7b7dd563b52a31a9f0f8c1dadbdb8b6ebd5e2d81e7476

and I found the there are some nodejs dependency inside this image,so failed.
[root@ip-172-18-6-25 ~]# docker run -ti docker.io/openshift/nodejs-010-centos7 /bin/bash
bash-4.2$ ls -rlta
total 16
drwxr-xr-x.   3 default root 4096 Mar 16 18:55 .pki
drwxr-xr-x.   4 default root 4096 Mar 16 18:55 ..
drwxr-xr-x.   4 default root 4096 Mar 16 19:00 .
drwxr-xr-x. 136 default root 4096 Mar 16 19:00 .npm

but incremental build succeed with registry.access.redhat.com/openshift3/nodejs-010-rhel7
[root@ip-172-18-6-25 ~]# docker run -ti registry.access.redhat.com/openshift3/nodejs-010-rhel7 /bin/bash
bash-4.2$ ls -rlt
total 0
bash-4.2$ ls -rlta
total 8
drwxr-xr-x. 4 default root 4096 Mar  1 11:40 ..
drwxr-xr-x. 2 default root 4096 Mar  1 11:40 .


Ben, will you rebuild the image of openshift/nodejs-010-centos7 or make incremental build more compatible ?

testing on the ose env , the hang problem still there , seems code not merged.
openshift v3.2.0.5
kubernetes v1.2.0-36-g4a3f9c5
etcd 2.2.5

Thanks Wang Haoran!


As discussed on IRC, the error message:

E0321 02:06:42.027486       1 util.go:91] mv: cannot move '/tmp/artifacts/.npm' to '/opt/app-root/src/.npm': Directory not empty

is coming from the assemble script:

https://github.com/wanghaoran1988/nodejs-ex/blob/4263a602f7c281f1648af5e9279eaa234aa65876/.sti/bin/assemble#L29


The assemble script needs to be changed to handle that case, not code in s2i/origin.



The difference between the CentOS and RHEL image with regard to the `~/.npm` directory is because in CentOS we install nodemon using npm, while in RHEL its installed via yum.

FWIW, after this commit updating the assemble script [1] I can successfully do incremental builds:


$ s2i build https://github.com/wanghaoran1988/nodejs-ex openshift/nodejs-010-centos7 nodetest
...

Several "npm http GET" lines; while


$ s2i build https://github.com/wanghaoran1988/nodejs-ex openshift/nodejs-010-centos7 nodetest --incremental
...

No "npm http GET" lines; all packages are restored and reused.



Haoran, could you confirm this is working for you too?



[1] https://github.com/wanghaoran1988/nodejs-ex/commit/75b12f8681207fffb4ba757b161d4e1bb31a95a2

Verified with latest puddle ,and update script[1]

openshift v3.2.0.6
kubernetes v1.2.0-36-g4a3f9c5
etcd 2.2.5

[1]https://github.com/wanghaoran1988/nodejs-ex/blob/master/.sti/bin/assemble#L29

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:1064