Bug 1315607 - Cannot do incremental build
Cannot do incremental build
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build (Show other bugs)
3.2.0
Unspecified Unspecified
medium Severity high
: ---
: ---
Assigned To: Rodolfo Carvalho
Wenjing Zheng
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-03-08 03:34 EST by Wang Haoran
Modified: 2016-05-12 12:31 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-05-12 12:31:49 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2016:1064 normal SHIPPED_LIVE Important: Red Hat OpenShift Enterprise 3.2 security, bug fix, and enhancement update 2016-05-12 16:19:17 EDT

  None (edit)
Description Wang Haoran 2016-03-08 03:34:46 EST
Description of problem:
Cannot do incremental build ,always run clean build

Version-Release number of selected component (if applicable):
openshift v3.1.1.911
kubernetes v1.2.0-alpha.7-703-gbc4550d
etcd 2.2.5


How reproducible:

always
Steps to Reproduce:
1.create a project
2.create app using nodejs-example template
 #oc new-app --template=nodejs-example
3.wait the first build complete
4.update the bc to add incremental: true,and the git repo, https://github.com/wanghaoran1988/nodejs-ex.git (which contains the incremental script.)
5. oc start-build nodejs-example
6. watch the build log
$oc logs build/nodejs-example-2 -f
Actual results:
I0308 03:19:30.658774       1 docker.go:308] Pulling Docker image haowang/nodejs-example-3:2c2d43c5 ...
I0308 03:19:37.315908       1 docker.go:311] An error was received from the PullImage call: Error: image haowang/nodejs-example-3:2c2d43c5 not found
I0308 03:19:37.358500       1 sti.go:349] Unable to pull previously build "haowang/nodejs-example-3:2c2d43c5" image: unable to get haowang/nodejs-example-3:2c2d43c5


Expected results:
should pull the previous build out image successfully and do incremental build

Additional info:
[vagrant@ose ~]$ oc logs -f nodejs-example-3-build
I0308 03:19:16.504970       1 builder.go:46] $BUILD env var is {"kind":"Build","apiVersion":"v1","metadata":{"name":"nodejs-example-3","namespace":"haowang","selfLink":"/oapi/v1/namespaces/haowang/builds/nodejs-example-3","uid":"6b81570a-e506-11e5-a697-fa163ed32d7c","resourceVersion":"31268","creationTimestamp":"2016-03-08T08:19:04Z","labels":{"app":"nodejs-example","buildconfig":"nodejs-example","openshift.io/build-config.name":"nodejs-example","template":"nodejs-example"},"annotations":{"openshift.io/build.number":"3"}},"spec":{"serviceAccount":"builder","source":{"type":"Git","git":{"uri":"https://github.com/wanghaoran1988/nodejs-ex.git"},"secrets":null},"strategy":{"type":"Source","sourceStrategy":{"from":{"kind":"DockerImage","name":"brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/nodejs-010-rhel7:latest"},"incremental":true}},"output":{"to":{"kind":"DockerImage","name":"172.31.215.114:5000/haowang/nodejs-example:latest"},"pushSecret":{"name":"builder-dockercfg-hpce4"}},"resources":{},"postCommit":{}},"status":{"phase":"New","outputDockerImageReference":"172.31.215.114:5000/haowang/nodejs-example:latest","config":{"kind":"BuildConfig","namespace":"haowang","name":"nodejs-example"}}}
 
I0308 03:19:16.536147       1 builder.go:57] Master version "v3.1.1.911", Builder version "v3.1.1.911"
I0308 03:19:16.538053       1 builder.go:145] Running build with cgroup limits: api.CGroupLimits{MemoryLimitBytes:9223372036854775807, CPUShares:2, CPUPeriod:100000, CPUQuota:-1, MemorySwap:9223372036854775807}
I0308 03:19:16.577626       1 sti.go:187] With force pull false, setting policies to if-not-present
I0308 03:19:16.577661       1 sti.go:193] The value of ALLOWED_UIDS is [1-]
I0308 03:19:16.577712       1 cfg.go:45] Locating docker auth for image brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/nodejs-010-rhel7:latest and type PULL_DOCKERCFG_PATH
I0308 03:19:16.577797       1 cfg.go:111] Using Docker authentication configuration in '/root/.docker/config.json'
I0308 03:19:16.578029       1 cfg.go:57] Problem accessing /root/.docker/config.json: stat /root/.docker/config.json: no such file or directory
I0308 03:19:16.578049       1 cfg.go:45] Locating docker auth for image 172.31.215.114:5000/haowang/nodejs-example:latest and type PUSH_DOCKERCFG_PATH
I0308 03:19:16.578106       1 cfg.go:111] Using Docker authentication configuration in '/var/run/secrets/openshift.io/push/.dockercfg'
I0308 03:19:16.578361       1 cfg.go:83] Using serviceaccount user for Docker authentication for image 172.31.215.114:5000/haowang/nodejs-example:latest
I0308 03:19:16.744419       1 docker.go:299] error inspecting image brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/nodejs-010-rhel7:latest: no such image
I0308 03:19:16.744472       1 docker.go:286] Image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/nodejs-010-rhel7:latest" not available locally, pulling ...
I0308 03:19:16.744491       1 docker.go:308] Pulling Docker image brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/nodejs-010-rhel7:latest ...
I0308 03:19:26.113959       1 sti.go:215] Creating a new S2I builder with build config: "Builder Name:\t\t\tNode.js 0.10\nBuilder Image:\t\t\tbrew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/nodejs-010-rhel7:latest\nSource:\t\t\t\tfile:///tmp/s2i-build429232122/upload/src\nOutput Image Tag:\t\thaowang/nodejs-example-3:2c2d43c5\nEnvironment:\t\t\tOPENSHIFT_BUILD_NAME=nodejs-example-3,OPENSHIFT_BUILD_NAMESPACE=haowang,OPENSHIFT_BUILD_SOURCE=https://github.com/wanghaoran1988/nodejs-ex.git\nIncremental Build:\t\t\x1b[1menabled\x1b[0m\nIncremental Image Pull User:\tserviceaccount\nRemove Old Build:\t\tdisabled\nBuilder Pull Policy:\t\tif-not-present\nQuiet:\t\t\t\tdisabled\nLayered Build:\t\t\tdisabled\nWorkdir:\t\t\t/tmp/s2i-build429232122\nDocker NetworkMode:\t\tcontainer:6c1a8a557b68ac5c790548411352d57124a72f67c3e6cc27b1a0947acb17bf4c\nDocker Endpoint:\t\tunix:///var/run/docker.sock\n"
I0308 03:19:26.237158       1 docker.go:290] Using locally available image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/nodejs-010-rhel7:latest"
I0308 03:19:26.247203       1 docker.go:290] Using locally available image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/nodejs-010-rhel7:latest"
I0308 03:19:26.247299       1 docker.go:410] Image contains io.openshift.s2i.scripts-url set to 'image:///usr/libexec/s2i'
I0308 03:19:26.247378       1 sti.go:221] Starting S2I build from haowang/nodejs-example-3 BuildConfig ...
I0308 03:19:26.247391       1 sti.go:140] Preparing to build haowang/nodejs-example-3:2c2d43c5
I0308 03:19:26.326913       1 source.go:197] Downloading "https://github.com/wanghaoran1988/nodejs-ex.git" ...
I0308 03:19:26.327006       1 source.go:109] git ls-remote https://github.com/wanghaoran1988/nodejs-ex.git --heads
I0308 03:19:26.327199       1 repository.go:290] Executing git ls-remote https://github.com/wanghaoran1988/nodejs-ex.git --heads
I0308 03:19:28.338246       1 source.go:208] Cloning source from https://github.com/wanghaoran1988/nodejs-ex.git
I0308 03:19:28.338319       1 repository.go:290] Executing git clone --recursive https://github.com/wanghaoran1988/nodejs-ex.git /tmp/s2i-build429232122/upload/src
I0308 03:19:30.596319       1 repository.go:315] Out: Cloning into '/tmp/s2i-build429232122/upload/src'...
I0308 03:19:30.596381       1 repository.go:290] Executing git config --get remote.origin.url
I0308 03:19:30.598643       1 repository.go:315] Out: https://github.com/wanghaoran1988/nodejs-ex.git
I0308 03:19:30.598674       1 repository.go:290] Executing git rev-parse --abbrev-ref HEAD
I0308 03:19:30.600934       1 repository.go:315] Out: master
I0308 03:19:30.600980       1 repository.go:290] Executing git rev-parse --verify HEAD
I0308 03:19:30.602924       1 repository.go:315] Out: 8edcd691bce0fa62870b6927a532509a4f9ddb46
I0308 03:19:30.602955       1 repository.go:290] Executing git --no-pager show -s --format=%an HEAD
I0308 03:19:30.605926       1 repository.go:315] Out: Haoran Wang
I0308 03:19:30.605964       1 repository.go:290] Executing git --no-pager show -s --format=%ae HEAD
I0308 03:19:30.608664       1 repository.go:315] Out: haowang@redhat.com
I0308 03:19:30.608727       1 repository.go:290] Executing git --no-pager show -s --format=%cn HEAD
I0308 03:19:30.611399       1 repository.go:315] Out: Haoran Wang
I0308 03:19:30.611432       1 repository.go:290] Executing git --no-pager show -s --format=%ce HEAD
I0308 03:19:30.613886       1 repository.go:315] Out: haowang@redhat.com
I0308 03:19:30.613916       1 repository.go:290] Executing git --no-pager show -s --format=%ad HEAD
I0308 03:19:30.616522       1 repository.go:315] Out: Thu Dec 3 11:13:02 2015 +0800
I0308 03:19:30.616559       1 repository.go:290] Executing git --no-pager show -s --format=%<(80,trunc)%s HEAD
I0308 03:19:30.618979       1 repository.go:315] Out: add test env
I0308 03:19:30.619041       1 common.go:86] Setting build revision to &api.GitSourceRevision{Commit:"8edcd691bce0fa62870b6927a532509a4f9ddb46", Author:api.SourceControlUser{Name:"Haoran Wang", Email:"haowang@redhat.com"}, Committer:api.SourceControlUser{Name:"Haoran Wang", Email:"haowang@redhat.com"}, Message:"add test env"}
I0308 03:19:30.658483       1 install.go:111] DEPRECATED: Use .s2i/bin instead of .sti/bin
I0308 03:19:30.658563       1 install.go:236] Using "assemble" installed from "<source-dir>/.sti/bin/assemble"
I0308 03:19:30.658616       1 install.go:236] Using "run" installed from "image:///usr/libexec/s2i/run"
I0308 03:19:30.658636       1 install.go:111] DEPRECATED: Use .s2i/bin instead of .sti/bin
I0308 03:19:30.658670       1 install.go:236] Using "save-artifacts" installed from "<source-dir>/.sti/bin/save-artifacts"
I0308 03:19:30.658751       1 ignore.go:58] .s2iignore file does not exist
I0308 03:19:30.658764       1 util.go:186] Pulling image "haowang/nodejs-example-3:2c2d43c5" ...
I0308 03:19:30.658774       1 docker.go:308] Pulling Docker image haowang/nodejs-example-3:2c2d43c5 ...
I0308 03:19:37.315908       1 docker.go:311] An error was received from the PullImage call: Error: image haowang/nodejs-example-3:2c2d43c5 not found
I0308 03:19:37.358500       1 sti.go:349] Unable to pull previously build "haowang/nodejs-example-3:2c2d43c5" image: unable to get haowang/nodejs-example-3:2c2d43c5
I0308 03:19:37.358542       1 sti.go:148] Clean build will be performed
I0308 03:19:37.358549       1 sti.go:151] Performing source build from file:///tmp/s2i-build429232122/upload/src
I0308 03:19:37.358556       1 sti.go:164] Running "assemble" in "haowang/nodejs-example-3:2c2d43c5"
I0308 03:19:37.358568       1 sti.go:412] Using image name brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/nodejs-010-rhel7:latest
I0308 03:19:37.358599       1 environment.go:31] DEPRECATED: Use .s2i/environment instead of .sti/environment
I0308 03:19:37.358656       1 environment.go:60] Setting 1 environment variables provided by environment file in sources
I0308 03:19:37.358812       1 sti.go:517] starting the source uploading ...
I0308 03:19:37.358836       1 tar.go:193] Adding "/tmp/s2i-build429232122/upload" to tar ...
I0308 03:19:37.359015       1 tar.go:260] Adding to tar: /tmp/s2i-build429232122/upload/scripts/assemble as scripts/assemble
I0308 03:19:37.454780       1 docker.go:456] Both scripts and untarred source will be placed in '/tmp'
I0308 03:19:37.454828       1 docker.go:479] Running "/bin/sh -c tar -C /tmp -xf - && /tmp/scripts/assemble" command in container ...
I0308 03:19:37.454851       1 docker.go:614] Creating container  using config: &{Hostname: Domainname: User: Memory:0 MemorySwap:0 MemoryReservation:0 KernelMemory:0 CPUShares:0 CPUSet: AttachStdin:false AttachStdout:true AttachStderr:false PortSpecs:[] ExposedPorts:map[] StopSignal: Tty:false OpenStdin:true StdinOnce:true Env:[test=test OPENSHIFT_BUILD_NAMESPACE=haowang OPENSHIFT_BUILD_SOURCE=https://github.com/wanghaoran1988/nodejs-ex.git OPENSHIFT_BUILD_NAME=nodejs-example-3] Cmd:[/bin/sh -c tar -C /tmp -xf - && /tmp/scripts/assemble] DNS:[] Image:brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/nodejs-010-rhel7:latest Volumes:map[] VolumeDriver: VolumesFrom: WorkingDir: MacAddress: Entrypoint:[] NetworkDisabled:false SecurityOpts:[] OnBuild:[] Mounts:[] Labels:map[]}, hostconfig: &{Binds:[] CapAdd:[] CapDrop:[] GroupAdd:[] ContainerIDFile: LxcConf:[] Privileged:false PortBindings:map[] Links:[] PublishAllPorts:false DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] VolumesFrom:[] NetworkMode:container:6c1a8a557b68ac5c790548411352d57124a72f67c3e6cc27b1a0947acb17bf4c IpcMode: PidMode: UTSMode: RestartPolicy:{Name: MaximumRetryCount:0} Devices:[] LogConfig:{Type: Config:map[]} ReadonlyRootfs:false SecurityOpt:[] CgroupParent: Memory:9223372036854775807 MemorySwap:9223372036854775807 MemorySwappiness:0 OOMKillDisable:false CPUShares:2 CPUSet: CPUSetCPUs: CPUSetMEMs: CPUQuota:-1 CPUPeriod:100000 BlkioWeight:0 Ulimits:[] VolumeDriver: OomScoreAdj:0}
I0308 03:19:41.360745       1 docker.go:621] Attaching to container
I0308 03:19:41.369113       1 docker.go:627] Starting container
I0308 03:19:41.369371       1 tar.go:260] Adding to tar: /tmp/s2i-build429232122/upload/scripts/save-artifacts as scripts/save-artifacts
I0308 03:19:41.374156       1 tar.go:260] Adding to tar: /tmp/s2i-build429232122/upload/src/.gitignore as src/.gitignore
I0308 03:19:41.374353       1 tar.go:260] Adding to tar: /tmp/s2i-build429232122/upload/src/.sti/environment as src/.sti/environment
I0308 03:19:41.374437       1 tar.go:260] Adding to tar: /tmp/s2i-build429232122/upload/src/README.md as src/README.md
I0308 03:19:41.374582       1 tar.go:260] Adding to tar: /tmp/s2i-build429232122/upload/src/openshift/templates/nodejs-mongodb.json as src/openshift/templates/nodejs-mongodb.json
I0308 03:19:41.374684       1 tar.go:260] Adding to tar: /tmp/s2i-build429232122/upload/src/openshift/templates/nodejs.json as src/openshift/templates/nodejs.json
I0308 03:19:41.374933       1 tar.go:260] Adding to tar: /tmp/s2i-build429232122/upload/src/package.json as src/package.json
I0308 03:19:41.375012       1 tar.go:260] Adding to tar: /tmp/s2i-build429232122/upload/src/server.js as src/server.js
I0308 03:19:41.375108       1 tar.go:260] Adding to tar: /tmp/s2i-build429232122/upload/src/views/index.html as src/views/index.html
I0308 03:19:42.444732       1 docker.go:548] Waiting for container
I0308 03:19:44.757821       1 sti.go:551] ---> Installing application source
I0308 03:19:44.794319       1 sti.go:551] ---> Building your Node application from source
^C[vagrant@ose ~]$ oc get is nodejs-example -o json
{
    "kind": "ImageStream",
    "apiVersion": "v1",
    "metadata": {
        "name": "nodejs-example",
        "namespace": "haowang",
        "selfLink": "/oapi/v1/namespaces/haowang/imagestreams/nodejs-example",
        "uid": "7f567f9c-e505-11e5-a697-fa163ed32d7c",
        "resourceVersion": "31363",
        "generation": 1,
        "creationTimestamp": "2016-03-08T08:12:27Z",
        "labels": {
            "app": "nodejs-example",
            "template": "nodejs-example"
        },
        "annotations": {
            "description": "Keeps track of changes in the application image",
            "openshift.io/generated-by": "OpenShiftNewApp"
        }
    },
    "spec": {},
    "status": {
        "dockerImageRepository": "172.31.215.114:5000/haowang/nodejs-example",
        "tags": [
            {
                "tag": "latest",
                "items": [
                    {
                        "created": "2016-03-08T08:20:52Z",
                        "dockerImageReference": "172.31.215.114:5000/haowang/nodejs-example@sha256:f7659498e5ea74b3c84f4bccc9600754474d4ffa6480000dfacdaff6752fa271",
                        "image": "sha256:f7659498e5ea74b3c84f4bccc9600754474d4ffa6480000dfacdaff6752fa271",
                        "generation": 1
                    },
                    {
                        "created": "2016-03-08T08:14:59Z",
                        "dockerImageReference": "172.31.215.114:5000/haowang/nodejs-example@sha256:2ecc4db258ef474f75f3806016d6f01309fd4591c8a58a75b85571d797834ca5",
                        "image": "sha256:2ecc4db258ef474f75f3806016d6f01309fd4591c8a58a75b85571d797834ca5",
                        "generation": 1
                    },
                    {
                        "created": "2016-03-08T08:13:13Z",
                        "dockerImageReference": "172.31.215.114:5000/haowang/nodejs-example@sha256:ee79cd4fd4b7830714686e5c162f2afd332d924919297af7d860919a049c9053",
                        "image": "sha256:ee79cd4fd4b7830714686e5c162f2afd332d924919297af7d860919a049c9053",
                        "generation": 1
                    }
                ]
            }
        ]
    }
}
[vagrant@ose ~]$ oc get bc -o json
{
    "kind": "List",
    "apiVersion": "v1",
    "metadata": {},
    "items": [
        {
            "kind": "BuildConfig",
            "apiVersion": "v1",
            "metadata": {
                "name": "nodejs-example",
                "namespace": "haowang",
                "selfLink": "/oapi/v1/namespaces/haowang/buildconfigs/nodejs-example",
                "uid": "7fbc456a-e505-11e5-a697-fa163ed32d7c",
                "resourceVersion": "31267",
                "creationTimestamp": "2016-03-08T08:12:28Z",
                "labels": {
                    "app": "nodejs-example",
                    "template": "nodejs-example"
                },
                "annotations": {
                    "description": "Defines how to build the application",
                    "openshift.io/generated-by": "OpenShiftNewApp"
                }
            },
            "spec": {
                "triggers": [
                    {
                        "type": "ImageChange",
                        "imageChange": {
                            "lastTriggeredImageID": "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/nodejs-010-rhel7:latest"
                        }
                    },
                    {
                        "type": "ConfigChange"
                    },
                    {
                        "type": "GitHub",
                        "github": {
                            "secret": "qSLm4nWGL2xUchGpkkvIVmj7b1agmNmf4YLekYrF"
                        }
                    },
                    {
                        "type": "Generic",
                        "generic": {
                            "secret": "ojh1c2VBtdtaeFHjTpDg51asUWq1iaS7uttd8QxL"
                        }
                    }
                ],
                "source": {
                    "type": "Git",
                    "git": {
                        "uri": "https://github.com/wanghaoran1988/nodejs-ex.git"
                    },
                    "secrets": null
                },
                "strategy": {
                    "type": "Source",
                    "sourceStrategy": {
                        "from": {
                            "kind": "ImageStreamTag",
                            "namespace": "openshift",
                            "name": "nodejs:0.10"
                        },
                        "incremental": true
                    }
                },
                "output": {
                    "to": {
                        "kind": "ImageStreamTag",
                        "name": "nodejs-example:latest"
                    }
                },
                "resources": {},
                "postCommit": {}
            },
            "status": {
                "lastVersion": 3
            }
        }
    ]
}
Comment 1 Rodolfo Carvalho 2016-03-09 10:11:39 EST
This should have been fixed by this PR (merged two days ago):
https://github.com/openshift/origin/pull/7713

Looking at the logs, seems that the version of openshift used here doesn't have that patch.

Wang Haoran, could you please try with a more recent version of origin?
Thanks
Comment 2 Wang Haoran 2016-03-09 21:40:20 EST
verified with the following version, and fell free to move the bug to ON_QA
openshift v1.1.3-553-g19dbf2a
kubernetes v1.2.0-alpha.7-703-gbc4550d
etcd 2.2.5
openshift/origin-docker-builder               latest              0d57ad7c9cc1

this is the incremental build log:
I0310 02:31:48.727816       1 util.go:186] Pulling image "172.30.214.88:5000/haowang/ruby-hello-world:latest" ...
I0310 02:31:48.727825       1 docker.go:308] Pulling Docker image 172.30.214.88:5000/haowang/ruby-hello-world:latest ...
I0310 02:31:49.302541       1 sti.go:146] Existing image for tag haowang/ruby-hello-world-6:a9d2d308 detected for incremental build
I0310 02:31:49.302616       1 sti.go:151] Performing source build from file:///tmp/s2i-build547406922/upload/src
I0310 02:31:49.302697       1 sti.go:377] Saving build artifacts from image 172.30.214.88:5000/haowang/ruby-hello-world:latest to path /tmp/s2i-build547406922/upload/artifacts
I0310 02:31:49.308702       1 sti.go:388] The assemble user is not set, defaulting to "1001" user
I0310 02:31:49.322838       1 docker.go:410] Image contains io.openshift.s2i.scripts-url set to 'image:///usr/libexec/s2i'
I0310 02:31:49.322853       1 docker.go:465] Base directory for STI scripts is '/usr/libexec/s2i'. Untarring destination is '/tmp'.
I0310 02:31:49.322861       1 docker.go:479] Running "/usr/libexec/s2i/save-artifacts" command in container ...
I0310 02:31:49.322871       1 docker.go:614] Creating container  using config: &{Hostname: Domainname: User:1001 Memory:0 MemorySwap:0 MemoryReservation:0 KernelMemory:0 CPUShares:0 CPUSet: AttachStdin:false AttachStdout:true AttachStderr:false PortSpecs:[] ExposedPorts:map[] StopSignal: Tty:false OpenStdin:false StdinOnce:false Env:[] Cmd:[/usr/libexec/s2i/save-artifacts] DNS:[] Image:172.30.214.88:5000/haowang/ruby-hello-world:latest Volumes:map[] VolumeDriver: VolumesFrom: WorkingDir: MacAddress: Entrypoint:[] NetworkDisabled:false SecurityOpts:[] OnBuild:[] Mounts:[] Labels:map[]}, hostconfig: &{Binds:[] CapAdd:[] CapDrop:[] GroupAdd:[] ContainerIDFile: LxcConf:[] Privileged:false PortBindings:map[] Links:[] PublishAllPorts:false DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] VolumesFrom:[] NetworkMode:container:263f40554cc16c48874c46549b54af4a14bf9d9911c4b3eeeadfe57a129e623a IpcMode: PidMode: UTSMode: RestartPolicy:{Name: MaximumRetryCount:0} Devices:[] LogConfig:{Type: Config:map[]} ReadonlyRootfs:false SecurityOpt:[] CgroupParent: Memory:9223372036854775807 MemorySwap:9223372036854775807 MemorySwappiness:0 OOMKillDisable:false CPUShares:2 CPUSet: CPUSetCPUs: CPUSetMEMs: CPUQuota:-1 CPUPeriod:100000 BlkioWeight:0 Ulimits:[] VolumeDriver: OomScoreAdj:0}
Comment 3 Wang Haoran 2016-03-11 01:48:29 EST
Verified with :
openshift v3.2.0.1
kubernetes v1.2.0-alpha.7-703-gbc4550d
etcd 2.2.5
Comment 4 Wang Haoran 2016-03-14 01:20:15 EDT
reopen this bug, since currently can pull the previous image successfully when do incremental build, but if test with a repo contains the save-artifacts script, build will hang ...
Steps:
1. create app using repo contains save-artifacts script
  oc new-app https://github.com/wanghaoran1988/nodejs-ex
2. update the bc to enable incremental build,wait the first build complete
3. start a new build, and watch the build logs, the build will stop at "Container wait returns with 0 and <nil>"

[root@ip-172-18-3-72 ~]# oc logs -f build/nodejs-ex-3
I0314 05:07:23.532627       1 builder.go:46] $BUILD env var is {"kind":"Build","apiVersion":"v1","metadata":{"name":"nodejs-ex-3","namespace":"haowang","selfLink":"/oapi/v1/namespaces/haowang/builds/nodejs-ex-3","uid":"a010f17a-e9a2-11e5-a7fd-0eb9d8036749","resourceVersion":"1205","creationTimestamp":"2016-03-14T05:07:18Z","labels":{"app":"nodejs-ex","buildconfig":"nodejs-ex","openshift.io/build-config.name":"nodejs-ex"},"annotations":{"openshift.io/build.number":"3"}},"spec":{"serviceAccount":"builder","source":{"type":"Git","git":{"uri":"https://github.com/wanghaoran1988/nodejs-ex"},"secrets":[]},"strategy":{"type":"Source","sourceStrategy":{"from":{"kind":"DockerImage","name":"registry.access.redhat.com/openshift3/nodejs-010-rhel7:latest"},"incremental":true}},"output":{"to":{"kind":"DockerImage","name":"172.30.207.39:5000/haowang/nodejs-ex:latest"},"pushSecret":{"name":"builder-dockercfg-giut4"}},"resources":{},"postCommit":{}},"status":{"phase":"New","outputDockerImageReference":"172.30.207.39:5000/haowang/nodejs-ex:latest","config":{"kind":"BuildConfig","namespace":"haowang","name":"nodejs-ex"}}}
 
I0314 05:07:23.536673       1 builder.go:57] Master version "v1.1.3-631-g2d2d659", Builder version "v1.1.3-631-g2d2d659"
I0314 05:07:23.537547       1 builder.go:145] Running build with cgroup limits: api.CGroupLimits{MemoryLimitBytes:9223372036854775807, CPUShares:2, CPUPeriod:100000, CPUQuota:-1, MemorySwap:9223372036854775807}
I0314 05:07:23.544657       1 sti.go:189] With force pull false, setting policies to if-not-present
I0314 05:07:23.544672       1 sti.go:195] The value of ALLOWED_UIDS is [1-]
I0314 05:07:23.544685       1 sti.go:203] The value of DROP_CAPS is [KILL,MKNOD,SETGID,SETUID,SYS_CHROOT]
I0314 05:07:23.544697       1 cfg.go:45] Locating docker auth for image registry.access.redhat.com/openshift3/nodejs-010-rhel7:latest and type PULL_DOCKERCFG_PATH
I0314 05:07:23.544752       1 cfg.go:111] Using Docker authentication configuration in '/root/.docker/config.json'
I0314 05:07:23.545224       1 cfg.go:57] Problem accessing /root/.docker/config.json: stat /root/.docker/config.json: no such file or directory
I0314 05:07:23.545241       1 cfg.go:45] Locating docker auth for image 172.30.207.39:5000/haowang/nodejs-ex:latest and type PUSH_DOCKERCFG_PATH
I0314 05:07:23.545262       1 cfg.go:111] Using Docker authentication configuration in '/var/run/secrets/openshift.io/push/.dockercfg'
I0314 05:07:23.545428       1 cfg.go:83] Using serviceaccount user for Docker authentication for image 172.30.207.39:5000/haowang/nodejs-ex:latest
I0314 05:07:23.551690       1 docker.go:291] Using locally available image "registry.access.redhat.com/openshift3/nodejs-010-rhel7:latest"
I0314 05:07:23.552980       1 sti.go:222] Creating a new S2I builder with build config: "Builder Name:\t\t\tNode.js 0.10\nBuilder Image:\t\t\tregistry.access.redhat.com/openshift3/nodejs-010-rhel7:latest\nSource:\t\t\t\tfile:///tmp/s2i-build580693189/upload/src\nOutput Image Tag:\t\thaowang/nodejs-ex-3:116b14c7\nEnvironment:\t\t\tOPENSHIFT_BUILD_NAME=nodejs-ex-3,OPENSHIFT_BUILD_NAMESPACE=haowang,OPENSHIFT_BUILD_SOURCE=https://github.com/wanghaoran1988/nodejs-ex\nIncremental Build:\t\t\x1b[1menabled\x1b[0m\nIncremental Image Pull User:\tserviceaccount\nRemove Old Build:\t\tdisabled\nBuilder Pull Policy:\t\tif-not-present\nPrevious Image Pull Policy:\talways\nQuiet:\t\t\t\tdisabled\nLayered Build:\t\t\tdisabled\nWorkdir:\t\t\t/tmp/s2i-build580693189\nDocker NetworkMode:\t\tcontainer:aabad3e0f88a6d7446605012fd6f8a63754a647a232f0b0de4a496e29e76924d\nDocker Endpoint:\t\tunix:///var/run/docker.sock\n"
I0314 05:07:23.554381       1 docker.go:291] Using locally available image "registry.access.redhat.com/openshift3/nodejs-010-rhel7:latest"
I0314 05:07:23.557916       1 docker.go:291] Using locally available image "registry.access.redhat.com/openshift3/nodejs-010-rhel7:latest"
I0314 05:07:23.557935       1 docker.go:411] Image contains io.openshift.s2i.scripts-url set to 'image:///usr/libexec/s2i'
I0314 05:07:23.557970       1 sti.go:228] Starting S2I build from haowang/nodejs-ex-3 BuildConfig ...
I0314 05:07:23.557978       1 sti.go:140] Preparing to build haowang/nodejs-ex-3:116b14c7
I0314 05:07:23.571536       1 source.go:197] Downloading "https://github.com/wanghaoran1988/nodejs-ex" ...
I0314 05:07:23.571567       1 source.go:109] git ls-remote https://github.com/wanghaoran1988/nodejs-ex --heads
I0314 05:07:23.571612       1 repository.go:290] Executing git ls-remote https://github.com/wanghaoran1988/nodejs-ex --heads
I0314 05:07:24.054552       1 source.go:208] Cloning source from https://github.com/wanghaoran1988/nodejs-ex
I0314 05:07:24.054834       1 repository.go:290] Executing git clone --recursive https://github.com/wanghaoran1988/nodejs-ex /tmp/s2i-build580693189/upload/src
I0314 05:07:24.486666       1 repository.go:315] Out: Cloning into '/tmp/s2i-build580693189/upload/src'...
I0314 05:07:24.486703       1 repository.go:290] Executing git config --get remote.origin.url
I0314 05:07:24.487908       1 repository.go:315] Out: https://github.com/wanghaoran1988/nodejs-ex
I0314 05:07:24.487927       1 repository.go:290] Executing git rev-parse --abbrev-ref HEAD
I0314 05:07:24.489089       1 repository.go:315] Out: master
I0314 05:07:24.489140       1 repository.go:290] Executing git rev-parse --verify HEAD
I0314 05:07:24.490230       1 repository.go:315] Out: 8edcd691bce0fa62870b6927a532509a4f9ddb46
I0314 05:07:24.490261       1 repository.go:290] Executing git --no-pager show -s --format=%an HEAD
I0314 05:07:24.491769       1 repository.go:315] Out: Haoran Wang
I0314 05:07:24.491789       1 repository.go:290] Executing git --no-pager show -s --format=%ae HEAD
I0314 05:07:24.493125       1 repository.go:315] Out: haowang@redhat.com
I0314 05:07:24.493146       1 repository.go:290] Executing git --no-pager show -s --format=%cn HEAD
I0314 05:07:24.506763       1 repository.go:315] Out: Haoran Wang
I0314 05:07:24.506785       1 repository.go:290] Executing git --no-pager show -s --format=%ce HEAD
I0314 05:07:24.508102       1 repository.go:315] Out: haowang@redhat.com
I0314 05:07:24.508155       1 repository.go:290] Executing git --no-pager show -s --format=%ad HEAD
I0314 05:07:24.509567       1 repository.go:315] Out: Thu Dec 3 11:13:02 2015 +0800
I0314 05:07:24.509588       1 repository.go:290] Executing git --no-pager show -s --format=%<(80,trunc)%s HEAD
I0314 05:07:24.510914       1 repository.go:315] Out: add test env
I0314 05:07:24.510940       1 common.go:86] Setting build revision to &api.GitSourceRevision{Commit:"8edcd691bce0fa62870b6927a532509a4f9ddb46", Author:api.SourceControlUser{Name:"Haoran Wang", Email:"haowang@redhat.com"}, Committer:api.SourceControlUser{Name:"Haoran Wang", Email:"haowang@redhat.com"}, Message:"add test env"}
I0314 05:07:24.569908       1 install.go:111] DEPRECATED: Use .s2i/bin instead of .sti/bin
I0314 05:07:24.569965       1 install.go:236] Using "assemble" installed from "<source-dir>/.sti/bin/assemble"
I0314 05:07:24.570000       1 install.go:236] Using "run" installed from "image:///usr/libexec/s2i/run"
I0314 05:07:24.570014       1 install.go:111] DEPRECATED: Use .s2i/bin instead of .sti/bin
I0314 05:07:24.570038       1 install.go:236] Using "save-artifacts" installed from "<source-dir>/.sti/bin/save-artifacts"
I0314 05:07:24.570154       1 ignore.go:58] .s2iignore file does not exist
I0314 05:07:24.570168       1 util.go:186] Pulling image "172.30.207.39:5000/haowang/nodejs-ex:latest" ...
I0314 05:07:24.570178       1 docker.go:309] Pulling Docker image 172.30.207.39:5000/haowang/nodejs-ex:latest ...
I0314 05:07:24.761334       1 sti.go:150] Existing image for tag 172.30.207.39:5000/haowang/nodejs-ex:latest detected for incremental build
I0314 05:07:24.761353       1 sti.go:155] Performing source build from file:///tmp/s2i-build580693189/upload/src
I0314 05:07:24.761436       1 sti.go:381] Saving build artifacts from image 172.30.207.39:5000/haowang/nodejs-ex:latest to path /tmp/s2i-build580693189/upload/artifacts
I0314 05:07:24.763309       1 sti.go:392] The assemble user is not set, defaulting to "1001" user
I0314 05:07:24.765105       1 docker.go:457] Both scripts and untarred source will be placed in '/tmp'
I0314 05:07:24.765124       1 docker.go:480] Running "/tmp/scripts/save-artifacts" command in container ...
I0314 05:07:24.765136       1 docker.go:616] Creating container  using config: &{Hostname: Domainname: User:1001 Memory:0 MemorySwap:0 MemoryReservation:0 KernelMemory:0 CPUShares:0 CPUSet: AttachStdin:false AttachStdout:true AttachStderr:false PortSpecs:[] ExposedPorts:map[] StopSignal: Tty:false OpenStdin:false StdinOnce:false Env:[] Cmd:[/tmp/scripts/save-artifacts] DNS:[] Image:172.30.207.39:5000/haowang/nodejs-ex:latest Volumes:map[] VolumeDriver: VolumesFrom: WorkingDir: MacAddress: Entrypoint:[] NetworkDisabled:false SecurityOpts:[] OnBuild:[] Mounts:[] Labels:map[]}, hostconfig: &{Binds:[] CapAdd:[] CapDrop:[KILL MKNOD SETGID SETUID SYS_CHROOT] GroupAdd:[] ContainerIDFile: LxcConf:[] Privileged:false PortBindings:map[] Links:[] PublishAllPorts:false DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] VolumesFrom:[] NetworkMode:container:aabad3e0f88a6d7446605012fd6f8a63754a647a232f0b0de4a496e29e76924d IpcMode: PidMode: UTSMode: RestartPolicy:{Name: MaximumRetryCount:0} Devices:[] LogConfig:{Type: Config:map[]} ReadonlyRootfs:false SecurityOpt:[] CgroupParent: Memory:9223372036854775807 MemorySwap:9223372036854775807 MemorySwappiness:0 OOMKillDisable:false CPUShares:2 CPUSet: CPUSetCPUs: CPUSetMEMs: CPUQuota:-1 CPUPeriod:100000 BlkioWeight:0 Ulimits:[] VolumeDriver: OomScoreAdj:0}
I0314 05:07:25.231548       1 docker.go:623] Attaching to container
I0314 05:07:25.233182       1 docker.go:629] Starting container
I0314 05:07:25.615653       1 tar.go:296] Creating directory /tmp/s2i-build580693189/upload/artifacts/.npm
.......
I0314 05:07:26.253225       1 tar.go:366] Extracting/writing /tmp/s2i-build580693189/upload/artifacts/.npm/vary/1.0.1/package/package.json
I0314 05:07:26.253287       1 tar.go:305] Creating directory /tmp/s2i-build580693189/upload/artifacts/.npm/vary/1.0.1
I0314 05:07:26.253297       1 tar.go:356] Creating /tmp/s2i-build580693189/upload/artifacts/.npm/vary/1.0.1/package.tgz
I0314 05:07:26.253752       1 tar.go:366] Extracting/writing /tmp/s2i-build580693189/upload/artifacts/.npm/vary/1.0.1/package.tgz
I0314 05:07:26.253810       1 tar.go:335] Done extracting tar stream
I0314 05:07:26.253821       1 docker.go:549] Waiting for container
I0314 05:07:26.319620       1 docker.go:551] Container wait returns with 0 and <nil>
Comment 5 Ben Parees 2016-03-16 22:31:29 EDT
Confirmed this is broken in s2i itself as well, incremental builds hang on attaching to the container.

Rodolfo, I wasn't able to chase it much further than that, but it's very easy to recreate w/ an incremental build using s2i.
Comment 6 Ben Parees 2016-03-16 22:40:44 EDT
ok I chased it a little further, the problem was introduced by this commit:
https://github.com/openshift/source-to-image/commit/c0b4d96f419747b19f61d1cc04a7e1e773b0fee1

prior to that commit, we were getting an error when attaching to the container (because the pipe was closed), but things were able to still proceed (I think it just means we get no output from the save-artifacts script when the attach fails).

after that commit, the attach doesn't fail, it hangs.

Which is not to say i think the fix in that commit was wrong, but it has exposed a deeper issue w/ attach.  So the quick fix is to stop attaching.  The better fix is to figure out why attach is hanging.

(this is the attach call that's hanging: https://github.com/openshift/source-to-image/blob/master/pkg/docker/docker.go#L540)
Comment 7 Wang Haoran 2016-03-16 23:14:34 EDT
Hi, Ben , I found when the pod hang their , the language container with the save-artifect already stopped, so attach hanging there without receive the attach sucucessful signal, we should found out why failed to attach to the container when the container is running.
Comment 8 Ben Parees 2016-03-16 23:46:13 EDT
the more i look at it, the less I understand why we're attaching stdout for the save-artifacts path.  We already want to send the stdout to the tar function we set as the OnStart, so i don't know that we should be attaching to stdout again.  it's possible this second attach is what's causing the hang.
Comment 9 Rodolfo Carvalho 2016-03-17 04:31:02 EDT
Ben, perhaps the time has come to review that code.

When I wrote the helper in origin to start a container I could not reuse the one in S2I, and the end result I got was much simpler than what's in S2I.

Possibly we don't even need to attach, all we need is to stream the logs (which is a different API call).

I'll look into it.
Comment 10 Rodolfo Carvalho 2016-03-17 05:13:54 EDT
Tentative to reproduce:


First run (image built from scratch):

[vagrant@localhost ~]$ s2i build https://github.com/bparees/incremental-app centos/ruby-22-centos7 incr-test --incremental
I0317 09:05:37.455744 17841 docker.go:287] Image "centos/ruby-22-centos7:latest" not available locally, pulling ...
I0317 09:06:20.359652 17841 clone.go:32] Downloading "https://github.com/bparees/incremental-app" ...
I0317 09:06:21.861743 17841 install.go:111] DEPRECATED: Use .s2i/bin instead of .sti/bin
I0317 09:06:21.862772 17841 install.go:236] Using "assemble" installed from "<source-dir>/.sti/bin/assemble"
I0317 09:06:21.863531 17841 install.go:236] Using "run" installed from "image:///usr/libexec/s2i/run"
I0317 09:06:21.863597 17841 install.go:111] DEPRECATED: Use .s2i/bin instead of .sti/bin
I0317 09:06:21.863631 17841 install.go:236] Using "save-artifacts" installed from "<source-dir>/.sti/bin/save-artifacts"
I0317 09:06:21.866941 17841 docker.go:287] Image "incr-test:latest" not available locally, pulling ...
I0317 09:06:27.210619 17841 environment.go:31] DEPRECATED: Use .s2i/environment instead of .sti/environment
I0317 09:06:27.210780 17841 environment.go:60] Setting 1 environment variables provided by environment file in sources
---> Installing application source ...
---> Building your Ruby application from source ...
---> Running 'bundle install --deployment' ...
Fetching gem metadata from https://rubygems.org/..........
Installing rack 1.6.4
Installing rack-protection 1.5.3
Installing tilt 2.0.1
Installing sinatra 1.4.6
Using bundler 1.7.8
Your bundle is complete!
It was installed into ./bundle
---> Cleaning up unused ruby gems ...
I0317 09:06:43.398408 17841 environment.go:31] DEPRECATED: Use .s2i/environment instead of .sti/environment
I0317 09:06:43.398468 17841 environment.go:60] Setting 1 environment variables provided by environment file in sources


OK.
Second run (incremental build):

[vagrant@localhost ~]$ s2i build https://github.com/bparees/incremental-app centos/ruby-22-centos7 incr-test --incremental 
I0317 09:09:05.923786 18837 clone.go:32] Downloading "https://github.com/bparees/incremental-app" ...
I0317 09:09:07.102825 18837 install.go:111] DEPRECATED: Use .s2i/bin instead of .sti/bin
I0317 09:09:07.102882 18837 install.go:236] Using "assemble" installed from "<source-dir>/.sti/bin/assemble"
I0317 09:09:07.103292 18837 install.go:236] Using "run" installed from "image:///usr/libexec/s2i/run"
I0317 09:09:07.103326 18837 install.go:111] DEPRECATED: Use .s2i/bin instead of .sti/bin
I0317 09:09:07.103341 18837 install.go:236] Using "save-artifacts" installed from "<source-dir>/.sti/bin/save-artifacts"
(hung up)


Reproduced successfully.

Sending SIGKILL to the process above I got:


^\SIGQUIT: quit
PC=0x441e71

goroutine 6 [syscall, 1 minutes]:
runtime.notetsleepg(0xb4f8c0, 0xffffffffffffffff, 0x1)
        /usr/lib/golang/src/runtime/lock_futex.go:201 +0x52 fp=0xc208019768 sp=0xc208019740
runtime.signal_recv(0x0)
        /usr/lib/golang/src/runtime/sigqueue.go:109 +0x135 fp=0xc2080197a0 sp=0xc208019768
os/signal.loop()
        /usr/lib/golang/src/os/signal/signal_unix.go:21 +0x1f fp=0xc2080197e0 sp=0xc2080197a0
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc2080197e8 sp=0xc2080197e0
created by os/signal.init·1
        /usr/lib/golang/src/os/signal/signal_unix.go:27 +0x35

goroutine 1 [semacquire]:
sync.(*WaitGroup).Wait(0xc20811dc40)
        /usr/lib/golang/src/sync/waitgroup.go:132 +0x169
github.com/openshift/source-to-image/pkg/docker.runContainerWait(0xc20811dc40, 0xc20805aaa0, 0xc208041380, 0x0, 0x0)
        /home/rodolfo/openshift/src/github.com/openshift/source-to-image/_output/local/go/src/github.com/openshift/source-to-image/pkg/docker/docker.go:553 +0x249
github.com/openshift/source-to-image/pkg/docker.(*stiDocker).RunContainer(0xc20805aaa0, 0x7ffd48124581, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/rodolfo/openshift/src/github.com/openshift/source-to-image/_output/local/go/src/github.com/openshift/source-to-image/pkg/docker/docker.go:644 +0xc2b
github.com/openshift/source-to-image/pkg/build/strategies/sti.(*STI).Save(0xc2080a4000, 0xc20806d900, 0x0, 0x0)
        /home/rodolfo/openshift/src/github.com/openshift/source-to-image/_output/local/go/src/github.com/openshift/source-to-image/pkg/build/strategies/sti/sti.go:414 +0xa22
github.com/openshift/source-to-image/pkg/build/strategies/sti.(*STI).Build(0xc2080a4000, 0xc20806d900, 0x0, 0x0, 0x0)
        /home/rodolfo/openshift/src/github.com/openshift/source-to-image/_output/local/go/src/github.com/openshift/source-to-image/pkg/build/strategies/sti/sti.go:157 +0x583
main.func·002(0xc2080ca3c0, 0xc208057040, 0x3, 0x4)
        /home/rodolfo/openshift/src/github.com/openshift/source-to-image/_output/local/go/src/github.com/openshift/source-to-image/cmd/s2i/main.go:148 +0x11fa
github.com/spf13/cobra.(*Command).execute(0xc2080ca3c0, 0xc208056f80, 0x4, 0x4, 0x0, 0x0)
        /home/rodolfo/openshift/src/github.com/openshift/source-to-image/Godeps/_workspace/src/github.com/spf13/cobra/command.go:547 +0x82f
github.com/spf13/cobra.(*Command).Execute(0xc2080ca000, 0x0, 0x0)
        /home/rodolfo/openshift/src/github.com/openshift/source-to-image/Godeps/_workspace/src/github.com/spf13/cobra/command.go:630 +0x414
main.main()
        /home/rodolfo/openshift/src/github.com/openshift/source-to-image/_output/local/go/src/github.com/openshift/source-to-image/cmd/s2i/main.go:375 +0x781
...



As I suspected, we have a problem in `runContainerWait`.

I'm going to build s2i with -race enabled and let the runtime give us more hints...
Comment 11 Rodolfo Carvalho 2016-03-17 06:09:08 EDT
From the stacktrace above, this is the line where we're waiting forever:

https://github.com/openshift/source-to-image/blob/9fe0bf74dd7658364e202ecce2a28f8cb7f83f66/pkg/docker/docker.go#L553

(sti.go) Build -> (sti.go) Save -> (docker.go) RunContainer -> (docker.go) runContainerWait


Didn't find any data races for this flow with the runtime race detector.

Working on a fix to the RunContainer logic.
Comment 12 Rodolfo Carvalho 2016-03-17 19:05:26 EDT
This PR should fix the problem in S2I:
https://github.com/openshift/source-to-image/pull/430

Once it gets merged we'll need to bump the version in origin.
Comment 13 Rodolfo Carvalho 2016-03-18 11:31:07 EDT
This is the origin PR we're waiting for:
https://github.com/openshift/origin/pull/8115
Comment 14 Wang Haoran 2016-03-20 22:29:49 EDT
I make a test again with latest origin AMI,the hang problem disappeared ,but find a new problem.
When I use openshift/nodejs-010-centos7, the incremental build failed with logs:
I0321 02:06:41.904260       1 tar.go:260] Adding to tar: /tmp/s2i-build849450176/upload/src/package.json as src/package.json
I0321 02:06:41.904325       1 tar.go:260] Adding to tar: /tmp/s2i-build849450176/upload/src/server.js as src/server.js
I0321 02:06:41.904449       1 tar.go:260] Adding to tar: /tmp/s2i-build849450176/upload/src/views/index.html as src/views/index.html
I0321 02:06:41.998286       1 sti.go:563] Restoring build artifacts
E0321 02:06:42.027486       1 util.go:91] mv: cannot move '/tmp/artifacts/.npm' to '/opt/app-root/src/.npm': Directory not empty
I0321 02:06:42.465807       1 docker.go:614] Removing container "41ad3e7beb1adca5158fe86fcc5dc3d1099320b67f2cf6649f434e4e24da0d35" ...
I0321 02:06:42.596145       1 docker.go:618] Removed container "41ad3e7beb1adca5158fe86fcc5dc3d1099320b67f2cf6649f434e4e24da0d35"
I0321 02:06:42.600316       1 cleanup.go:23] Removing temporary directory /tmp/s2i-build849450176
I0321 02:06:42.600333       1 fs.go:156] Removing directory '/tmp/s2i-build849450176'
F0321 02:06:43.009280       1 builder.go:204] Error: build error: non-zero (13) exit code from openshift/nodejs-010-centos7@sha256:3379dca65270afbbb4a7b7dd563b52a31a9f0f8c1dadbdb8b6ebd5e2d81e7476

and I found the there are some nodejs dependency inside this image,so failed.
[root@ip-172-18-6-25 ~]# docker run -ti docker.io/openshift/nodejs-010-centos7 /bin/bash
bash-4.2$ ls -rlta
total 16
drwxr-xr-x.   3 default root 4096 Mar 16 18:55 .pki
drwxr-xr-x.   4 default root 4096 Mar 16 18:55 ..
drwxr-xr-x.   4 default root 4096 Mar 16 19:00 .
drwxr-xr-x. 136 default root 4096 Mar 16 19:00 .npm

but incremental build succeed with registry.access.redhat.com/openshift3/nodejs-010-rhel7
[root@ip-172-18-6-25 ~]# docker run -ti registry.access.redhat.com/openshift3/nodejs-010-rhel7 /bin/bash
bash-4.2$ ls -rlt
total 0
bash-4.2$ ls -rlta
total 8
drwxr-xr-x. 4 default root 4096 Mar  1 11:40 ..
drwxr-xr-x. 2 default root 4096 Mar  1 11:40 .


Ben, will you rebuild the image of openshift/nodejs-010-centos7 or make incremental build more compatible ?
Comment 15 Wang Haoran 2016-03-21 02:47:14 EDT
testing on the ose env , the hang problem still there , seems code not merged.
openshift v3.2.0.5
kubernetes v1.2.0-36-g4a3f9c5
etcd 2.2.5
Comment 16 Rodolfo Carvalho 2016-03-21 06:38:19 EDT
Thanks Wang Haoran!


As discussed on IRC, the error message:

E0321 02:06:42.027486       1 util.go:91] mv: cannot move '/tmp/artifacts/.npm' to '/opt/app-root/src/.npm': Directory not empty

is coming from the assemble script:

https://github.com/wanghaoran1988/nodejs-ex/blob/4263a602f7c281f1648af5e9279eaa234aa65876/.sti/bin/assemble#L29


The assemble script needs to be changed to handle that case, not code in s2i/origin.



The difference between the CentOS and RHEL image with regard to the `~/.npm` directory is because in CentOS we install nodemon using npm, while in RHEL its installed via yum.
Comment 17 Rodolfo Carvalho 2016-03-22 05:09:50 EDT
FWIW, after this commit updating the assemble script [1] I can successfully do incremental builds:


$ s2i build https://github.com/wanghaoran1988/nodejs-ex openshift/nodejs-010-centos7 nodetest
...

Several "npm http GET" lines; while


$ s2i build https://github.com/wanghaoran1988/nodejs-ex openshift/nodejs-010-centos7 nodetest --incremental
...

No "npm http GET" lines; all packages are restored and reused.



Haoran, could you confirm this is working for you too?



[1] https://github.com/wanghaoran1988/nodejs-ex/commit/75b12f8681207fffb4ba757b161d4e1bb31a95a2
Comment 18 Wang Haoran 2016-03-22 05:54:07 EDT
Verified with latest puddle ,and update script[1]

openshift v3.2.0.6
kubernetes v1.2.0-36-g4a3f9c5
etcd 2.2.5

[1]https://github.com/wanghaoran1988/nodejs-ex/blob/master/.sti/bin/assemble#L29
Comment 20 errata-xmlrpc 2016-05-12 12:31:49 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:1064

Note You need to log in before you can comment on or make changes to this bug.