Hide Forgot
Description of problem: Given a build that extracts content from an image using: source: images: - as: null from: kind: ImageStreamTag name: pipeline:src paths: - destinationDir: . sourcePath: /go/src/github.com/openshift/origin///. type: Image and expects to reference a file named images/hypershift/Dockerfile.rhel that resides within /go/src/github.com/openshift/origin///. strategy: dockerStrategy: env: - name: "BUILD_LOGLEVEL" value: "5" dockerfilePath: images/hypershift/Dockerfile.rhel The build should work (it worked on 3.11). In 4.x the file gets injected at the wrong path: '/var/lib/containers/storage/overlay/571af4abe3c5c63492db55bab0630a99be4f790724e5b4f88561d0b45a0ee169/merged/go/src/github.com/openshift/origin/images/hypershift/Dockerfile.rhel’ -> ‘/tmp/build/inputs/origin/images/hypershift/Dockerfile.rhel’ which ultimately fails at the build step: F0409 13:02:22.707506 1 helpers.go:119] error: build error: open /tmp/build/inputs/images/hypershift/Dockerfile.rhel: no such file or directory because the file actually exists at: F0409 13:02:22.707506 1 helpers.go:119] error: build error: open /tmp/build/inputs/origin/images/hypershift/Dockerfile.rhel: no such file or directory Which means something has changed from 3.x to 4.x in terms of whether the "origin" dir is included as part of the input file. How reproducible: always Additional Info: our docs say "If the source path ends in /. then the content of the directory will be copied, but the directory itself will not be created at the destination." (https://docs.okd.io/latest/dev_guide/builds/build_inputs.html#image-source) I suspect the multiple slashes are throwing off the behavior. If it were just "sourcePath: /go/src/github.com/openshift/origin/." i think we'd all agree that "origin" should not be included based on that doc. But with "sourcePath: /go/src/github.com/openshift/origin///." it's less clear what is "correct" behavior. But again, we need to re-align this with what 3.x did and 3.x appears to have treated "//" as the dir to be ignored, not "origin", in the case of a trailing "."
> because the file actually exists at: F0409 13:02:22.707506 1 helpers.go:119] error: build error: open /tmp/build/inputs/origin/images/hypershift/Dockerfile.rhel: no such file or directory that shouldn't have been pasted as an error. meant to just say the file actually exists at /tmp/build/inputs/origin/images/hypershift/Dockerfile.rhel Changing the build reference to: dockerfilePath: origin/images/hypershift/Dockerfile.rhel makes the build succeed.
Linking to 1694859 - based on my analysis the symlink issue and this bug are one and the same. The 4.1 copy behavior changed because we use raw Linux cp commands to extract source. Bug is that part of the code used filepath.Join [1], which drops trailing "/.". Adding logic to detect this and pass a glob to cp ("/*") should fix the issue. [1] https://golang.org/pkg/path/filepath/#Join
source: images: - as: null from: kind: ImageStreamTag name: ruby-hello-world:latest namespace: xiuwang paths: - destinationDir: . sourcePath: /opt/app-root/. type: Image strategy: dockerStrategy: dockerfilePath: src/Dockerfile env: - name: BUILD_LOGLEVEL value: "5" type: Docker with 4.0.0-0.ci-2019-04-14-035215 payload 1. sourcepath end up with /. app-root dir has been copied to des dir. The build log: ‘/var/lib/containers/storage/overlay/e4235489f2a0faf66a11b023bd03a236bd1ae49c63f40ef6a6e2a4f00ff34775/merged/opt/app-root/src/Dockerfile’ -> ‘/tmp/build/inputs/app-root/src/Dockerfile’ build failed： F0416 06:57:57.377253 1 helpers.go:119] error: open /tmp/build/inputs/src/Dockerfile: no such file or directory 2.sourcepath end up with ///. , app-root has been copied to des dir. ‘/var/lib/containers/storage/overlay/78fe876c3ac02fdb65431b0c2d3cd8ddc22fa522b45033d06be0752888a449ac/merged/opt/app-root/src/Dockerfile’ -> ‘/tmp/build/inputs/app-root/src/Dockerfile’ Test with 4.0.0-0.ci-2019-04-15-230244 payload 1.sourcepath end up with /. ,app-root doesn't copied to des dir. It's correct hehavior as doc https://docs.okd.io/latest/dev_guide/builds/build_inputs.html#image-source ‘/var/lib/containers/storage/overlay/eee8cfac84977135e5fb97df83908876ceed9f351ba33d52f6f98836ef0e82a5/merged/opt/app-root/./src/Dockerfile’ -> ‘/tmp/build/inputs/./src/Dockerfile’ 2.sourcepath end up with ///. app-root has been copied to des dir. ‘/var/lib/containers/storage/overlay/a1cea26cc9cbe869313b0bd9ea5db966732e45ddd35cdffbf266c4f59333509e/merged/opt/app-root/src/Dockerfile’ -> ‘/tmp/build/inputs/app-root/src/Dockerfile’ 3.sourcepath end up with /*， build got error I0416 06:55:59.923926 1 source.go:154] Extracting path /opt/app-root/* from image image-registry.openshift-image-registry.svc:5000/default/ruby-hello-world@sha256:0002146e78f0c201d25a598ddb959c9afe2c173c9eca99cc8d5d14f9de91c588 to . I0416 06:55:59.925838 1 source.go:439] copying image content: cp: cannot stat ‘/var/lib/containers/storage/overlay/0887802391d2a5c46420b787aa9e7fba55410ffea4eb90f3210877c884031726/merged/opt/app-root/*’: No such file or directory F0416 06:56:02.409790 1 helpers.go:119] error: error copying source path /opt/app-root/* to .: exit status 1 So, IMO, I should assignee back, since southpath endup with ///. and /* behavior incorrect.
PR with fix: https://github.com/openshift/builder/pull/59 Note that filepaths with "/*" are not supported - per doc we only support filenames, directories, or the "/." syntax. This issue was prematurely set to MODIFIED - is ready to test now.
Senario 1 : Sourpath End up with /. ,the directory itself isn't be created at the destination. This behaves correct. $oc describe builds multi-slashs-2 -n test Strategy: Docker Dockerfile Path: src/Dockerfile Image Source: copies /opt/app-root///. from image-registry.openshift-image-registry.svc:5000/test/ruby-hello-world@sha256:9d861be820ae20baf5eaae0bcb0d0c1e29f77bdac53ecd64a9aed87e20604723 to . $oc build-logs multi-slashs-2 -n test | grep Dockerfile ‘/var/lib/containers/storage/overlay/8fe22b6656098414336ebe1e8426a76bee1d5b74a3d128e1e705aa3dae65c7b3/merged/opt/app-root/./src/Dockerfile’ -> ‘/tmp/build/inputs/./src/Dockerfile’ Senario 2： Sourcepath end up with ///., '//' will be ingored, be treat as /. . This behaves correct. $ oc describe builds multi-slashs-1 -n test Strategy: Docker Dockerfile Path: src/Dockerfile Image Source: copies /opt/app-root/. from image-registry.openshift-image-registry.svc:5000/test/ruby-hello-world@sha256:9d861be820ae20baf5eaae0bcb0d0c1e29f77bdac53ecd64a9aed87e20604723 to . $ oc build-logs multi-slashs-1 -n test | grep Dockerfile ‘/var/lib/containers/storage/overlay/6070ee9d58f1d4b4e678106f44aea268bb87a9218710e0f229bc7e1009075ae9/merged/opt/app-root/./src/Dockerfile’ -> ‘/tmp/build/inputs/./src/Dockerfile’ Test with payload 4.0.0-0.ci-2019-04-16-221411. Move this bug as verified. Thanks Adma!
Sorry the typo, two scenarios description need switch. Correct them. (In reply to XiuJuan Wang from comment #6) > Senario 1 : Sourpath End up with /. ,the directory itself isn't be created > at the destination. This behaves correct. > > $oc describe builds multi-slashs-2 -n test > Strategy: Docker > Dockerfile Path: src/Dockerfile > Image Source: copies /opt/app-root///. from > image-registry.openshift-image-registry.svc:5000/test/ruby-hello- > world@sha256: > 9d861be820ae20baf5eaae0bcb0d0c1e29f77bdac53ecd64a9aed87e20604723 to . > > $oc build-logs multi-slashs-2 -n test | grep Dockerfile > ‘/var/lib/containers/storage/overlay/ > 8fe22b6656098414336ebe1e8426a76bee1d5b74a3d128e1e705aa3dae65c7b3/merged/opt/ > app-root/./src/Dockerfile’ -> ‘/tmp/build/inputs/./src/Dockerfile’ > The scenario should be "Sourcepath ends up with ///. ,'//' will be ingored, be treat as /. This behaves correct." > Senario 2： Sourcepath end up with ///., '//' will be ingored, be treat as /. > . This behaves correct. > $ oc describe builds multi-slashs-1 -n test > Strategy: Docker > Dockerfile Path: src/Dockerfile > Image Source: copies /opt/app-root/. from > image-registry.openshift-image-registry.svc:5000/test/ruby-hello- > world@sha256: > 9d861be820ae20baf5eaae0bcb0d0c1e29f77bdac53ecd64a9aed87e20604723 to . > > $ oc build-logs multi-slashs-1 -n test | grep Dockerfile > ‘/var/lib/containers/storage/overlay/ > 6070ee9d58f1d4b4e678106f44aea268bb87a9218710e0f229bc7e1009075ae9/merged/opt/ > app-root/./src/Dockerfile’ -> ‘/tmp/build/inputs/./src/Dockerfile’ > The scenario should be "Sourcepath ends up with /. ,the directory itself isn't be created at the destination. This behaves correct." > > Test with payload 4.0.0-0.ci-2019-04-16-221411. > Move this bug as verified. > > Thanks Adma!
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758