Bug 1698152 - image extraction path behavior has changed
Summary: image extraction path behavior has changed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.1.0
Assignee: Adam Kaplan
QA Contact: wewang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-09 17:26 UTC by Ben Parees
Modified: 2019-06-04 10:47 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-04 10:47:18 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 None None None 2019-06-04 10:47:25 UTC
Red Hat Bugzilla 1694859 None VERIFIED image source extraction w/ symlink fails 2019-04-29 19:24:58 UTC

Internal Links: 1694859

Description Ben Parees 2019-04-09 17:26:32 UTC
Description of problem:
Given a build that extracts content from an image using:

  source:
    images:
    - as: null
      from:
        kind: ImageStreamTag
        name: pipeline:src
      paths:
      - destinationDir: .
        sourcePath: /go/src/github.com/openshift/origin///.
    type: Image


and expects to reference a file named images/hypershift/Dockerfile.rhel that resides within /go/src/github.com/openshift/origin///.

  strategy:
    dockerStrategy:
      env:
      - name: "BUILD_LOGLEVEL"
        value: "5"
      dockerfilePath: images/hypershift/Dockerfile.rhel


The build should work (it worked on 3.11).

In 4.x the file gets injected at the wrong path:

'/var/lib/containers/storage/overlay/571af4abe3c5c63492db55bab0630a99be4f790724e5b4f88561d0b45a0ee169/merged/go/src/github.com/openshift/origin/images/hypershift/Dockerfile.rhel’ -> ‘/tmp/build/inputs/origin/images/hypershift/Dockerfile.rhel’

which ultimately fails at the build step:
F0409 13:02:22.707506       1 helpers.go:119] error: build error: open /tmp/build/inputs/images/hypershift/Dockerfile.rhel: no such file or directory

because the file actually exists at:
F0409 13:02:22.707506       1 helpers.go:119] error: build error: open /tmp/build/inputs/origin/images/hypershift/Dockerfile.rhel: no such file or directory


Which means something has changed from 3.x to 4.x in terms of whether the "origin" dir is included as part of the input file.


How reproducible:
always


Additional Info:

our docs say "If the source path ends in /. then the content of the directory will be copied, but the directory itself will not be created at the destination."  (https://docs.okd.io/latest/dev_guide/builds/build_inputs.html#image-source)

I suspect the multiple slashes are throwing off the behavior.  If it were just "sourcePath: /go/src/github.com/openshift/origin/." i think we'd all agree that "origin" should not be included based on that doc.  But with "sourcePath: /go/src/github.com/openshift/origin///." it's less clear what is "correct" behavior.  But again, we need to re-align this with what 3.x did and 3.x appears to have treated "//" as the dir to be ignored, not "origin", in the case of a trailing "."

Comment 1 Ben Parees 2019-04-09 17:49:01 UTC
> because the file actually exists at:
F0409 13:02:22.707506       1 helpers.go:119] error: build error: open /tmp/build/inputs/origin/images/hypershift/Dockerfile.rhel: no such file or directory

that shouldn't have been pasted as an error.  meant to just say the file actually exists at /tmp/build/inputs/origin/images/hypershift/Dockerfile.rhel 

Changing the build reference to:
dockerfilePath: origin/images/hypershift/Dockerfile.rhel

makes the build succeed.

Comment 2 Adam Kaplan 2019-04-11 12:49:25 UTC
Linking to 1694859 - based on my analysis the symlink issue and this bug are one and the same.

The 4.1 copy behavior changed because we use raw Linux cp commands to extract source. Bug is that part of the code used filepath.Join [1], which drops trailing "/.". Adding logic to detect this and pass a glob to cp ("/*") should fix the issue.

[1] https://golang.org/pkg/path/filepath/#Join

Comment 4 XiuJuan Wang 2019-04-16 08:05:28 UTC
    source:
      images:
      - as: null
        from:
          kind: ImageStreamTag
          name: ruby-hello-world:latest
          namespace: xiuwang
        paths:
        - destinationDir: .
          sourcePath: /opt/app-root/.
      type: Image
    strategy:
      dockerStrategy:
        dockerfilePath: src/Dockerfile
        env:
        - name: BUILD_LOGLEVEL
          value: "5"
      type: Docker

with 4.0.0-0.ci-2019-04-14-035215 payload
1. sourcepath end up with /. app-root dir has been copied to des dir.

The build log:
‘/var/lib/containers/storage/overlay/e4235489f2a0faf66a11b023bd03a236bd1ae49c63f40ef6a6e2a4f00ff34775/merged/opt/app-root/src/Dockerfile’ -> ‘/tmp/build/inputs/app-root/src/Dockerfile’
build failed:
F0416 06:57:57.377253       1 helpers.go:119] error: open /tmp/build/inputs/src/Dockerfile: no such file or directory


2.sourcepath end up with ///. , app-root has been copied to des dir.

‘/var/lib/containers/storage/overlay/78fe876c3ac02fdb65431b0c2d3cd8ddc22fa522b45033d06be0752888a449ac/merged/opt/app-root/src/Dockerfile’ -> ‘/tmp/build/inputs/app-root/src/Dockerfile’


Test with 4.0.0-0.ci-2019-04-15-230244 payload
1.sourcepath end up with /.  ,app-root doesn't copied to des dir. It's correct hehavior as doc https://docs.okd.io/latest/dev_guide/builds/build_inputs.html#image-source
‘/var/lib/containers/storage/overlay/eee8cfac84977135e5fb97df83908876ceed9f351ba33d52f6f98836ef0e82a5/merged/opt/app-root/./src/Dockerfile’ -> ‘/tmp/build/inputs/./src/Dockerfile’

2.sourcepath end up with ///. app-root has been copied to des dir.
‘/var/lib/containers/storage/overlay/a1cea26cc9cbe869313b0bd9ea5db966732e45ddd35cdffbf266c4f59333509e/merged/opt/app-root/src/Dockerfile’ -> ‘/tmp/build/inputs/app-root/src/Dockerfile’
3.sourcepath end up with /*, build got error 
I0416 06:55:59.923926       1 source.go:154] Extracting path /opt/app-root/* from image image-registry.openshift-image-registry.svc:5000/default/ruby-hello-world@sha256:0002146e78f0c201d25a598ddb959c9afe2c173c9eca99cc8d5d14f9de91c588 to .
I0416 06:55:59.925838       1 source.go:439] copying image content: cp: cannot stat ‘/var/lib/containers/storage/overlay/0887802391d2a5c46420b787aa9e7fba55410ffea4eb90f3210877c884031726/merged/opt/app-root/*’: No such file or directory
F0416 06:56:02.409790       1 helpers.go:119] error: error copying source path /opt/app-root/* to .: exit status 1


So, IMO, I should assignee back, since southpath endup with ///. and  /* behavior incorrect.

Comment 5 Adam Kaplan 2019-04-16 14:04:03 UTC
PR with fix: https://github.com/openshift/builder/pull/59

Note that filepaths with "/*" are not supported - per doc we only support filenames, directories, or the "/." syntax.

This issue was prematurely set to MODIFIED - is ready to test now.

Comment 6 XiuJuan Wang 2019-04-17 05:31:05 UTC
Senario 1 : Sourpath End up with /. ,the directory itself isn't be created at the destination. This behaves correct.

$oc describe  builds multi-slashs-2  -n test
Strategy:		Docker
Dockerfile Path:	src/Dockerfile
Image Source:		copies /opt/app-root///. from image-registry.openshift-image-registry.svc:5000/test/ruby-hello-world@sha256:9d861be820ae20baf5eaae0bcb0d0c1e29f77bdac53ecd64a9aed87e20604723 to .

$oc build-logs  multi-slashs-2  -n test  | grep  Dockerfile
‘/var/lib/containers/storage/overlay/8fe22b6656098414336ebe1e8426a76bee1d5b74a3d128e1e705aa3dae65c7b3/merged/opt/app-root/./src/Dockerfile’ -> ‘/tmp/build/inputs/./src/Dockerfile’

Senario 2: Sourcepath end up with ///., '//' will be ingored, be treat as /. . This behaves correct.
$ oc describe  builds multi-slashs-1  -n test
Strategy:		Docker
Dockerfile Path:	src/Dockerfile
Image Source:		copies /opt/app-root/. from image-registry.openshift-image-registry.svc:5000/test/ruby-hello-world@sha256:9d861be820ae20baf5eaae0bcb0d0c1e29f77bdac53ecd64a9aed87e20604723 to .

$  oc build-logs  multi-slashs-1  -n test  | grep  Dockerfile
‘/var/lib/containers/storage/overlay/6070ee9d58f1d4b4e678106f44aea268bb87a9218710e0f229bc7e1009075ae9/merged/opt/app-root/./src/Dockerfile’ -> ‘/tmp/build/inputs/./src/Dockerfile’


Test with payload 4.0.0-0.ci-2019-04-16-221411.
Move this bug as verified.

Thanks Adma!

Comment 7 XiuJuan Wang 2019-04-17 05:36:45 UTC
Sorry the typo, two scenarios description need switch. Correct them.

(In reply to XiuJuan Wang from comment #6)
> Senario 1 : Sourpath End up with /. ,the directory itself isn't be created
> at the destination. This behaves correct.
> 
> $oc describe  builds multi-slashs-2  -n test
> Strategy:		Docker
> Dockerfile Path:	src/Dockerfile
> Image Source:		copies /opt/app-root///. from
> image-registry.openshift-image-registry.svc:5000/test/ruby-hello-
> world@sha256:
> 9d861be820ae20baf5eaae0bcb0d0c1e29f77bdac53ecd64a9aed87e20604723 to .
> 
> $oc build-logs  multi-slashs-2  -n test  | grep  Dockerfile
> ‘/var/lib/containers/storage/overlay/
> 8fe22b6656098414336ebe1e8426a76bee1d5b74a3d128e1e705aa3dae65c7b3/merged/opt/
> app-root/./src/Dockerfile’ -> ‘/tmp/build/inputs/./src/Dockerfile’
> 
The scenario should be "Sourcepath ends up with ///. ,'//' will be ingored, be treat as /. This behaves correct."

> Senario 2: Sourcepath end up with ///., '//' will be ingored, be treat as /.
> . This behaves correct.
> $ oc describe  builds multi-slashs-1  -n test
> Strategy:		Docker
> Dockerfile Path:	src/Dockerfile
> Image Source:		copies /opt/app-root/. from
> image-registry.openshift-image-registry.svc:5000/test/ruby-hello-
> world@sha256:
> 9d861be820ae20baf5eaae0bcb0d0c1e29f77bdac53ecd64a9aed87e20604723 to .
> 
> $  oc build-logs  multi-slashs-1  -n test  | grep  Dockerfile
> ‘/var/lib/containers/storage/overlay/
> 6070ee9d58f1d4b4e678106f44aea268bb87a9218710e0f229bc7e1009075ae9/merged/opt/
> app-root/./src/Dockerfile’ -> ‘/tmp/build/inputs/./src/Dockerfile’
> 
The scenario should be "Sourcepath ends up with /. ,the directory itself isn't be created at the destination. This behaves correct."

> 
> Test with payload 4.0.0-0.ci-2019-04-16-221411.
> Move this bug as verified.
> 
> Thanks Adma!

Comment 9 errata-xmlrpc 2019-06-04 10:47:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.