Bug 1707941 - image build doesn't handle COPY correctly in some cases
Summary: image build doesn't handle COPY correctly in some cases
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.2.0
Assignee: Nalin Dahyabhai
QA Contact: weiwei jiang
URL:
Whiteboard:
Depends On:
Blocks: 1710008
TreeView+ depends on / blocked
 
Reported: 2019-05-08 17:59 UTC by Ben Parees
Modified: 2019-10-16 06:28 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Logic which handled COPY and ADD instructions in cases where the build context directory included a .dockerignore file would not correctly handle some symbolic links and subdirectories. Consequence: An affected build would fail while attempting to process a COPY or ADD instruction which triggered the bug. Fix: Logic which handles this case has been extended. Result: These errors should no longer occur.
Clone Of:
: 1710008 (view as bug list)
Environment:
Last Closed: 2019-10-16 06:28:42 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:2922 0 None None None 2019-10-16 06:28:57 UTC

Description Ben Parees 2019-05-08 17:59:44 UTC
Description of problem:
in a dockerfile,
COPY . .

is failing in some cases.


How reproducible:
always

Steps to Reproduce:
1. git clone git:operator-framework/helm.git
2. buildah bud .
3. see failure:
error building at STEP "COPY . .": error copying "/home/bparees/git/gocode/src/github.com/openshift/helm/pkg/chartutil/testdata/joonix/charts/frobnitz" to "/home/bparees/.local/share/containers/storage/vfs/dir/a95fa17f13262c63706f22e35a8c0186a522bff0df57c97028c88867df39bd02/go/src/k8s.io/helm": Can't copy a directory

4. docker build .
5. see success


Actual results:
buildah bud fails, docker build succeeds. 

Expected results:
both should succeed

Additional info:

This is a blocker for OCP4.1 because ocp image builds are experiencing the same failure.  buildah is just an easy reproducer.

There are also similar looking cases of COPY that seem to work fine:

git clone git:openshift/elasticsearch-operator.git
buildah bud .

succeeds despite doing pretty much the same COPY operation: https://github.com/openshift/elasticsearch-operator/blob/master/Dockerfile#L3

Comment 1 Ben Parees 2019-05-08 18:46:40 UTC
full list of github repos i'm seeing this issue with:
operator-framework/helm
openshift/multus-admission-controller
openshift/node_exporter
openshift/grafana


I am also seeing a slightly different issue on these repos, but the overall effect is the same in that docker builds them fine, buildah fails:

operator-framework/operator-registry fails with:
STEP 13: RUN mkdir /registry
STEP 14: WORKDIR /registry
STEP 15: COPY --from=builder /go/src/github.com/operator-framework/operator-registry/bin/initializer /bin/initializer
STEP 16: COPY --from=builder /go/src/github.com/operator-framework/operator-registry/bin/registry-server /bin/registry-server
STEP 17: COPY --from=builder /go/src/github.com/operator-framework/operator-registry/bin/configmap-server /bin/configmap-server
STEP 18: COPY --from=builder /go/src/github.com/operator-framework/operator-registry/bin/appregistry-server /bin/appregistry-server
STEP 19: COPY --from=builder /go/bin/grpc_health_probe /bin/grpc_health_probe
STEP 20: RUN chgrp -R 0 /registry &&     chgrp -R 0 /dev &&     chmod -R g+rwx /registry &&     chmod -R g+rwx /dev
chgrp: changing group of '/dev/urandom': Permission denied
chgrp: changing group of '/dev/zero': Permission denied
chgrp: changing group of '/dev/tty': Permission denied
chgrp: changing group of '/dev/full': Permission denied
chgrp: changing group of '/dev/random': Permission denied
chgrp: changing group of '/dev/null': Permission denied
error building at STEP "RUN chgrp -R 0 /registry &&     chgrp -R 0 /dev &&     chmod -R g+rwx /registry &&     chmod -R g+rwx /dev": error while running runtime: exit status 1
ERRO[0260] exit status 1                                



openshift/cluster-api-provider-azure fails with:
STEP 1: FROM registry.svc.ci.openshift.org/openshift/release:golang-1.10 AS builder
STEP 2: WORKDIR /go/src/sigs.k8s.io/cluster-api-provider-azure
STEP 3: COPY pkg/    pkg/
STEP 4: COPY cmd/    cmd/
STEP 5: COPY vendor/ vendor/
error building at STEP "COPY vendor/ vendor/": error copying "/home/bparees/git/gocode/src/github.com/openshift/cluster-api-provider-azure/vendor/k8s.io/kubernetes/.bazelrc" to "/home/bparees/.local/share/containers/storage/vfs/dir/b2e6a7668c62fa0e1d9ac68cb38bf1bf367131424c88cfaef259cf7861a8b264/go/src/sigs.k8s.io/cluster-api-provider-azure/vendor": stat /home/bparees/git/gocode/src/github.com/openshift/cluster-api-provider-azure/vendor/k8s.io/kubernetes/.bazelrc: no such file or directory
ERRO[0115] exit status 1

Comment 2 Nalin Dahyabhai 2019-05-08 19:43:06 UTC
It looks like the handling of .dockerignore files has difficulty with symbolic links (and probably other non-directory, non-regular items).

Comment 4 Ben Parees 2019-05-08 22:45:17 UTC
Just to clarify the impact of this bug, based on my understanding from Nalin:

if you have a image build context directory containing:

1) a .dockerignore
2) a symlink (or other "unusual" file type)

and then you do a
COPY . /somedir

in your dockerfile.

Then your build will fail.  It does not matter if the .dockerignore references the symlink or not.

For the RUN issue, we should split it out into a separate (4.1.z+4.2.0) targeted bug as it's an unrelated issue and less severe in terms of likely users impacted.

Comment 5 Nalin Dahyabhai 2019-05-09 15:30:30 UTC
https://github.com/containers/buildah/pull/1583 should fix the issues with symbolic links.

Comment 6 Nalin Dahyabhai 2019-05-13 14:15:17 UTC
https://github.com/openshift/builder/pull/72 should merge the fix into the builder.

Comment 8 weiwei jiang 2019-06-28 11:25:09 UTC
Checked and this issue has been fixed.

# oc debug pods/custom-builder-image-5-build --keep-init-containers=false
Starting pod/custom-builder-image-5-build-debug, command was: openshift-docker-build --loglevel=0
If you don't see a command prompt, try pressing enter.
sh-4.2# openshift-docker-build version 
openshift-docker-build v4.2.0-201906251832+f815795-dirty

# oc new-build --binary --strategy=docker --name custom-builder-image
# git clone --depth=1 https://github.com/operator-framework/helm.git && cd helm
# oc start-build custom-builder-image --from-dir . -F
Uploading directory "." as binary input for the build ...
........
Uploading finished
build.build.openshift.io/custom-builder-image-5 started
Receiving source from STDIN as archive ...
Caching blobs under "/var/cache/blobs".

Pulling image centos:7 ...
Getting image source signatures
Copying blob sha256:8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df
Copying config sha256:9f38484d220fa527b1fb19747638497179500a1bed8bf0498eb788229229e6e1
Writing manifest to image destination
Storing signatures

Pulling image openshift/origin-release:golang-1.10 ...
Getting image source signatures
Copying blob sha256:8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df
Copying blob sha256:ff2175d8c0494533c9f3103d69268a6b96f2ef40d2cc1152f3731b33a4af5739
Copying config sha256:53106b8a7e26448da5bfdebccdc1247261e99830ff0a8b373059b77567b4bd89
Writing manifest to image destination
Storing signatures
STEP 1: FROM openshift/origin-release:golang-1.10 AS build
STEP 2: RUN yum install --setopt=skip_missing_names_on_install=False -y         hg git make     && yum clean all     && rm -rf /var/cache/yum
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
 * base: ty1.mirror.newmediaexpress.com
 * epel: d2lzkl7pfhq30w.cloudfront.net
 * extras: ty1.mirror.newmediaexpress.com
 * updates: ty1.mirror.newmediaexpress.com
Package mercurial-2.6.2-8.el7_4.x86_64 already installed and latest version
Package git-1.8.3.1-20.el7.x86_64 already installed and latest version
Package 1:make-3.82-23.el7.x86_64 already installed and latest version
Nothing to do
Loaded plugins: fastestmirror, ovl
Cleaning repos: base cbs-paas7-openshift-multiarch-el7-build epel extras updates
Cleaning up list of fastest mirrors
time="2019-06-28T10:19:32Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah335658036/mnt/rootfs\": error checking if \"/tmp/buildah335658036/mnt/rootfs/sys/fs/cgroup/memory\" is mounted: no such file or directory"
time="2019-06-28T10:19:32Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah335658036/mnt/rootfs\": error checking if \"/tmp/buildah335658036/mnt/rootfs/sys/fs/cgroup/memory\" is mounted: no such file or directory"
--> fa9f4c0184934e3ce6af5a4fc562183429702c58bdf4eb5f92a23aa21920ddfe
STEP 3: ENV GOPATH=/go
--> 5037a5378e056ba6cb348bd1380caa0998753116cc1ebed3b5e0c9064111d270
STEP 4: RUN mkdir -p /go/src/k8s.io/helm
time="2019-06-28T10:19:36Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah114799931/mnt/rootfs\": error checking if \"/tmp/buildah114799931/mnt/rootfs/sys/fs/cgroup/memory\" is mounted: no such file or directory"
time="2019-06-28T10:19:36Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah114799931/mnt/rootfs\": error checking if \"/tmp/buildah114799931/mnt/rootfs/sys/fs/cgroup/memory\" is mounted: no such file or directory"
--> 38766cb78011962fd067840e4ed000e9d8091551091f71d3a76197964dacb481
STEP 5: WORKDIR /go/src/k8s.io/helm
--> 7cd9e9eef2581a53be972934336e836a11e558691540b6dca60de0cf238510ca
STEP 6: COPY . .

--> 839567bd6e91e4aa9b0519854b29c5c2453480b65d465e96f5ab16a6db6d8b2e
STEP 7: ENV VERSION=""
--> 3d11f29153e5368787af5779f325996522d289e49f3f68d130b8ad4b8e07989a
STEP 8: RUN make build
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
GOBIN=/go/src/k8s.io/helm/bin go install  -tags '' -ldflags ' -X k8s.io/helm/pkg/version.GitCommit= -X k8s.io/helm/pkg/version.GitTreeState=clean' k8s.io/helm/cmd/...
time="2019-06-28T11:03:56Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah623872892/mnt/rootfs\": error checking if \"/tmp/buildah623872892/mnt/rootfs/sys/fs/cgroup/memory\" is mounted: no such file or directory"
time="2019-06-28T11:03:56Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah623872892/mnt/rootfs\": error checking if \"/tmp/buildah623872892/mnt/rootfs/sys/fs/cgroup/memory\" is mounted: no such file or directory"
--> f690a83a11ad6b63f1d7683f56e0753434e48c24a753d512dea2ee578daabbff
STEP 9: RUN make docker-binary
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o ./rootfs/tiller -a -installsuffix cgo -tags '' -ldflags ' -X k8s.io/helm/pkg/version.GitCommit= -X k8s.io/helm/pkg/version.GitTreeState=clean' k8s.io/helm/cmd/tiller
time="2019-06-28T11:06:42Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah591548176/mnt/rootfs\": error checking if \"/tmp/buildah591548176/mnt/rootfs/sys/fs/cgroup/memory\" is mounted: no such file or directory"
time="2019-06-28T11:06:42Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah591548176/mnt/rootfs\": error checking if \"/tmp/buildah591548176/mnt/rootfs/sys/fs/cgroup/memory\" is mounted: no such file or directory"
--> 35a3cc6f624988732dec71c3eeea4c78ceca4456b05a4535df73bbc006991706
STEP 10: FROM centos:7
STEP 11: COPY --from=build /go/src/k8s.io/helm/rootfs/tiller /usr/local/bin
--> d14afc3bdda3c2b599c21ba532325775bf5477bd5f600cbe259152000af8f323
STEP 12: COPY --from=build /go/src/k8s.io/helm/bin/helm /usr/local/bin
--> 83323f4dac0080817ae968d4dfb004707e43f4c8234d2aa1b705a24970d1388b
STEP 13: RUN ln -s /usr/local/bin/tiller /tiller
time="2019-06-28T11:06:59Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah543403741/mnt/rootfs\": error checking if \"/tmp/buildah543403741/mnt/rootfs/sys/fs/cgroup/memory\" is mounted: no such file or directory"
time="2019-06-28T11:06:59Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah543403741/mnt/rootfs\": error checking if \"/tmp/buildah543403741/mnt/rootfs/sys/fs/cgroup/memory\" is mounted: no such file or directory"
--> 8e5a834d83db882dbc62f5a844cf07c1ea43ef0decd2f5154038d42da30a8ce3
STEP 14: LABEL io.k8s.display-name="OpenShift metering-helm" io.k8s.description="This is a base image used by operator-metering to assist in managing the lifecycle of the Openshift Metering components." io.openshift.tags="openshift" maintainer="Chance Zibolski <czibolsk>"
--> c1a97cb298f7d1b24b80d5d88bd4a4cb321bc6c44653a5469a8c080568993346
STEP 15: ENV "OPENSHIFT_BUILD_NAME"="custom-builder-image-5" "OPENSHIFT_BUILD_NAMESPACE"="default"
--> abd7e3bb5df416f81115ed6c5c35860218131f288939fad78835fc95e3b8efea
STEP 16: LABEL "io.openshift.build.name"="custom-builder-image-5" "io.openshift.build.namespace"="default"
STEP 17: COMMIT temp.builder.openshift.io/default/custom-builder-image-5:dd58f34c
--> a4ed398f48fc917456f9b84f875eca1c2bd0b5aaa91b1554ee9f781f94a955dd

Pushing image image-registry.openshift-image-registry.svc:5000/default/custom-builder-image:latest ...
Getting image source signatures
Copying blob sha256:0fab0bbe45247a2523db4aaaf82bb9bfd617b04beff8cedbb2beb6dd7a1bf829
Copying blob sha256:c9d4d91dcf32accbb32eafd706023d452b88a922bc9326624c15c5004df337ea
Copying blob sha256:4ebcdc256c0129fa885807cb66f869f853941daabbdcd6bd6180dc36fe9a6ba0
Copying blob sha256:8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df
Copying config sha256:a4ed398f48fc917456f9b84f875eca1c2bd0b5aaa91b1554ee9f781f94a955dd
Writing manifest to image destination
Storing signatures
Successfully pushed image-registry.openshift-image-registry.svc:5000/default/custom-builder-image:latest@sha256:141db0dd2de6fee4a6b93e5fd4fe86bcc2cf88718948db723107ac5539291cc3
Push successful

Comment 9 Radomir Ludva 2019-07-17 11:27:26 UTC
Exactly the same issue but not with COPY statement in Docker file. So I am not sure if to create now Bugzilla or if a fix for this BZ will fix also this exactly the same issue in OCP 4.1:

Dockerfile
----------
FROM registry.redhat.io/ubi7/ubi

EXPOSE 4440

# Configure env variables
ENV RDECK_BASE '/var/lib/rundeck'
ENV RDECK_CONFIG '/etc/rundeck'

# Where to store the DB and project definitions and logs VOLUME ["/var/rundeck", "/var/lib/rundeck/logs"]

# Install rundeck
RUN yum install -y http://repo.rundeck.org/latest.rpm && \
    yum install -y java-1.8.0-openjdk gettext rundeck rundeck-config git \
      python27-python-virtualenv python27-python-pip rundeck-cli bc openssh-clients && \
    yum clean all && \
    scl enable python27 'pip install kubernetes==9.0.0'


# Create rundeck folders and give appropriate permissions RUN mkdir -p $RDECK_BASE && chmod -R a+rw $RDECK_BASE && chmod -R a+rw /var/log/rundeck && \
RUN chmod -R a+rw /tmp/rundeck && mkdir -p /rundeck-config && chmod -R a+rw $RDECK_CONFIG && \
    chmod -R a+rwx /rundeck-config

Log error message
-----------------
time="2019-07-16T14:04:38Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah922100776/mnt/rootfs\": error checking if \"/tmp/buildah922100776/mnt/rootfs/sys/fs/cgroup/blkio\" is mounted: no such file or directory"
time="2019-07-16T14:04:38Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah922100776/mnt/rootfs\": error checking if \"/tmp/buildah922100776/mnt/rootfs/sys/fs/cgroup/blkio\" is mounted: no such file or directory"

Comment 10 Ben Parees 2019-07-17 13:32:41 UTC
Best thing to do is open a new BZ so the team can be sure to evaluate it properly.  They'll close it if it's already resolved.

Also if we have more scenarios that are breaking under 4.1, that may increase the pressure to backport the fix to 4.1, even if it is the same underlying fix.

Comment 11 errata-xmlrpc 2019-10-16 06:28:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922


Note You need to log in before you can comment on or make changes to this bug.