Description of problem: in a dockerfile, COPY . . is failing in some cases. How reproducible: always Steps to Reproduce: 1. git clone git:operator-framework/helm.git 2. buildah bud . 3. see failure: error building at STEP "COPY . .": error copying "/home/bparees/git/gocode/src/github.com/openshift/helm/pkg/chartutil/testdata/joonix/charts/frobnitz" to "/home/bparees/.local/share/containers/storage/vfs/dir/a95fa17f13262c63706f22e35a8c0186a522bff0df57c97028c88867df39bd02/go/src/k8s.io/helm": Can't copy a directory 4. docker build . 5. see success Actual results: buildah bud fails, docker build succeeds. Expected results: both should succeed Additional info: This is a blocker for OCP4.1 because ocp image builds are experiencing the same failure. buildah is just an easy reproducer. There are also similar looking cases of COPY that seem to work fine: git clone git:openshift/elasticsearch-operator.git buildah bud . succeeds despite doing pretty much the same COPY operation: https://github.com/openshift/elasticsearch-operator/blob/master/Dockerfile#L3
full list of github repos i'm seeing this issue with: operator-framework/helm openshift/multus-admission-controller openshift/node_exporter openshift/grafana I am also seeing a slightly different issue on these repos, but the overall effect is the same in that docker builds them fine, buildah fails: operator-framework/operator-registry fails with: STEP 13: RUN mkdir /registry STEP 14: WORKDIR /registry STEP 15: COPY --from=builder /go/src/github.com/operator-framework/operator-registry/bin/initializer /bin/initializer STEP 16: COPY --from=builder /go/src/github.com/operator-framework/operator-registry/bin/registry-server /bin/registry-server STEP 17: COPY --from=builder /go/src/github.com/operator-framework/operator-registry/bin/configmap-server /bin/configmap-server STEP 18: COPY --from=builder /go/src/github.com/operator-framework/operator-registry/bin/appregistry-server /bin/appregistry-server STEP 19: COPY --from=builder /go/bin/grpc_health_probe /bin/grpc_health_probe STEP 20: RUN chgrp -R 0 /registry && chgrp -R 0 /dev && chmod -R g+rwx /registry && chmod -R g+rwx /dev chgrp: changing group of '/dev/urandom': Permission denied chgrp: changing group of '/dev/zero': Permission denied chgrp: changing group of '/dev/tty': Permission denied chgrp: changing group of '/dev/full': Permission denied chgrp: changing group of '/dev/random': Permission denied chgrp: changing group of '/dev/null': Permission denied error building at STEP "RUN chgrp -R 0 /registry && chgrp -R 0 /dev && chmod -R g+rwx /registry && chmod -R g+rwx /dev": error while running runtime: exit status 1 ERRO[0260] exit status 1 openshift/cluster-api-provider-azure fails with: STEP 1: FROM registry.svc.ci.openshift.org/openshift/release:golang-1.10 AS builder STEP 2: WORKDIR /go/src/sigs.k8s.io/cluster-api-provider-azure STEP 3: COPY pkg/ pkg/ STEP 4: COPY cmd/ cmd/ STEP 5: COPY vendor/ vendor/ error building at STEP "COPY vendor/ vendor/": error copying "/home/bparees/git/gocode/src/github.com/openshift/cluster-api-provider-azure/vendor/k8s.io/kubernetes/.bazelrc" to "/home/bparees/.local/share/containers/storage/vfs/dir/b2e6a7668c62fa0e1d9ac68cb38bf1bf367131424c88cfaef259cf7861a8b264/go/src/sigs.k8s.io/cluster-api-provider-azure/vendor": stat /home/bparees/git/gocode/src/github.com/openshift/cluster-api-provider-azure/vendor/k8s.io/kubernetes/.bazelrc: no such file or directory ERRO[0115] exit status 1
It looks like the handling of .dockerignore files has difficulty with symbolic links (and probably other non-directory, non-regular items).
Just to clarify the impact of this bug, based on my understanding from Nalin: if you have a image build context directory containing: 1) a .dockerignore 2) a symlink (or other "unusual" file type) and then you do a COPY . /somedir in your dockerfile. Then your build will fail. It does not matter if the .dockerignore references the symlink or not. For the RUN issue, we should split it out into a separate (4.1.z+4.2.0) targeted bug as it's an unrelated issue and less severe in terms of likely users impacted.
https://github.com/containers/buildah/pull/1583 should fix the issues with symbolic links.
https://github.com/openshift/builder/pull/72 should merge the fix into the builder.
Checked and this issue has been fixed. # oc debug pods/custom-builder-image-5-build --keep-init-containers=false Starting pod/custom-builder-image-5-build-debug, command was: openshift-docker-build --loglevel=0 If you don't see a command prompt, try pressing enter. sh-4.2# openshift-docker-build version openshift-docker-build v4.2.0-201906251832+f815795-dirty # oc new-build --binary --strategy=docker --name custom-builder-image # git clone --depth=1 https://github.com/operator-framework/helm.git && cd helm # oc start-build custom-builder-image --from-dir . -F Uploading directory "." as binary input for the build ... ........ Uploading finished build.build.openshift.io/custom-builder-image-5 started Receiving source from STDIN as archive ... Caching blobs under "/var/cache/blobs". Pulling image centos:7 ... Getting image source signatures Copying blob sha256:8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df Copying config sha256:9f38484d220fa527b1fb19747638497179500a1bed8bf0498eb788229229e6e1 Writing manifest to image destination Storing signatures Pulling image openshift/origin-release:golang-1.10 ... Getting image source signatures Copying blob sha256:8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df Copying blob sha256:ff2175d8c0494533c9f3103d69268a6b96f2ef40d2cc1152f3731b33a4af5739 Copying config sha256:53106b8a7e26448da5bfdebccdc1247261e99830ff0a8b373059b77567b4bd89 Writing manifest to image destination Storing signatures STEP 1: FROM openshift/origin-release:golang-1.10 AS build STEP 2: RUN yum install --setopt=skip_missing_names_on_install=False -y hg git make && yum clean all && rm -rf /var/cache/yum Loaded plugins: fastestmirror, ovl Determining fastest mirrors * base: ty1.mirror.newmediaexpress.com * epel: d2lzkl7pfhq30w.cloudfront.net * extras: ty1.mirror.newmediaexpress.com * updates: ty1.mirror.newmediaexpress.com Package mercurial-2.6.2-8.el7_4.x86_64 already installed and latest version Package git-1.8.3.1-20.el7.x86_64 already installed and latest version Package 1:make-3.82-23.el7.x86_64 already installed and latest version Nothing to do Loaded plugins: fastestmirror, ovl Cleaning repos: base cbs-paas7-openshift-multiarch-el7-build epel extras updates Cleaning up list of fastest mirrors time="2019-06-28T10:19:32Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah335658036/mnt/rootfs\": error checking if \"/tmp/buildah335658036/mnt/rootfs/sys/fs/cgroup/memory\" is mounted: no such file or directory" time="2019-06-28T10:19:32Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah335658036/mnt/rootfs\": error checking if \"/tmp/buildah335658036/mnt/rootfs/sys/fs/cgroup/memory\" is mounted: no such file or directory" --> fa9f4c0184934e3ce6af5a4fc562183429702c58bdf4eb5f92a23aa21920ddfe STEP 3: ENV GOPATH=/go --> 5037a5378e056ba6cb348bd1380caa0998753116cc1ebed3b5e0c9064111d270 STEP 4: RUN mkdir -p /go/src/k8s.io/helm time="2019-06-28T10:19:36Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah114799931/mnt/rootfs\": error checking if \"/tmp/buildah114799931/mnt/rootfs/sys/fs/cgroup/memory\" is mounted: no such file or directory" time="2019-06-28T10:19:36Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah114799931/mnt/rootfs\": error checking if \"/tmp/buildah114799931/mnt/rootfs/sys/fs/cgroup/memory\" is mounted: no such file or directory" --> 38766cb78011962fd067840e4ed000e9d8091551091f71d3a76197964dacb481 STEP 5: WORKDIR /go/src/k8s.io/helm --> 7cd9e9eef2581a53be972934336e836a11e558691540b6dca60de0cf238510ca STEP 6: COPY . . --> 839567bd6e91e4aa9b0519854b29c5c2453480b65d465e96f5ab16a6db6d8b2e STEP 7: ENV VERSION="" --> 3d11f29153e5368787af5779f325996522d289e49f3f68d130b8ad4b8e07989a STEP 8: RUN make build fatal: Not a git repository (or any of the parent directories): .git fatal: Not a git repository (or any of the parent directories): .git fatal: Not a git repository (or any of the parent directories): .git fatal: Not a git repository (or any of the parent directories): .git fatal: Not a git repository (or any of the parent directories): .git GOBIN=/go/src/k8s.io/helm/bin go install -tags '' -ldflags ' -X k8s.io/helm/pkg/version.GitCommit= -X k8s.io/helm/pkg/version.GitTreeState=clean' k8s.io/helm/cmd/... time="2019-06-28T11:03:56Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah623872892/mnt/rootfs\": error checking if \"/tmp/buildah623872892/mnt/rootfs/sys/fs/cgroup/memory\" is mounted: no such file or directory" time="2019-06-28T11:03:56Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah623872892/mnt/rootfs\": error checking if \"/tmp/buildah623872892/mnt/rootfs/sys/fs/cgroup/memory\" is mounted: no such file or directory" --> f690a83a11ad6b63f1d7683f56e0753434e48c24a753d512dea2ee578daabbff STEP 9: RUN make docker-binary fatal: Not a git repository (or any of the parent directories): .git fatal: Not a git repository (or any of the parent directories): .git fatal: Not a git repository (or any of the parent directories): .git fatal: Not a git repository (or any of the parent directories): .git fatal: Not a git repository (or any of the parent directories): .git GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o ./rootfs/tiller -a -installsuffix cgo -tags '' -ldflags ' -X k8s.io/helm/pkg/version.GitCommit= -X k8s.io/helm/pkg/version.GitTreeState=clean' k8s.io/helm/cmd/tiller time="2019-06-28T11:06:42Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah591548176/mnt/rootfs\": error checking if \"/tmp/buildah591548176/mnt/rootfs/sys/fs/cgroup/memory\" is mounted: no such file or directory" time="2019-06-28T11:06:42Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah591548176/mnt/rootfs\": error checking if \"/tmp/buildah591548176/mnt/rootfs/sys/fs/cgroup/memory\" is mounted: no such file or directory" --> 35a3cc6f624988732dec71c3eeea4c78ceca4456b05a4535df73bbc006991706 STEP 10: FROM centos:7 STEP 11: COPY --from=build /go/src/k8s.io/helm/rootfs/tiller /usr/local/bin --> d14afc3bdda3c2b599c21ba532325775bf5477bd5f600cbe259152000af8f323 STEP 12: COPY --from=build /go/src/k8s.io/helm/bin/helm /usr/local/bin --> 83323f4dac0080817ae968d4dfb004707e43f4c8234d2aa1b705a24970d1388b STEP 13: RUN ln -s /usr/local/bin/tiller /tiller time="2019-06-28T11:06:59Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah543403741/mnt/rootfs\": error checking if \"/tmp/buildah543403741/mnt/rootfs/sys/fs/cgroup/memory\" is mounted: no such file or directory" time="2019-06-28T11:06:59Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah543403741/mnt/rootfs\": error checking if \"/tmp/buildah543403741/mnt/rootfs/sys/fs/cgroup/memory\" is mounted: no such file or directory" --> 8e5a834d83db882dbc62f5a844cf07c1ea43ef0decd2f5154038d42da30a8ce3 STEP 14: LABEL io.k8s.display-name="OpenShift metering-helm" io.k8s.description="This is a base image used by operator-metering to assist in managing the lifecycle of the Openshift Metering components." io.openshift.tags="openshift" maintainer="Chance Zibolski <czibolsk>" --> c1a97cb298f7d1b24b80d5d88bd4a4cb321bc6c44653a5469a8c080568993346 STEP 15: ENV "OPENSHIFT_BUILD_NAME"="custom-builder-image-5" "OPENSHIFT_BUILD_NAMESPACE"="default" --> abd7e3bb5df416f81115ed6c5c35860218131f288939fad78835fc95e3b8efea STEP 16: LABEL "io.openshift.build.name"="custom-builder-image-5" "io.openshift.build.namespace"="default" STEP 17: COMMIT temp.builder.openshift.io/default/custom-builder-image-5:dd58f34c --> a4ed398f48fc917456f9b84f875eca1c2bd0b5aaa91b1554ee9f781f94a955dd Pushing image image-registry.openshift-image-registry.svc:5000/default/custom-builder-image:latest ... Getting image source signatures Copying blob sha256:0fab0bbe45247a2523db4aaaf82bb9bfd617b04beff8cedbb2beb6dd7a1bf829 Copying blob sha256:c9d4d91dcf32accbb32eafd706023d452b88a922bc9326624c15c5004df337ea Copying blob sha256:4ebcdc256c0129fa885807cb66f869f853941daabbdcd6bd6180dc36fe9a6ba0 Copying blob sha256:8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df Copying config sha256:a4ed398f48fc917456f9b84f875eca1c2bd0b5aaa91b1554ee9f781f94a955dd Writing manifest to image destination Storing signatures Successfully pushed image-registry.openshift-image-registry.svc:5000/default/custom-builder-image:latest@sha256:141db0dd2de6fee4a6b93e5fd4fe86bcc2cf88718948db723107ac5539291cc3 Push successful
Exactly the same issue but not with COPY statement in Docker file. So I am not sure if to create now Bugzilla or if a fix for this BZ will fix also this exactly the same issue in OCP 4.1: Dockerfile ---------- FROM registry.redhat.io/ubi7/ubi EXPOSE 4440 # Configure env variables ENV RDECK_BASE '/var/lib/rundeck' ENV RDECK_CONFIG '/etc/rundeck' # Where to store the DB and project definitions and logs VOLUME ["/var/rundeck", "/var/lib/rundeck/logs"] # Install rundeck RUN yum install -y http://repo.rundeck.org/latest.rpm && \ yum install -y java-1.8.0-openjdk gettext rundeck rundeck-config git \ python27-python-virtualenv python27-python-pip rundeck-cli bc openssh-clients && \ yum clean all && \ scl enable python27 'pip install kubernetes==9.0.0' # Create rundeck folders and give appropriate permissions RUN mkdir -p $RDECK_BASE && chmod -R a+rw $RDECK_BASE && chmod -R a+rw /var/log/rundeck && \ RUN chmod -R a+rw /tmp/rundeck && mkdir -p /rundeck-config && chmod -R a+rw $RDECK_CONFIG && \ chmod -R a+rwx /rundeck-config Log error message ----------------- time="2019-07-16T14:04:38Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah922100776/mnt/rootfs\": error checking if \"/tmp/buildah922100776/mnt/rootfs/sys/fs/cgroup/blkio\" is mounted: no such file or directory" time="2019-07-16T14:04:38Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah922100776/mnt/rootfs\": error checking if \"/tmp/buildah922100776/mnt/rootfs/sys/fs/cgroup/blkio\" is mounted: no such file or directory"
Best thing to do is open a new BZ so the team can be sure to evaluate it properly. They'll close it if it's already resolved. Also if we have more scenarios that are breaking under 4.1, that may increase the pressure to backport the fix to 4.1, even if it is the same underlying fix.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922