Description of problem: When I execute a Source-to-Image build that uses an existing ImageStreamTag as an image source (rather than using an image from Dockerhub, for example), OpenShift fails to commit the s2i container build, throwing the error: "error reading blob from source image." (Very similar to: https://bugzilla.redhat.com/show_bug.cgi?id=1720730) Version-Release number of selected component (if applicable): v4.1.0 How reproducible: Always Steps to Reproduce: 1. Create s2i assemble+run scripts 2. Create Dockerfile that pulls in those s2i scripts (e.g., "COPY s2i/bin /usr/libexec/s2i") 3. Create a BuildConfig template that will be used to generate an image from said Dockerfile. (The image installs yum/dnf/pip packages with root privileges; it does not execute the s2i scripts.) Call the template "source-buildconfig.yaml". 4. oc create -f source-buildconfig.yaml && oc new-app --template=my-dockerstrategy-buildconfig-template 5. oc start-build 6. After image successfully builds and the ImageStream tag has been created, create a BuildConfig template to generate an image using "sourceStrategy." The image generated from step #5 will be used as the BuildConfig's source image, with the s2i scripts already included in said source image. 7. oc create -f s2i-buildconfig.yaml && oc new-app --template=my-sourcestrategy-buildconfig-template Actual results: Caching blobs under "/var/cache/blobs". Getting image source signatures Copying blob sha256:24e41a8e056b3ac8c0418e3e6d7aa0a7bc0812e30827d03b193df3e474f6c03d Copying blob sha256:79d11c1a86c4521558eb9e4b4173811ad15cd2c865eda4eb15a1a2e90af94136 Copying blob sha256:f84885f5ab7924c1c7a0a7d12861c16473ae09f5baedd0b2a8d7fcd9b017b8e5 Copying blob sha256:c8d67acdb2ffaebd638cf55a8fccc63693211060670aa7f0ea1d65b5d2c674dd Copying blob sha256:e1b9b31a5585a2d263b5aa958337bbb1a125fab1d3b14f740e8e6ca93bd3e821 Copying blob sha256:3a674f761814e88c7e8f5a55dfa291bd1c62285f7f3cced2c6e3f6b7e295ab85 Copying blob sha256:c4fe78b1a926e2fe835db146b1bf6ece2c76d431b081e32c77e1a6e7e926069d Copying config sha256:672e01b9107dddfbfedc83b693f461fc915a862f0d63a5b2451dfd74eabc95ee Writing manifest to image destination Storing signatures 672e01b9107dddfbfedc83b693f461fc915a862f0d63a5b2451dfd74eabc95ee Generating dockerfile with builder image image-registry.openshift-image-registry.svc:5000/openshift-image-registry/tensorflow-fftw-base-ubi7-gpu@sha256:93b73c4d061adb4797cf8de5cb607f07162aa1f0ea635c5d102895c798d1da5e STEP 1: FROM image-registry.openshift-image-registry.svc:5000/openshift-image-registry/tensorflow-fftw-base-ubi7-gpu@sha256:93b73c4d061adb4797cf8de5cb607f07162aa1f0ea635c5d102895c798d1da5e STEP 2: LABEL "io.openshift.build.image"="image-registry.openshift-image-registry.svc:5000/openshift-image-registry/tensorflow-fftw-base-ubi7-gpu@sha256:93b73c4d061adb4797cf8de5cb607f07162aa1f0ea635c5d102895c798d1da5e" "io.openshift.build.source-location"="/tmp/build/inputs" STEP 3: ENV OPENSHIFT_BUILD_NAME="tensorflow-fftw-s2i-build-gpu-ubi-aws-1" OPENSHIFT_BUILD_NAMESPACE="openshift-image-registry" RHEL_VERSION="7" BACKEND="fftw" DEVICE="gpu" NUM_DEVICES="2" CC="/usr/bin/gcc" WHICH_SOURCE="s3" PYTHON_EXECUTABLE="/usr/local/bin/python3" NCCL="[redacted]" CUDNN="[redacted]" TENSORRT="[redacted]" AWS_PROFILE="[redacted]" AWS_REGION="[redacted]" AWS_ACCESS_KEY="[redacted]" AWS_SECRET_ACCESS_KEY="[redacted]" STEP 4: USER root STEP 5: COPY upload/src /tmp/src STEP 6: RUN chown -R 1001:0 /tmp/src time="2019-11-10T11:31:20Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah638272527/mnt/rootfs\": error checking if \"/tmp/buildah638272527/mnt/rootfs/sys/fs/cgroup/blkio\" is mounted: no such file or directory" time="2019-11-10T11:31:20Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah638272527/mnt/rootfs\": error checking if \"/tmp/buildah638272527/mnt/rootfs/sys/fs/cgroup/blkio\" is mounted: no such file or directory" STEP 7: USER 1001 STEP 8: RUN /usr/libexec/s2i/assemble time="2019-11-10T12:51:34Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah105507608/mnt/rootfs\": error checking if \"/tmp/buildah105507608/mnt/rootfs/sys/fs/cgroup/blkio\" is mounted: no such file or directory" time="2019-11-10T12:51:35Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah105507608/mnt/rootfs\": error checking if \"/tmp/buildah105507608/mnt/rootfs/sys/fs/cgroup/blkio\" is mounted: no such file or directory" STEP 9: CMD /usr/libexec/s2i/run STEP 10: COMMIT temp.builder.openshift.io/openshift-image-registry/tensorflow-fftw-s2i-build-gpu-ubi-aws-1:788daa28 Getting image source signatures Copying blob sha256:6e3724f37b3ac2ed0c87dda2b6650e85783e1f8db7641a6e346b700dff902e66 Copying blob sha256:4306bb455f8f972e2030b3183297ba4923ad123973197a246e1ac1d88e736f3e Copying blob sha256:1e3bae1f064db4668810ac1c0eed6ac791e155566957c1664e5e5e5c2e8ef528 Copying blob sha256:2fbd9a6697f671bce0f878f741dc35967990740953e734cbe96bc02737460871 Copying blob sha256:f168b9267108db78ca746e0d8230eb5b755aed9ea64159456836d6da532020bb Copying blob sha256:20b7094b3ee984040e88d6d364df9739c1ad1a1e5a46c3d74fb2adc31786a437 Copying blob sha256:0d777e6651bfd039e134a0849b073481d4c5c296cfccfd58d159732f12301b50 error: build error: error committing container for step {Env:[AWS_SECRET_ACCESS_KEY=[redacted] AWS_ACCESS_KEY=[redacted] AWS_REGION=[redacted] AWS_PROFILE=[redacted] TENSORRT=[redacted] CUDNN=[redacted] NCCL=[redacted] PYTHON_EXECUTABLE=/usr/local/bin/python3 WHICH_SOURCE=s3 CC=/usr/bin/gcc NUM_DEVICES=2 DEVICE=gpu BACKEND=fftw RHEL_VERSION=7 OPENSHIFT_BUILD_NAMESPACE=openshift-image-registry OPENSHIFT_BUILD_NAME=tensorflow-fftw-s2i-build-gpu-ubi-aws-1 PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=oci CUDA_VERSION=10.0.130 CUDA_PKG_VERSION=10-0-10.0.130-1 LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 NVIDIA_VISIBLE_DEVICES=all NVIDIA_DRIVER_CAPABILITIES=compute,utility NVIDIA_REQUIRE_CUDA=cuda>=10.0 brand=tesla,driver>=384,driver<385 brand=tesla,driver>=410,driver<411 LANG=en_US.utf8 BUILDER_VERSION=1.0 RHEL_VERSION=7 PYTHON_VERSION=3.6.9 ANSIBLE_VERSION=v2.8.6 YUM_TIMEOUT=300 HOME=/home APPDIR=/home/benchmark_app USER=1001 PLAYBOOKS_DIR=/home/benchmark_app/playbooks FFTW_PREREQS_PLAYBOOK=/home/benchmark_app/playbooks/fftw_package_installation TF_PREREQS_PLAYBOOK=/home/benchmark_app/playbooks/tensorflow_package_installation FFTW_INSTALLATION=/home/benchmark_app/playbooks/FFTW_installation TENSORFLOW_INSTALLATION=/home/benchmark_app/playbooks/TensorFlow_installation TENSORFLOW_MODELS=/home/benchmark_app/playbooks/TensorFlow_Models NCCL_INSTALLATION=/home/benchmark_app/playbooks/nccl_installation CUDNN_INSTALLATION=/home/benchmark_app/playbooks/cudnn_installation TENSORRT_INSTALLATION=/home/benchmark_app/playbooks/tensorrt_installation SCRIPTS_DIR=/home/benchmark_app/scripts CUDA_PACKAGE_VERSION=10-0 OPENSHIFT_BUILD_NAME=tensorflow-fftw-build-image-ubi7-gpu-1 OPENSHIFT_BUILD_NAMESPACE=openshift-image-registry OPENSHIFT_BUILD_SOURCE=https://github.com/redhat-performance/aimlperf_reg_tests.git OPENSHIFT_BUILD_REFERENCE=master OPENSHIFT_BUILD_COMMIT=dba7a70ff9adbc0f43c99803590182acc08664a2 OPENSHIFT_BUILD_NAME=tensorflow-fftw-s2i-build-gpu-ubi-aws-1 OPENSHIFT_BUILD_NAMESPACE=openshift-image-registry RHEL_VERSION=7 BACKEND=fftw DEVICE=gpu NUM_DEVICES=2 CC=/usr/bin/gcc WHICH_SOURCE=s3 PYTHON_EXECUTABLE=/usr/local/bin/python3 NCCL=[redacted] CUDNN=[redacted] TENSORRT=[redacted] AWS_PROFILE=[redacted] AWS_REGION=[redacted] AWS_ACCESS_KEY=[redacted] AWS_SECRET_ACCESS_KEY=[redacted]] Command:cmd Args:[/usr/libexec/s2i/run] Flags:[] Attrs:map[] Message:CMD /usr/libexec/s2i/run Original:CMD /usr/libexec/s2i/run}: error copying layers and metadata for container "d7e5219ce2d176eb6042323c05e9a5c56ca8c3a5d754450237e75b2466fec586": Error reading blob sha256:621b5b2eabc61514bc65cd4489b848f4b44afc45b5ff402f8d2dbf40d1bc3b13: error reading blob from source image "containers-storage:tensorflow-fftw-base-ubi7-gpu-working-container": error opening file "/tmp/buildah383727552/sha256:621b5b2eabc61514bc65cd4489b848f4b44afc45b5ff402f8d2dbf40d1bc3b13" to buffer layer blob: open /tmp/buildah383727552/sha256:621b5b2eabc61514bc65cd4489b848f4b44afc45b5ff402f8d2dbf40d1bc3b13: no such file or directory Expected results: Container image commits successfully Additional info: I'm using Ansible playbooks to install packages from within the /usr/libexec/s2i/assemble script, so I've just removed the (lengthy) Ansible logs from my output above. No errors occur from my Ansible playbooks.
Looks identical enough that I'd consider this a duplicate. *** This bug has been marked as a duplicate of bug 1720730 ***