Bug 1770557 - s2i commit failure with custom Image Stream: "error reading blob from source image"
Summary: s2i commit failure with custom Image Stream: "error reading blob from source ...
Keywords:
Status: CLOSED DUPLICATE of bug 1720730
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 4.1.z
Hardware: Unspecified
OS: Linux
unspecified
medium
Target Milestone: ---
: ---
Assignee: Adam Kaplan
QA Contact: wewang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-11-10 14:19 UTC by Courtney Pacheco
Modified: 2019-11-11 20:41 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-11 20:41:44 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Courtney Pacheco 2019-11-10 14:19:26 UTC
Description of problem:
When I execute a Source-to-Image build that uses an existing ImageStreamTag as an image source (rather than using an image from Dockerhub, for example), OpenShift fails to commit the s2i container build, throwing the error: "error reading blob from source image." (Very similar to: https://bugzilla.redhat.com/show_bug.cgi?id=1720730)

Version-Release number of selected component (if applicable):
v4.1.0

How reproducible:
Always

Steps to Reproduce:
1. Create s2i assemble+run scripts
2. Create Dockerfile that pulls in those s2i scripts (e.g., "COPY s2i/bin /usr/libexec/s2i")
3. Create a BuildConfig template that will be used to generate an image from said Dockerfile. (The image installs yum/dnf/pip packages with root privileges; it does not execute the s2i scripts.) Call the template "source-buildconfig.yaml".
4. oc create -f source-buildconfig.yaml && oc new-app --template=my-dockerstrategy-buildconfig-template
5. oc start-build
6. After image successfully builds and the ImageStream tag has been created, create a BuildConfig template to generate an image using "sourceStrategy." The image generated from step #5 will be used as the BuildConfig's source image, with the s2i scripts already included in said source image.
7. oc create -f s2i-buildconfig.yaml && oc new-app --template=my-sourcestrategy-buildconfig-template

Actual results:
Caching blobs under "/var/cache/blobs".
Getting image source signatures
Copying blob sha256:24e41a8e056b3ac8c0418e3e6d7aa0a7bc0812e30827d03b193df3e474f6c03d
Copying blob sha256:79d11c1a86c4521558eb9e4b4173811ad15cd2c865eda4eb15a1a2e90af94136
Copying blob sha256:f84885f5ab7924c1c7a0a7d12861c16473ae09f5baedd0b2a8d7fcd9b017b8e5
Copying blob sha256:c8d67acdb2ffaebd638cf55a8fccc63693211060670aa7f0ea1d65b5d2c674dd
Copying blob sha256:e1b9b31a5585a2d263b5aa958337bbb1a125fab1d3b14f740e8e6ca93bd3e821
Copying blob sha256:3a674f761814e88c7e8f5a55dfa291bd1c62285f7f3cced2c6e3f6b7e295ab85
Copying blob sha256:c4fe78b1a926e2fe835db146b1bf6ece2c76d431b081e32c77e1a6e7e926069d
Copying config sha256:672e01b9107dddfbfedc83b693f461fc915a862f0d63a5b2451dfd74eabc95ee
Writing manifest to image destination
Storing signatures
672e01b9107dddfbfedc83b693f461fc915a862f0d63a5b2451dfd74eabc95ee
Generating dockerfile with builder image image-registry.openshift-image-registry.svc:5000/openshift-image-registry/tensorflow-fftw-base-ubi7-gpu@sha256:93b73c4d061adb4797cf8de5cb607f07162aa1f0ea635c5d102895c798d1da5e
STEP 1: FROM image-registry.openshift-image-registry.svc:5000/openshift-image-registry/tensorflow-fftw-base-ubi7-gpu@sha256:93b73c4d061adb4797cf8de5cb607f07162aa1f0ea635c5d102895c798d1da5e
STEP 2: LABEL "io.openshift.build.image"="image-registry.openshift-image-registry.svc:5000/openshift-image-registry/tensorflow-fftw-base-ubi7-gpu@sha256:93b73c4d061adb4797cf8de5cb607f07162aa1f0ea635c5d102895c798d1da5e" "io.openshift.build.source-location"="/tmp/build/inputs"
STEP 3: ENV OPENSHIFT_BUILD_NAME="tensorflow-fftw-s2i-build-gpu-ubi-aws-1" OPENSHIFT_BUILD_NAMESPACE="openshift-image-registry" RHEL_VERSION="7" BACKEND="fftw" DEVICE="gpu" NUM_DEVICES="2" CC="/usr/bin/gcc" WHICH_SOURCE="s3" PYTHON_EXECUTABLE="/usr/local/bin/python3" NCCL="[redacted]" CUDNN="[redacted]" TENSORRT="[redacted]" AWS_PROFILE="[redacted]" AWS_REGION="[redacted]" AWS_ACCESS_KEY="[redacted]" AWS_SECRET_ACCESS_KEY="[redacted]"
STEP 4: USER root
STEP 5: COPY upload/src /tmp/src
STEP 6: RUN chown -R 1001:0 /tmp/src
time="2019-11-10T11:31:20Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah638272527/mnt/rootfs\": error checking if \"/tmp/buildah638272527/mnt/rootfs/sys/fs/cgroup/blkio\" is mounted: no such file or directory"
time="2019-11-10T11:31:20Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah638272527/mnt/rootfs\": error checking if \"/tmp/buildah638272527/mnt/rootfs/sys/fs/cgroup/blkio\" is mounted: no such file or directory"
STEP 7: USER 1001
STEP 8: RUN /usr/libexec/s2i/assemble
time="2019-11-10T12:51:34Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah105507608/mnt/rootfs\": error checking if \"/tmp/buildah105507608/mnt/rootfs/sys/fs/cgroup/blkio\" is mounted: no such file or directory"
time="2019-11-10T12:51:35Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah105507608/mnt/rootfs\": error checking if \"/tmp/buildah105507608/mnt/rootfs/sys/fs/cgroup/blkio\" is mounted: no such file or directory"
STEP 9: CMD /usr/libexec/s2i/run
STEP 10: COMMIT temp.builder.openshift.io/openshift-image-registry/tensorflow-fftw-s2i-build-gpu-ubi-aws-1:788daa28
Getting image source signatures
Copying blob sha256:6e3724f37b3ac2ed0c87dda2b6650e85783e1f8db7641a6e346b700dff902e66
Copying blob sha256:4306bb455f8f972e2030b3183297ba4923ad123973197a246e1ac1d88e736f3e
Copying blob sha256:1e3bae1f064db4668810ac1c0eed6ac791e155566957c1664e5e5e5c2e8ef528
Copying blob sha256:2fbd9a6697f671bce0f878f741dc35967990740953e734cbe96bc02737460871
Copying blob sha256:f168b9267108db78ca746e0d8230eb5b755aed9ea64159456836d6da532020bb
Copying blob sha256:20b7094b3ee984040e88d6d364df9739c1ad1a1e5a46c3d74fb2adc31786a437
Copying blob sha256:0d777e6651bfd039e134a0849b073481d4c5c296cfccfd58d159732f12301b50
error: build error: error committing container for step {Env:[AWS_SECRET_ACCESS_KEY=[redacted] AWS_ACCESS_KEY=[redacted] AWS_REGION=[redacted] AWS_PROFILE=[redacted] TENSORRT=[redacted] CUDNN=[redacted] NCCL=[redacted] PYTHON_EXECUTABLE=/usr/local/bin/python3 WHICH_SOURCE=s3 CC=/usr/bin/gcc NUM_DEVICES=2 DEVICE=gpu BACKEND=fftw RHEL_VERSION=7 OPENSHIFT_BUILD_NAMESPACE=openshift-image-registry OPENSHIFT_BUILD_NAME=tensorflow-fftw-s2i-build-gpu-ubi-aws-1 PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=oci CUDA_VERSION=10.0.130 CUDA_PKG_VERSION=10-0-10.0.130-1 LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 NVIDIA_VISIBLE_DEVICES=all NVIDIA_DRIVER_CAPABILITIES=compute,utility NVIDIA_REQUIRE_CUDA=cuda>=10.0 brand=tesla,driver>=384,driver<385 brand=tesla,driver>=410,driver<411 LANG=en_US.utf8 BUILDER_VERSION=1.0 RHEL_VERSION=7 PYTHON_VERSION=3.6.9 ANSIBLE_VERSION=v2.8.6 YUM_TIMEOUT=300 HOME=/home APPDIR=/home/benchmark_app USER=1001 PLAYBOOKS_DIR=/home/benchmark_app/playbooks FFTW_PREREQS_PLAYBOOK=/home/benchmark_app/playbooks/fftw_package_installation TF_PREREQS_PLAYBOOK=/home/benchmark_app/playbooks/tensorflow_package_installation FFTW_INSTALLATION=/home/benchmark_app/playbooks/FFTW_installation TENSORFLOW_INSTALLATION=/home/benchmark_app/playbooks/TensorFlow_installation TENSORFLOW_MODELS=/home/benchmark_app/playbooks/TensorFlow_Models NCCL_INSTALLATION=/home/benchmark_app/playbooks/nccl_installation CUDNN_INSTALLATION=/home/benchmark_app/playbooks/cudnn_installation TENSORRT_INSTALLATION=/home/benchmark_app/playbooks/tensorrt_installation SCRIPTS_DIR=/home/benchmark_app/scripts CUDA_PACKAGE_VERSION=10-0 OPENSHIFT_BUILD_NAME=tensorflow-fftw-build-image-ubi7-gpu-1 OPENSHIFT_BUILD_NAMESPACE=openshift-image-registry OPENSHIFT_BUILD_SOURCE=https://github.com/redhat-performance/aimlperf_reg_tests.git OPENSHIFT_BUILD_REFERENCE=master OPENSHIFT_BUILD_COMMIT=dba7a70ff9adbc0f43c99803590182acc08664a2                 OPENSHIFT_BUILD_NAME=tensorflow-fftw-s2i-build-gpu-ubi-aws-1 OPENSHIFT_BUILD_NAMESPACE=openshift-image-registry RHEL_VERSION=7 BACKEND=fftw DEVICE=gpu NUM_DEVICES=2 CC=/usr/bin/gcc WHICH_SOURCE=s3 PYTHON_EXECUTABLE=/usr/local/bin/python3 NCCL=[redacted] CUDNN=[redacted] TENSORRT=[redacted] AWS_PROFILE=[redacted] AWS_REGION=[redacted] AWS_ACCESS_KEY=[redacted] AWS_SECRET_ACCESS_KEY=[redacted]] Command:cmd Args:[/usr/libexec/s2i/run] Flags:[] Attrs:map[] Message:CMD /usr/libexec/s2i/run Original:CMD /usr/libexec/s2i/run}: error copying layers and metadata for container "d7e5219ce2d176eb6042323c05e9a5c56ca8c3a5d754450237e75b2466fec586": Error reading blob sha256:621b5b2eabc61514bc65cd4489b848f4b44afc45b5ff402f8d2dbf40d1bc3b13: error reading blob from source image "containers-storage:tensorflow-fftw-base-ubi7-gpu-working-container": error opening file "/tmp/buildah383727552/sha256:621b5b2eabc61514bc65cd4489b848f4b44afc45b5ff402f8d2dbf40d1bc3b13" to buffer layer blob: open /tmp/buildah383727552/sha256:621b5b2eabc61514bc65cd4489b848f4b44afc45b5ff402f8d2dbf40d1bc3b13: no such file or directory

Expected results:
Container image commits successfully

Additional info:
I'm using Ansible playbooks to install packages from within the /usr/libexec/s2i/assemble script, so I've just removed the (lengthy) Ansible logs from my output above. No errors occur from my Ansible playbooks.

Comment 1 Adam Kaplan 2019-11-11 20:41:44 UTC
Looks identical enough that I'd consider this a duplicate.

*** This bug has been marked as a duplicate of bug 1720730 ***


Note You need to log in before you can comment on or make changes to this bug.