Bug 1720730 - nodejs s2i build failure: 'error reading blob from source image' [NEEDINFO]
Summary: nodejs s2i build failure: 'error reading blob from source image'
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers
Version: 4.2.z
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.6.0
Assignee: Nalin Dahyabhai
QA Contact: MinLi
URL:
Whiteboard:
: 1770557 (view as bug list)
Depends On:
Blocks: 1892671
TreeView+ depends on / blocked
 
Reported: 2019-06-14 17:11 UTC by Justin Pierce
Modified: 2020-10-29 12:33 UTC (History)
22 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: when committing an image, the image builder logic would compute the contents of new layers twice. The logic which cached layer contents depended on the products of those calculations remaining consistent. Consequence: if a new layer's contents changed between the two computations, the cache would be unable to supply the layer's contents when they were needed for committing or pushing an image. Fix: The contents of new layers are no longer computed twice, as doing so was neither necessary nor intentional. Result: Image builds and pushes should no longer fail with the "error reading blob from source" error.
Clone Of:
Environment:
Last Closed: 2020-10-27 15:54:19 UTC
Target Upstream Version:
adam.kaplan: needinfo? (jupierce)
ocasalsa: needinfo? (nalin)
tsweeney: needinfo? (nalin)


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift builder pull 168 0 None closed Bug 1720730: bump github.com/containers/buildah to v1.14.11 2021-02-15 15:45:59 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 15:54:36 UTC

Description Justin Pierce 2019-06-14 17:11:53 UTC
Description of problem:
Starter clusters run a job which periodically runs an s2i build job to assess cluster / release health.

Version-Release number of selected component (if applicable):
v4.1.0

How reproducible:
Infrequent

Steps to Reproduce:
1. oc new-app https://github.com/sclorg/nodejs-ex
2.
3.

Actual results:
[ec2-user@stg-2.bastion us-east-1 ~]$ oc logs crel-monitors-app-creation-test-1-build
Caching blobs under "/var/cache/blobs".
Getting image source signatures
Copying blob sha256:79e6026c00814beeeb15d4da99c69f211289421da4674c910d3dd2ce3e3e6c2b
Copying blob sha256:256b176beaff7815db2a93ee2071621ae88f451bb1e198ca73010ed5bba79b65
Copying blob sha256:03f625cd1c87c58145f4dccb474c2bde16e630ed0a73b04c78b17a32df8ed6c4
Copying blob sha256:e6ecde3f02819eb52cd96cfa211ee4e6c654cfeda797cf1556fe4dd18d477f28
Copying blob sha256:704165bbeb28700bb2c89a86f143b57115a7e6f511a0d8b1529bf07356a1cfa3
Copying blob sha256:ba1c23d1d04040ca2953599550576c3fac98539afe81978d96665a8fd6305d85
Copying config sha256:93de1230c12b512ebbaf28b159f450a44c632eda06bdc0754236f403f5876234
Writing manifest to image destination
Storing signatures
93de1230c12b512ebbaf28b159f450a44c632eda06bdc0754236f403f5876234
Generating dockerfile with builder image image-registry.openshift-image-registry.svc:5000/openshift-monitoring/crel-monitors-app-creation-nodejs@sha256:3cc041334eef8d5853078a0190e46a2998a70ad98320db512968f1de0561705e
STEP 1: FROM image-registry.openshift-image-registry.svc:5000/openshift-monitoring/crel-monitors-app-creation-nodejs@sha256:3cc041334eef8d5853078a0190e46a2998a70ad98320db512968f1de0561705e
STEP 2: LABEL "io.openshift.build.source-location"="https://github.com/sclorg/nodejs-ex" "io.openshift.build.image"="image-registry.openshift-image-registry.svc:5000/openshift-monitoring/crel-monitors-app-creation-nodejs@sha256:3cc041334eef8d5853078a0190e46a2998a70ad98320db512968f1de0561705e" "io.openshift.build.commit.author"="Honza Horak <hhorak@redhat.com>" "io.openshift.build.commit.date"="Tue Oct 16 15:45:10 2018 +0200" "io.openshift.build.commit.id"="e59fe7571f883db2ae2e53d555aef6d145c6f032" "io.openshift.build.commit.ref"="master" "io.openshift.build.commit.message"="Merge pull request #206 from liangxia/okd"
STEP 3: ENV OPENSHIFT_BUILD_NAME="crel-monitors-app-creation-test-1" OPENSHIFT_BUILD_NAMESPACE="openshift-monitoring" OPENSHIFT_BUILD_SOURCE="https://github.com/sclorg/nodejs-ex" OPENSHIFT_BUILD_COMMIT="e59fe7571f883db2ae2e53d555aef6d145c6f032"
STEP 4: USER root
STEP 5: COPY upload/src /tmp/src
STEP 6: RUN chown -R 1001:0 /tmp/src
time="2019-06-14T16:42:15Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah208180852/mnt/rootfs\": error checking if \"/tmp/buildah208180852/mnt/rootfs/sys/fs/cgroup/hugetlb\" is mounted: no such file or directory"
time="2019-06-14T16:42:15Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah208180852/mnt/rootfs\": error checking if \"/tmp/buildah208180852/mnt/rootfs/sys/fs/cgroup/hugetlb\" is mounted: no such file or directory"
STEP 7: USER 1001
STEP 8: RUN /usr/libexec/s2i/assemble
---> Installing application source
---> Building your Node application from source
Current git config
url.https://github.com.insteadof=git@github.com:
url.https://.insteadof=ssh://
url.https://github.com.insteadof=ssh://git@github.com
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
remote.origin.url=https://github.com/sclorg/nodejs-ex
remote.origin.fetch=+refs/heads/master:refs/remotes/origin/master
branch.master.remote=origin
branch.master.merge=refs/heads/master
---> Installing dependencies
---> Using 'npm install -s --only=production'
added 119 packages from 343 contributors and audited 205 packages in 2.753s
found 5 vulnerabilities (2 low, 1 moderate, 1 high, 1 critical)
  run `npm audit fix` to fix them, or `npm audit` for details
---> Pruning the development dependencies
npm info it worked if it ends with ok
npm info using npm@6.4.1
npm info using node@v10.12.0
npm timing stage:loadCurrentTree Completed in 263ms
npm timing stage:loadIdealTree:cloneCurrentTree Completed in 3ms
npm timing stage:loadIdealTree:loadShrinkwrap Completed in 117ms
npm timing stage:loadIdealTree:loadAllDepsIntoIdealTree Completed in 58ms
npm timing stage:loadIdealTree Completed in 214ms
npm timing stage:generateActionsToTake Completed in 17ms
npm timing audit compress Completed in 8ms
npm info audit Submitting payload of 6939bytes
npm timing stage:executeActions Completed in 188ms
npm timing stage:rollbackFailedOptional Completed in 1ms
npm timing stage:runTopLevelLifecycles Completed in 730ms
npm timing audit submit Completed in 461ms
npm http fetch POST 200 https://registry.npmjs.org/-/npm/v1/security/audits/quick 335ms
npm timing audit body Completed in 1ms
audited 205 packages in 1.018s
found 5 vulnerabilities (2 low, 1 moderate, 1 high, 1 critical)
  run `npm audit fix` to fix them, or `npm audit` for details
npm timing npm Completed in 1340ms
npm info ok 
---> Cleaning up npm cache
---> Fix permissions on app-root
time="2019-06-14T16:42:21Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah153080817/mnt/rootfs\": error checking if \"/tmp/buildah153080817/mnt/rootfs/sys/fs/cgroup/hugetlb\" is mounted: no such file or directory"
time="2019-06-14T16:42:21Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah153080817/mnt/rootfs\": error checking if \"/tmp/buildah153080817/mnt/rootfs/sys/fs/cgroup/hugetlb\" is mounted: no such file or directory"
STEP 9: CMD /usr/libexec/s2i/run
STEP 10: COMMIT temp.builder.openshift.io/openshift-monitoring/crel-monitors-app-creation-test-1:9d06bbc5
Getting image source signatures
Copying blob sha256:1d31b5806ba40b5f67bde96f18a181668348934a44c9253b420d5f04cfb4e37a
Copying blob sha256:8a3ecdef9d718dcbb591ae5cd686f396e0502124a9d16599f5b5445ebd10a42b
Copying blob sha256:8c5aacfc54e911677fd9a99cecd6e99ba21335c3817c1bf4bbced91a7410b61f
Copying blob sha256:da876e87981d9b4b818ececd8ca4fc286155758c8e30f2699513c540f03fb834
Copying blob sha256:610f3ca133f72b4abcef0c57482352ffd393fcd9604a98f55ca6dea7bab1c87d
Copying blob sha256:0886316cbb8d35a07f2b503e30b787063b0e71c7a8c6331309967a5c02d8784d
error: build error: error committing container for step {Env:[OPENSHIFT_BUILD_COMMIT=e59fe7571f883db2ae2e53d555aef6d145c6f032 OPENSHIFT_BUILD_SOURCE=https://github.com/sclorg/nodejs-ex OPENSHIFT_BUILD_NAMESPACE=openshift-monitoring OPENSHIFT_BUILD_NAME=crel-monitors-app-creation-test-1 PATH=/opt/app-root/src/node_modules/.bin/:/opt/app-root/src/.npm-global/bin/:/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin SUMMARY=Platform for building and running Node.js 10.12.0 applications DESCRIPTION=Node.js  available as docker container is a base platform for building and running various Node.js  applications and frameworks. Node.js is a platform built on Chrome's JavaScript runtime for easily building fast, scalable network applications. Node.js uses an event-driven, non-blocking I/O model that makes it lightweight and efficient, perfect for data-intensive real-time applications that run across distributed devices. STI_SCRIPTS_URL=image:///usr/libexec/s2i STI_SCRIPTS_PATH=/usr/libexec/s2i APP_ROOT=/opt/app-root HOME=/opt/app-root/src BASH_ENV=/opt/app-root/etc/scl_enable ENV=/opt/app-root/etc/scl_enable PROMPT_COMMAND=. /opt/app-root/etc/scl_enable NODEJS_SCL=rh-nodejs8 NPM_RUN=start NODE_VERSION=10.12.0 NPM_VERSION=6.4.1 NODE_LTS=false NPM_CONFIG_LOGLEVEL=info NPM_CONFIG_PREFIX=/opt/app-root/src/.npm-global NPM_CONFIG_TARBALL=/usr/share/node/node-v10.12.0-headers.tar.gz DEBUG_PORT=5858     OPENSHIFT_BUILD_NAME=crel-monitors-app-creation-test-1 OPENSHIFT_BUILD_NAMESPACE=openshift-monitoring OPENSHIFT_BUILD_SOURCE=https://github.com/sclorg/nodejs-ex OPENSHIFT_BUILD_COMMIT=e59fe7571f883db2ae2e53d555aef6d145c6f032] Command:cmd Args:[/usr/libexec/s2i/run] Flags:[] Attrs:map[] Message:CMD /usr/libexec/s2i/run Original:CMD /usr/libexec/s2i/run}: error copying layers and metadata for container "67aeb3700e3a42957dd62747196b8e6a3c173dac909e419f333eda9a8a49b909": Error reading blob sha256:8f5f2fa4e24a2935b1c9b90d5e0affc0afd1efe3229074aafd1af04e789d8e8f: error reading blob from source image "containers-storage:crel-monitors-app-creation-nodejs-working-container": error opening file "/tmp/buildah864503225/sha256:8f5f2fa4e24a2935b1c9b90d5e0affc0afd1efe3229074aafd1af04e789d8e8f" to buffer layer blob: open /tmp/buildah864503225/sha256:8f5f2fa4e24a2935b1c9b90d5e0affc0afd1efe3229074aafd1af04e789d8e8f: no such file or directory

Comment 1 Adam Kaplan 2019-06-17 20:00:09 UTC
@Nalin - based on the logs, looks like a layer got deleted from underneath buildah...any idea how this could happen?

Comment 2 Nalin Dahyabhai 2019-08-20 22:28:08 UTC
@adam It's not something we expect, but if there was a problem generating the layer blob file, I'd also have expected an error message.  Any chance we can get a log from a reproducing run with BUILD_LOGLEVEL=6, so that we can more easily follow what's going on during that COMMIT step?

Comment 3 Adam Kaplan 2019-08-30 01:18:47 UTC
@Justin can you try `oc new-app --loglevel=6 https://github.com/sclorg/nodejs-ex`?

Targeting 4.3 as it is not likely we'll fix this in time for 4.2

Comment 4 Justin Pierce 2019-08-30 12:58:01 UTC
I've increased the loglevel in our monitoring code. If it happens again, we will update the BZ.

Comment 6 Adam Kaplan 2019-11-11 20:41:44 UTC
*** Bug 1770557 has been marked as a duplicate of this bug. ***

Comment 9 Adler 2020-03-12 16:18:57 UTC
We have been having this issue in 4.2 clusters:

time="2020-03-12T15:59:33Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah416993853/mnt/rootfs\": error checking if \"/tmp/buildah416993853/mnt/rootfs/sys/fs/cgroup/hugetlb\" is mounted: no such file or directory"
time="2020-03-12T15:59:33Z" level=debug msg="error cleaning up intermediate chroot bind mounts: <nil>"
time="2020-03-12T15:59:33Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah416993853/mnt/rootfs\": error checking if \"/tmp/buildah416993853/mnt/rootfs/sys/fs/cgroup/hugetlb\" is mounted: no such file or directory"
time="2020-03-12T15:59:33Z" level=debug msg="error cleaning up intermediate mount NS: <nil>"
time="2020-03-12T15:59:33Z" level=debug msg=COMMIT
time="2020-03-12T15:59:33Z" level=debug msg="parsed reference into \"[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.imagestore=/var/lib/shared]docker.io/library/40e402db80b43803dfa961ad8fbf2918e91e49ad7cc99b25a7b3d59be0c8161b-tmp:latest\""
time="2020-03-12T15:59:33Z" level=debug msg="parsed reference into \"[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.imagestore=/var/lib/shared]@021b56a848a78cc8571f81f8ec55f381ca0640588335be8a6dfa833acb82a43d\""
time="2020-03-12T15:59:33Z" level=debug msg="base image \"021b56a848a78cc8571f81f8ec55f381ca0640588335be8a6dfa833acb82a43d\" is already present in local storage, no need to copy its layers"
time="2020-03-12T15:59:33Z" level=debug msg="starting to write to image \"containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.imagestore=/var/lib/shared]docker.io/library/40e402db80b43803dfa961ad8fbf2918e91e49ad7cc99b25a7b3d59be0c8161b-tmp:latest\" using blob cache in \"/var/cache/blobs\""
time="2020-03-12T15:59:33Z" level=debug msg="layer list: [\"2f4129b6dea024391198419be227846bebc4a63d94a5c276c2920a3701f41651\" \"005e12f2f5ea5e2173bf1d412c05792030fbf26c83de504765642fcd1327132f\" \"70b473ffac7a6e0ffc93e96946096851f0ade78b662176be1b4341df3331795c\" \"43ec515f3a59344a3872f435bcafa94b2e5351fcc8cf7f3eff67f12f36491002\" \"14aeefdbf4cb8900aa9bfc76142942d31cab0f58d0ad71ffa9ff9bd801caba7a\"]"
time="2020-03-12T15:59:33Z" level=debug msg="using \"/tmp/buildah481524101\" to hold temporary data"
time="2020-03-12T15:59:33Z" level=debug msg="Tar with options on /var/lib/containers/storage/overlay/14aeefdbf4cb8900aa9bfc76142942d31cab0f58d0ad71ffa9ff9bd801caba7a/diff"
time="2020-03-12T15:59:38Z" level=debug msg="layer \"14aeefdbf4cb8900aa9bfc76142942d31cab0f58d0ad71ffa9ff9bd801caba7a\" size is 595258368 bytes"
time="2020-03-12T15:59:38Z" level=debug msg="OCIv1 config = {\"created\":\"2020-03-12T15:59:33.235872979Z\",\"architecture\":\"amd64\",\"os\":\"linux\",\"config\":{\"User\":\"1001\",\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"container=oci\",\"OPENSHIFT_BUILD_NAME=openjdk-devel-1\",\"OPENSHIFT_BUILD_NAMESPACE=zz-cluster-load-test-0231\",\"BUILD_LOGLEVEL=6\"],\"Cmd\":[\"/bin/bash\"],\"WorkingDir\":\"/builds\",\"Labels\":{\"architecture\":\"x86_64\",\"authoritative-source-url\":\"registry.access.redhat.com\",\"build-date\":\"2020-01-28T10:58:57.851295\",\"com.redhat.build-host\":\"cpt-1004.osbs.prod.upshift.rdu2.redhat.com\",\"com.redhat.component\":\"ubi7-minimal-container\",\"com.redhat.license_terms\":\"https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI\",\"description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"distribution-scope\":\"public\",\"io.k8s.description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"io.k8s.display-name\":\"Red Hat Universal Base Image 7 Minimal\",\"io.openshift.build.name\":\"openjdk-devel-1\",\"io.openshift.build.namespace\":\"zz-cluster-load-test-0231\",\"io.openshift.tags\":\"minimal rhel7\",\"maintainer\":\"Red Hat, Inc.\",\"name\":\"ubi7-minimal\",\"release\":\"303\",\"summary\":\"Provides the latest release of the minimal Red Hat Universal Base Image 7.\",\"url\":\"https://access.redhat.com/containers/#/registry.access.redhat.com/ubi7-minimal/images/7.7-303\",\"vcs-ref\":\"7e9dd8b61a03561c4b5f70b214bc26d19c4f25a4\",\"vcs-type\":\"git\",\"vendor\":\"Red Hat, Inc.\",\"version\":\"7.7\"}},\"rootfs\":{\"type\":\"layers\",\"diff_ids\":[\"sha256:2f4129b6dea024391198419be227846bebc4a63d94a5c276c2920a3701f41651\",\"sha256:a5d78edbc6523a89a2687ed7e2818bc15851a6fd62ac5833a4b12c2b38d07cad\",\"sha256:7772f548ccef4b2910e63af9bc31ab8324f58749436043e5c7ed1d263f0a5aa8\",\"sha256:603a0d3f7ac375cb6d28e4b4495d41524347d96730ad9451923a202ee1ba7634\",\"sha256:6d6ef9321d9777d2be90d5e747f19554f13e29d08bb558daf6e00a644b15f12f\"]},\"history\":[{\"created\":\"2020-01-28T10:59:06.259857804Z\",\"comment\":\"Imported from -\"},{\"created\":\"2020-01-28T10:59:15.548872Z\"},{\"created\":\"2020-03-12T14:58:55.410578153Z\",\"created_by\":\"/bin/sh -c microdnf install which java-11-openjdk-devel \\u0026\\u0026 mkdir /builds \\u0026\\u0026 chgrp -R 0 /builds \\u0026\\u0026 chmod -R g=u /builds\"},{\"created\":\"2020-03-12T14:59:11.113871897Z\",\"created_by\":\"/bin/sh -c #(nop) USER 1001\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:11.89385222Z\",\"created_by\":\"/bin/sh -c #(nop) WORKDIR /builds\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:12.658658208Z\",\"created_by\":\"/bin/sh -c #(nop) ENV \\\"OPENSHIFT_BUILD_NAME\\\"=\\\"openjdk-devel-1\\\" \\\"OPENSHIFT_BUILD_NAMESPACE\\\"=\\\"zz-cluster-load-test-0231\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:13.436671505Z\",\"created_by\":\"/bin/sh -c #(nop) LABEL \\\"io.openshift.build.name\\\"=\\\"openjdk-devel-1\\\" \\\"io.openshift.build.namespace\\\"=\\\"zz-cluster-load-test-0231\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T15:57:15.906697248Z\",\"created_by\":\"/bin/sh -c #(nop) ENV \\\"BUILD_LOGLEVEL\\\"=\\\"6\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T15:57:18.063087881Z\",\"created_by\":\"/bin/sh -c #(nop) COPY . / ./\"},{\"created\":\"2020-03-12T15:59:33.235872979Z\",\"created_by\":\"/bin/sh -c ./gradlew build \\u0026\\u0026 mv build/libs/spring-boot-rest-example-*.jar spring-boot-rest-example.jar\"}]}"
time="2020-03-12T15:59:38Z" level=debug msg="OCIv1 manifest = {\"schemaVersion\":2,\"config\":{\"mediaType\":\"application/vnd.oci.image.config.v1+json\",\"digest\":\"sha256:ea126b0f7200880c6ece6817e92dca1334eab853edd79f349ae072b9af3f929b\",\"size\":3700},\"layers\":[{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:2f4129b6dea024391198419be227846bebc4a63d94a5c276c2920a3701f41651\",\"size\":83005440},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:a5d78edbc6523a89a2687ed7e2818bc15851a6fd62ac5833a4b12c2b38d07cad\",\"size\":20480},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:7772f548ccef4b2910e63af9bc31ab8324f58749436043e5c7ed1d263f0a5aa8\",\"size\":358737408},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:603a0d3f7ac375cb6d28e4b4495d41524347d96730ad9451923a202ee1ba7634\",\"size\":1838080},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:6d6ef9321d9777d2be90d5e747f19554f13e29d08bb558daf6e00a644b15f12f\",\"size\":595258368}]}"
time="2020-03-12T15:59:38Z" level=debug msg="Docker v2s2 config = {\"created\":\"2020-03-12T15:59:33.235872979Z\",\"container_config\":{\"Hostname\":\"27dbfa3c598a\",\"Domainname\":\"\",\"User\":\"1001\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"container=oci\",\"OPENSHIFT_BUILD_NAME=openjdk-devel-1\",\"OPENSHIFT_BUILD_NAMESPACE=zz-cluster-load-test-0231\",\"BUILD_LOGLEVEL=6\"],\"Cmd\":[\"/bin/bash\"],\"ArgsEscaped\":true,\"Image\":\"5b069ae67d4e7eaea54e5001b89fbd5c076bd27f2ba04bfa87745c3c96e687a9\",\"Volumes\":{},\"WorkingDir\":\"/builds\",\"Entrypoint\":[],\"OnBuild\":[],\"Labels\":{\"architecture\":\"x86_64\",\"authoritative-source-url\":\"registry.access.redhat.com\",\"build-date\":\"2020-01-28T10:58:57.851295\",\"com.redhat.build-host\":\"cpt-1004.osbs.prod.upshift.rdu2.redhat.com\",\"com.redhat.component\":\"ubi7-minimal-container\",\"com.redhat.license_terms\":\"https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI\",\"description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"distribution-scope\":\"public\",\"io.k8s.description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"io.k8s.display-name\":\"Red Hat Universal Base Image 7 Minimal\",\"io.openshift.build.name\":\"openjdk-devel-1\",\"io.openshift.build.namespace\":\"zz-cluster-load-test-0231\",\"io.openshift.tags\":\"minimal rhel7\",\"maintainer\":\"Red Hat, Inc.\",\"name\":\"ubi7-minimal\",\"release\":\"303\",\"summary\":\"Provides the latest release of the minimal Red Hat Universal Base Image 7.\",\"url\":\"https://access.redhat.com/containers/#/registry.access.redhat.com/ubi7-minimal/images/7.7-303\",\"vcs-ref\":\"7e9dd8b61a03561c4b5f70b214bc26d19c4f25a4\",\"vcs-type\":\"git\",\"vendor\":\"Red Hat, Inc.\",\"version\":\"7.7\"}},\"config\":{\"Hostname\":\"27dbfa3c598a\",\"Domainname\":\"\",\"User\":\"1001\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"container=oci\",\"OPENSHIFT_BUILD_NAME=openjdk-devel-1\",\"OPENSHIFT_BUILD_NAMESPACE=zz-cluster-load-test-0231\",\"BUILD_LOGLEVEL=6\"],\"Cmd\":[\"/bin/bash\"],\"ArgsEscaped\":true,\"Image\":\"5b069ae67d4e7eaea54e5001b89fbd5c076bd27f2ba04bfa87745c3c96e687a9\",\"Volumes\":{},\"WorkingDir\":\"/builds\",\"Entrypoint\":[],\"OnBuild\":[],\"Labels\":{\"architecture\":\"x86_64\",\"authoritative-source-url\":\"registry.access.redhat.com\",\"build-date\":\"2020-01-28T10:58:57.851295\",\"com.redhat.build-host\":\"cpt-1004.osbs.prod.upshift.rdu2.redhat.com\",\"com.redhat.component\":\"ubi7-minimal-container\",\"com.redhat.license_terms\":\"https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI\",\"description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"distribution-scope\":\"public\",\"io.k8s.description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"io.k8s.display-name\":\"Red Hat Universal Base Image 7 Minimal\",\"io.openshift.build.name\":\"openjdk-devel-1\",\"io.openshift.build.namespace\":\"zz-cluster-load-test-0231\",\"io.openshift.tags\":\"minimal rhel7\",\"maintainer\":\"Red Hat, Inc.\",\"name\":\"ubi7-minimal\",\"release\":\"303\",\"summary\":\"Provides the latest release of the minimal Red Hat Universal Base Image 7.\",\"url\":\"https://access.redhat.com/containers/#/registry.access.redhat.com/ubi7-minimal/images/7.7-303\",\"vcs-ref\":\"7e9dd8b61a03561c4b5f70b214bc26d19c4f25a4\",\"vcs-type\":\"git\",\"vendor\":\"Red Hat, Inc.\",\"version\":\"7.7\"}},\"architecture\":\"amd64\",\"os\":\"linux\",\"parent\":\"sha256:021b56a848a78cc8571f81f8ec55f381ca0640588335be8a6dfa833acb82a43d\",\"rootfs\":{\"type\":\"layers\",\"diff_ids\":[\"sha256:2f4129b6dea024391198419be227846bebc4a63d94a5c276c2920a3701f41651\",\"sha256:a5d78edbc6523a89a2687ed7e2818bc15851a6fd62ac5833a4b12c2b38d07cad\",\"sha256:7772f548ccef4b2910e63af9bc31ab8324f58749436043e5c7ed1d263f0a5aa8\",\"sha256:603a0d3f7ac375cb6d28e4b4495d41524347d96730ad9451923a202ee1ba7634\",\"sha256:6d6ef9321d9777d2be90d5e747f19554f13e29d08bb558daf6e00a644b15f12f\"]},\"history\":[{\"created\":\"2020-01-28T10:59:06.259857804Z\",\"comment\":\"Imported from -\"},{\"created\":\"2020-01-28T10:59:15.548872Z\"},{\"created\":\"2020-03-12T14:58:55.410578153Z\",\"created_by\":\"/bin/sh -c microdnf install which java-11-openjdk-devel \\u0026\\u0026 mkdir /builds \\u0026\\u0026 chgrp -R 0 /builds \\u0026\\u0026 chmod -R g=u /builds\"},{\"created\":\"2020-03-12T14:59:11.113871897Z\",\"created_by\":\"/bin/sh -c #(nop) USER 1001\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:11.89385222Z\",\"created_by\":\"/bin/sh -c #(nop) WORKDIR /builds\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:12.658658208Z\",\"created_by\":\"/bin/sh -c #(nop) ENV \\\"OPENSHIFT_BUILD_NAME\\\"=\\\"openjdk-devel-1\\\" \\\"OPENSHIFT_BUILD_NAMESPACE\\\"=\\\"zz-cluster-load-test-0231\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:13.436671505Z\",\"created_by\":\"/bin/sh -c #(nop) LABEL \\\"io.openshift.build.name\\\"=\\\"openjdk-devel-1\\\" \\\"io.openshift.build.namespace\\\"=\\\"zz-cluster-load-test-0231\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T15:57:15.906697248Z\",\"created_by\":\"/bin/sh -c #(nop) ENV \\\"BUILD_LOGLEVEL\\\"=\\\"6\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T15:57:18.063087881Z\",\"created_by\":\"/bin/sh -c #(nop) COPY . / ./\"},{\"created\":\"2020-03-12T15:59:33.235872979Z\",\"created_by\":\"/bin/sh -c ./gradlew build \\u0026\\u0026 mv build/libs/spring-boot-rest-example-*.jar spring-boot-rest-example.jar\"}]}"
time="2020-03-12T15:59:38Z" level=debug msg="Docker v2s2 manifest = {\"schemaVersion\":2,\"mediaType\":\"application/vnd.docker.distribution.manifest.v2+json\",\"config\":{\"mediaType\":\"application/vnd.docker.container.image.v1+json\",\"size\":6214,\"digest\":\"sha256:2c12cedbbd68807c0864eb3f97e8267b3dba63128333e0831dcb58247b14151d\"},\"layers\":[{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":83005440,\"digest\":\"sha256:2f4129b6dea024391198419be227846bebc4a63d94a5c276c2920a3701f41651\"},{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":20480,\"digest\":\"sha256:a5d78edbc6523a89a2687ed7e2818bc15851a6fd62ac5833a4b12c2b38d07cad\"},{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":358737408,\"digest\":\"sha256:7772f548ccef4b2910e63af9bc31ab8324f58749436043e5c7ed1d263f0a5aa8\"},{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":1838080,\"digest\":\"sha256:603a0d3f7ac375cb6d28e4b4495d41524347d96730ad9451923a202ee1ba7634\"},{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":595258368,\"digest\":\"sha256:6d6ef9321d9777d2be90d5e747f19554f13e29d08bb558daf6e00a644b15f12f\"}]}"
time="2020-03-12T15:59:38Z" level=debug msg="starting to read from image \"containers-storage:021b56a848a78cc8571f81f8ec55f381ca0640588335be8a6dfa833acb82a43d-working-container\" using blob cache in \"/var/cache/blobs\" (compression=0)"
time="2020-03-12T15:59:38Z" level=debug msg="Using blob info cache at /var/lib/containers/storage/cache/blob-info-cache-v1.boltdb"
time="2020-03-12T15:59:38Z" level=debug msg="IsRunningImageAllowed for image containers-storage:"
time="2020-03-12T15:59:38Z" level=debug msg=" Using transport \"containers-storage\" policy section "
time="2020-03-12T15:59:38Z" level=debug msg=" Requirement 0: allowed"
time="2020-03-12T15:59:38Z" level=debug msg="Overall: allowed"
time="2020-03-12T15:59:38Z" level=debug msg="start reading config"
time="2020-03-12T15:59:38Z" level=debug msg="finished reading config"
time="2020-03-12T15:59:38Z" level=debug msg="Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json]"
time="2020-03-12T15:59:38Z" level=debug msg="... will first try using the original manifest unmodified"
time="2020-03-12T15:59:38Z" level=debug msg="layer list: [\"2f4129b6dea024391198419be227846bebc4a63d94a5c276c2920a3701f41651\" \"005e12f2f5ea5e2173bf1d412c05792030fbf26c83de504765642fcd1327132f\" \"70b473ffac7a6e0ffc93e96946096851f0ade78b662176be1b4341df3331795c\" \"43ec515f3a59344a3872f435bcafa94b2e5351fcc8cf7f3eff67f12f36491002\" \"14aeefdbf4cb8900aa9bfc76142942d31cab0f58d0ad71ffa9ff9bd801caba7a\"]"
time="2020-03-12T15:59:38Z" level=debug msg="using \"/tmp/buildah301210912\" to hold temporary data"
time="2020-03-12T15:59:38Z" level=debug msg="Tar with options on /var/lib/containers/storage/overlay/14aeefdbf4cb8900aa9bfc76142942d31cab0f58d0ad71ffa9ff9bd801caba7a/diff"
time="2020-03-12T15:59:43Z" level=debug msg="layer \"14aeefdbf4cb8900aa9bfc76142942d31cab0f58d0ad71ffa9ff9bd801caba7a\" size is 595258880 bytes"
time="2020-03-12T15:59:43Z" level=debug msg="OCIv1 config = {\"created\":\"2020-03-12T15:59:33.235872979Z\",\"architecture\":\"amd64\",\"os\":\"linux\",\"config\":{\"User\":\"1001\",\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"container=oci\",\"OPENSHIFT_BUILD_NAME=openjdk-devel-1\",\"OPENSHIFT_BUILD_NAMESPACE=zz-cluster-load-test-0231\",\"BUILD_LOGLEVEL=6\"],\"Cmd\":[\"/bin/bash\"],\"WorkingDir\":\"/builds\",\"Labels\":{\"architecture\":\"x86_64\",\"authoritative-source-url\":\"registry.access.redhat.com\",\"build-date\":\"2020-01-28T10:58:57.851295\",\"com.redhat.build-host\":\"cpt-1004.osbs.prod.upshift.rdu2.redhat.com\",\"com.redhat.component\":\"ubi7-minimal-container\",\"com.redhat.license_terms\":\"https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI\",\"description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"distribution-scope\":\"public\",\"io.k8s.description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"io.k8s.display-name\":\"Red Hat Universal Base Image 7 Minimal\",\"io.openshift.build.name\":\"openjdk-devel-1\",\"io.openshift.build.namespace\":\"zz-cluster-load-test-0231\",\"io.openshift.tags\":\"minimal rhel7\",\"maintainer\":\"Red Hat, Inc.\",\"name\":\"ubi7-minimal\",\"release\":\"303\",\"summary\":\"Provides the latest release of the minimal Red Hat Universal Base Image 7.\",\"url\":\"https://access.redhat.com/containers/#/registry.access.redhat.com/ubi7-minimal/images/7.7-303\",\"vcs-ref\":\"7e9dd8b61a03561c4b5f70b214bc26d19c4f25a4\",\"vcs-type\":\"git\",\"vendor\":\"Red Hat, Inc.\",\"version\":\"7.7\"}},\"rootfs\":{\"type\":\"layers\",\"diff_ids\":[\"sha256:2f4129b6dea024391198419be227846bebc4a63d94a5c276c2920a3701f41651\",\"sha256:a5d78edbc6523a89a2687ed7e2818bc15851a6fd62ac5833a4b12c2b38d07cad\",\"sha256:7772f548ccef4b2910e63af9bc31ab8324f58749436043e5c7ed1d263f0a5aa8\",\"sha256:603a0d3f7ac375cb6d28e4b4495d41524347d96730ad9451923a202ee1ba7634\",\"sha256:f63167fca0795125dcffccfd0fbd4a2d15d27ea63ff043d5497d0ff3a5fe7461\"]},\"history\":[{\"created\":\"2020-01-28T10:59:06.259857804Z\",\"comment\":\"Imported from -\"},{\"created\":\"2020-01-28T10:59:15.548872Z\"},{\"created\":\"2020-03-12T14:58:55.410578153Z\",\"created_by\":\"/bin/sh -c microdnf install which java-11-openjdk-devel \\u0026\\u0026 mkdir /builds \\u0026\\u0026 chgrp -R 0 /builds \\u0026\\u0026 chmod -R g=u /builds\"},{\"created\":\"2020-03-12T14:59:11.113871897Z\",\"created_by\":\"/bin/sh -c #(nop) USER 1001\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:11.89385222Z\",\"created_by\":\"/bin/sh -c #(nop) WORKDIR /builds\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:12.658658208Z\",\"created_by\":\"/bin/sh -c #(nop) ENV \\\"OPENSHIFT_BUILD_NAME\\\"=\\\"openjdk-devel-1\\\" \\\"OPENSHIFT_BUILD_NAMESPACE\\\"=\\\"zz-cluster-load-test-0231\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:13.436671505Z\",\"created_by\":\"/bin/sh -c #(nop) LABEL \\\"io.openshift.build.name\\\"=\\\"openjdk-devel-1\\\" \\\"io.openshift.build.namespace\\\"=\\\"zz-cluster-load-test-0231\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T15:57:15.906697248Z\",\"created_by\":\"/bin/sh -c #(nop) ENV \\\"BUILD_LOGLEVEL\\\"=\\\"6\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T15:57:18.063087881Z\",\"created_by\":\"/bin/sh -c #(nop) COPY . / ./\"},{\"created\":\"2020-03-12T15:59:33.235872979Z\",\"created_by\":\"/bin/sh -c ./gradlew build \\u0026\\u0026 mv build/libs/spring-boot-rest-example-*.jar spring-boot-rest-example.jar\"}]}"
time="2020-03-12T15:59:43Z" level=debug msg="OCIv1 manifest = {\"schemaVersion\":2,\"config\":{\"mediaType\":\"application/vnd.oci.image.config.v1+json\",\"digest\":\"sha256:be3c7079027d858d63054999e3e41654394a93184acc9abf3ecd17a4e7d4b586\",\"size\":3700},\"layers\":[{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:2f4129b6dea024391198419be227846bebc4a63d94a5c276c2920a3701f41651\",\"size\":83005440},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:a5d78edbc6523a89a2687ed7e2818bc15851a6fd62ac5833a4b12c2b38d07cad\",\"size\":20480},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:7772f548ccef4b2910e63af9bc31ab8324f58749436043e5c7ed1d263f0a5aa8\",\"size\":358737408},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:603a0d3f7ac375cb6d28e4b4495d41524347d96730ad9451923a202ee1ba7634\",\"size\":1838080},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:f63167fca0795125dcffccfd0fbd4a2d15d27ea63ff043d5497d0ff3a5fe7461\",\"size\":595258880}]}"
time="2020-03-12T15:59:43Z" level=debug msg="Docker v2s2 config = {\"created\":\"2020-03-12T15:59:33.235872979Z\",\"container_config\":{\"Hostname\":\"27dbfa3c598a\",\"Domainname\":\"\",\"User\":\"1001\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"container=oci\",\"OPENSHIFT_BUILD_NAME=openjdk-devel-1\",\"OPENSHIFT_BUILD_NAMESPACE=zz-cluster-load-test-0231\",\"BUILD_LOGLEVEL=6\"],\"Cmd\":[\"/bin/bash\"],\"ArgsEscaped\":true,\"Image\":\"5b069ae67d4e7eaea54e5001b89fbd5c076bd27f2ba04bfa87745c3c96e687a9\",\"Volumes\":{},\"WorkingDir\":\"/builds\",\"Entrypoint\":[],\"OnBuild\":[],\"Labels\":{\"architecture\":\"x86_64\",\"authoritative-source-url\":\"registry.access.redhat.com\",\"build-date\":\"2020-01-28T10:58:57.851295\",\"com.redhat.build-host\":\"cpt-1004.osbs.prod.upshift.rdu2.redhat.com\",\"com.redhat.component\":\"ubi7-minimal-container\",\"com.redhat.license_terms\":\"https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI\",\"description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"distribution-scope\":\"public\",\"io.k8s.description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"io.k8s.display-name\":\"Red Hat Universal Base Image 7 Minimal\",\"io.openshift.build.name\":\"openjdk-devel-1\",\"io.openshift.build.namespace\":\"zz-cluster-load-test-0231\",\"io.openshift.tags\":\"minimal rhel7\",\"maintainer\":\"Red Hat, Inc.\",\"name\":\"ubi7-minimal\",\"release\":\"303\",\"summary\":\"Provides the latest release of the minimal Red Hat Universal Base Image 7.\",\"url\":\"https://access.redhat.com/containers/#/registry.access.redhat.com/ubi7-minimal/images/7.7-303\",\"vcs-ref\":\"7e9dd8b61a03561c4b5f70b214bc26d19c4f25a4\",\"vcs-type\":\"git\",\"vendor\":\"Red Hat, Inc.\",\"version\":\"7.7\"}},\"config\":{\"Hostname\":\"27dbfa3c598a\",\"Domainname\":\"\",\"User\":\"1001\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"container=oci\",\"OPENSHIFT_BUILD_NAME=openjdk-devel-1\",\"OPENSHIFT_BUILD_NAMESPACE=zz-cluster-load-test-0231\",\"BUILD_LOGLEVEL=6\"],\"Cmd\":[\"/bin/bash\"],\"ArgsEscaped\":true,\"Image\":\"5b069ae67d4e7eaea54e5001b89fbd5c076bd27f2ba04bfa87745c3c96e687a9\",\"Volumes\":{},\"WorkingDir\":\"/builds\",\"Entrypoint\":[],\"OnBuild\":[],\"Labels\":{\"architecture\":\"x86_64\",\"authoritative-source-url\":\"registry.access.redhat.com\",\"build-date\":\"2020-01-28T10:58:57.851295\",\"com.redhat.build-host\":\"cpt-1004.osbs.prod.upshift.rdu2.redhat.com\",\"com.redhat.component\":\"ubi7-minimal-container\",\"com.redhat.license_terms\":\"https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI\",\"description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"distribution-scope\":\"public\",\"io.k8s.description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"io.k8s.display-name\":\"Red Hat Universal Base Image 7 Minimal\",\"io.openshift.build.name\":\"openjdk-devel-1\",\"io.openshift.build.namespace\":\"zz-cluster-load-test-0231\",\"io.openshift.tags\":\"minimal rhel7\",\"maintainer\":\"Red Hat, Inc.\",\"name\":\"ubi7-minimal\",\"release\":\"303\",\"summary\":\"Provides the latest release of the minimal Red Hat Universal Base Image 7.\",\"url\":\"https://access.redhat.com/containers/#/registry.access.redhat.com/ubi7-minimal/images/7.7-303\",\"vcs-ref\":\"7e9dd8b61a03561c4b5f70b214bc26d19c4f25a4\",\"vcs-type\":\"git\",\"vendor\":\"Red Hat, Inc.\",\"version\":\"7.7\"}},\"architecture\":\"amd64\",\"os\":\"linux\",\"parent\":\"sha256:021b56a848a78cc8571f81f8ec55f381ca0640588335be8a6dfa833acb82a43d\",\"rootfs\":{\"type\":\"layers\",\"diff_ids\":[\"sha256:2f4129b6dea024391198419be227846bebc4a63d94a5c276c2920a3701f41651\",\"sha256:a5d78edbc6523a89a2687ed7e2818bc15851a6fd62ac5833a4b12c2b38d07cad\",\"sha256:7772f548ccef4b2910e63af9bc31ab8324f58749436043e5c7ed1d263f0a5aa8\",\"sha256:603a0d3f7ac375cb6d28e4b4495d41524347d96730ad9451923a202ee1ba7634\",\"sha256:f63167fca0795125dcffccfd0fbd4a2d15d27ea63ff043d5497d0ff3a5fe7461\"]},\"history\":[{\"created\":\"2020-01-28T10:59:06.259857804Z\",\"comment\":\"Imported from -\"},{\"created\":\"2020-01-28T10:59:15.548872Z\"},{\"created\":\"2020-03-12T14:58:55.410578153Z\",\"created_by\":\"/bin/sh -c microdnf install which java-11-openjdk-devel \\u0026\\u0026 mkdir /builds \\u0026\\u0026 chgrp -R 0 /builds \\u0026\\u0026 chmod -R g=u /builds\"},{\"created\":\"2020-03-12T14:59:11.113871897Z\",\"created_by\":\"/bin/sh -c #(nop) USER 1001\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:11.89385222Z\",\"created_by\":\"/bin/sh -c #(nop) WORKDIR /builds\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:12.658658208Z\",\"created_by\":\"/bin/sh -c #(nop) ENV \\\"OPENSHIFT_BUILD_NAME\\\"=\\\"openjdk-devel-1\\\" \\\"OPENSHIFT_BUILD_NAMESPACE\\\"=\\\"zz-cluster-load-test-0231\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:13.436671505Z\",\"created_by\":\"/bin/sh -c #(nop) LABEL \\\"io.openshift.build.name\\\"=\\\"openjdk-devel-1\\\" \\\"io.openshift.build.namespace\\\"=\\\"zz-cluster-load-test-0231\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T15:57:15.906697248Z\",\"created_by\":\"/bin/sh -c #(nop) ENV \\\"BUILD_LOGLEVEL\\\"=\\\"6\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T15:57:18.063087881Z\",\"created_by\":\"/bin/sh -c #(nop) COPY . / ./\"},{\"created\":\"2020-03-12T15:59:33.235872979Z\",\"created_by\":\"/bin/sh -c ./gradlew build \\u0026\\u0026 mv build/libs/spring-boot-rest-example-*.jar spring-boot-rest-example.jar\"}]}"
time="2020-03-12T15:59:43Z" level=debug msg="Docker v2s2 manifest = {\"schemaVersion\":2,\"mediaType\":\"application/vnd.docker.distribution.manifest.v2+json\",\"config\":{\"mediaType\":\"application/vnd.docker.container.image.v1+json\",\"size\":6214,\"digest\":\"sha256:aa1a743c9b4d9390a9e3e1fbd4de7fcd88ffe7b3e1bcbbc085d1f4c6c80ebc13\"},\"layers\":[{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":83005440,\"digest\":\"sha256:2f4129b6dea024391198419be227846bebc4a63d94a5c276c2920a3701f41651\"},{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":20480,\"digest\":\"sha256:a5d78edbc6523a89a2687ed7e2818bc15851a6fd62ac5833a4b12c2b38d07cad\"},{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":358737408,\"digest\":\"sha256:7772f548ccef4b2910e63af9bc31ab8324f58749436043e5c7ed1d263f0a5aa8\"},{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":1838080,\"digest\":\"sha256:603a0d3f7ac375cb6d28e4b4495d41524347d96730ad9451923a202ee1ba7634\"},{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":595258880,\"digest\":\"sha256:f63167fca0795125dcffccfd0fbd4a2d15d27ea63ff043d5497d0ff3a5fe7461\"}]}"
time="2020-03-12T15:59:43Z" level=debug msg="starting to read from image \"containers-storage:021b56a848a78cc8571f81f8ec55f381ca0640588335be8a6dfa833acb82a43d-working-container\" using blob cache in \"/var/cache/blobs\" (compression=0)"
time="2020-03-12T15:59:43Z" level=debug msg="finished reading from image \"containers-storage:021b56a848a78cc8571f81f8ec55f381ca0640588335be8a6dfa833acb82a43d-working-container\" using blob cache: cache had 0 hits, 1 misses, 0 errors"
time="2020-03-12T15:59:43Z" level=debug msg="Skipping blob sha256:2f4129b6dea024391198419be227846bebc4a63d94a5c276c2920a3701f41651 (already present):"
time="2020-03-12T15:59:43Z" level=debug msg="Skipping blob sha256:a5d78edbc6523a89a2687ed7e2818bc15851a6fd62ac5833a4b12c2b38d07cad (already present):"
time="2020-03-12T15:59:43Z" level=debug msg="Skipping blob sha256:7772f548ccef4b2910e63af9bc31ab8324f58749436043e5c7ed1d263f0a5aa8 (already present):"
time="2020-03-12T15:59:43Z" level=debug msg="Skipping blob sha256:603a0d3f7ac375cb6d28e4b4495d41524347d96730ad9451923a202ee1ba7634 (already present):"
time="2020-03-12T15:59:43Z" level=debug msg="error reading layer \"sha256:f63167fca0795125dcffccfd0fbd4a2d15d27ea63ff043d5497d0ff3a5fe7461\": open /tmp/buildah481524101/sha256:f63167fca0795125dcffccfd0fbd4a2d15d27ea63ff043d5497d0ff3a5fe7461: no such file or directory"
time="2020-03-12T15:59:44Z" level=debug msg="finished reading from image \"containers-storage:021b56a848a78cc8571f81f8ec55f381ca0640588335be8a6dfa833acb82a43d-working-container\" using blob cache: cache had 0 hits, 3 misses, 0 errors"
time="2020-03-12T15:59:44Z" level=debug msg="finished writing to image \"containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.imagestore=/var/lib/shared]docker.io/library/40e402db80b43803dfa961ad8fbf2918e91e49ad7cc99b25a7b3d59be0c8161b-tmp:latest\" using blob cache"
I0312 15:59:45.073275       1 round_trippers.go:438] GET https://172.30.0.1:443/apis/build.openshift.io/v1/namespaces/zz-cluster-load-test-0231/builds/advanced-deployment-builder-2 200 OK in 8 milliseconds
I0312 15:59:45.083682       1 round_trippers.go:438] PUT https://172.30.0.1:443/apis/build.openshift.io/v1/namespaces/zz-cluster-load-test-0231/builds/advanced-deployment-builder-2/details 200 OK in 10 milliseconds
I0312 15:59:45.087652       1 round_trippers.go:438] GET https://172.30.0.1:443/apis/build.openshift.io/v1/namespaces/zz-cluster-load-test-0231/builds/advanced-deployment-builder-2 200 OK in 3 milliseconds
I0312 15:59:45.096922       1 round_trippers.go:438] PUT https://172.30.0.1:443/apis/build.openshift.io/v1/namespaces/zz-cluster-load-test-0231/builds/advanced-deployment-builder-2/details 200 OK in 8 milliseconds
F0312 15:59:45.151449       1 helpers.go:114] error: build error: error committing container for step {Env:[BUILD_LOGLEVEL=6 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=oci OPENSHIFT_BUILD_NAME=openjdk-devel-1 OPENSHIFT_BUILD_NAMESPACE=zz-cluster-load-test-0231  BUILD_LOGLEVEL=6] Command:run Args:[./gradlew build && mv build/libs/spring-boot-rest-example-*.jar spring-boot-rest-example.jar] Flags:[] Attrs:map[] Message:RUN ./gradlew build && mv build/libs/spring-boot-rest-example-*.jar spring-boot-rest-example.jar Original:RUN ./gradlew build && mv build/libs/spring-boot-rest-example-*.jar spring-boot-rest-example.jar}: error copying layers and metadata for container "954b93827620aafe2f6d0b77909f7382e92c6758830da0b50244281ebfd5f29e": Error reading blob sha256:f63167fca0795125dcffccfd0fbd4a2d15d27ea63ff043d5497d0ff3a5fe7461: error reading blob from source image "containers-storage:021b56a848a78cc8571f81f8ec55f381ca0640588335be8a6dfa833acb82a43d-working-container": error opening file "/tmp/buildah481524101/sha256:f63167fca0795125dcffccfd0fbd4a2d15d27ea63ff043d5497d0ff3a5fe7461" to buffer layer blob: open /tmp/buildah481524101/sha256:f63167fca0795125dcffccfd0fbd4a2d15d27ea63ff043d5497d0ff3a5fe7461: no such file or directory

Comment 26 Nalin Dahyabhai 2020-07-30 21:53:34 UTC
This looks like a bad interaction between the image committing logic and the blob caching logic.  While I can't directly reproduce the failure here, it's telling that when committing every layer, the job of generating a configuration blob and manifest for the to-be-committed image is being run twice, and each time, we cache copies of the blobs that represent any layers that aren't already in the blob cache in a per-job unique temporary directory.  In the log, between the two jobs, the content (and both the size and the digest of the content) of the newest layer is different.  We then appear to be attempting to read the cached copy of the blob for one layer from the wrong directory.  My thinking is that preventing the job from needing to be run a second time will avoid any such mismatch.

Opened https://github.com/containers/buildah/pull/2500 with a candidate patch for the library that we consume.  If/when it's merged, we'll need to update the version of the library that we use in OpenShift to fix it here.

Comment 27 David Hernández Fernández 2020-08-04 13:11:29 UTC
Nalin, the PR is already merged, let us know if you could share next steps about delivering ERRATA with new version of the library.

Comment 38 MinLi 2020-09-28 08:03:31 UTC
test many times and trigger no error like 'error reading blob from source image'
verified on version : 4.6.0-0.nightly-2020-09-26-202331

$ oc logs -f build/nodejs-ex-1 --loglevel=8
I0928 15:10:23.979101   20160 loader.go:375] Config loaded from file:  ./kubeconfig_debug
I0928 15:10:23.987991   20160 round_trippers.go:420] GET https://api.minmli092746.qe.devcluster.openshift.com:6443/apis/build.openshift.io/v1/namespaces/minmli/builds/nodejs-ex-1
I0928 15:10:23.988026   20160 round_trippers.go:427] Request Headers:
I0928 15:10:23.988045   20160 round_trippers.go:431]     Accept: application/json, */*
I0928 15:10:23.988062   20160 round_trippers.go:431]     User-Agent: oc/4.5.0 (linux/amd64) kubernetes/9933eb9
I0928 15:10:24.529402   20160 round_trippers.go:446] Response Status: 200 OK in 541 milliseconds
I0928 15:10:24.529449   20160 round_trippers.go:449] Response Headers:
I0928 15:10:24.529494   20160 round_trippers.go:452]     Content-Length: 3597
I0928 15:10:24.529541   20160 round_trippers.go:452]     Audit-Id: 8a72c055-abd3-45b9-b343-63fe35a6a95e
I0928 15:10:24.529567   20160 round_trippers.go:452]     Cache-Control: no-cache, private
I0928 15:10:24.529591   20160 round_trippers.go:452]     Cache-Control: no-store
I0928 15:10:24.529614   20160 round_trippers.go:452]     Content-Type: application/json
I0928 15:10:24.529637   20160 round_trippers.go:452]     Date: Mon, 28 Sep 2020 07:10:24 GMT
I0928 15:10:24.529659   20160 round_trippers.go:452]     X-Kubernetes-Pf-Flowschema-Uid: 5b1f1dcd-b82a-4310-a2c2-5d634c8d1281
I0928 15:10:24.529680   20160 round_trippers.go:452]     X-Kubernetes-Pf-Prioritylevel-Uid: 35be4319-ece5-40cc-891f-4bb32f85874e
I0928 15:10:24.529795   20160 request.go:1068] Response Body: {"kind":"Build","apiVersion":"build.openshift.io/v1","metadata":{"name":"nodejs-ex-1","namespace":"minmli","selfLink":"/apis/build.openshift.io/v1/namespaces/minmli/builds/nodejs-ex-1","uid":"5d9a9787-ce4f-44ca-b4ec-cf41190e466a","resourceVersion":"1387368","creationTimestamp":"2020-09-28T07:05:52Z","labels":{"app":"nodejs-ex","app.kubernetes.io/component":"nodejs-ex","app.kubernetes.io/instance":"nodejs-ex","buildconfig":"nodejs-ex","openshift.io/build-config.name":"nodejs-ex","openshift.io/build.start-policy":"Serial"},"annotations":{"openshift.io/build-config.name":"nodejs-ex","openshift.io/build.number":"1","openshift.io/build.pod-name":"nodejs-ex-1-build"},"ownerReferences":[{"apiVersion":"build.openshift.io/v1","kind":"BuildConfig","name":"nodejs-ex","uid":"e97301f6-327b-4f08-bd8d-34504912bd51","controller":true}]},"spec":{"serviceAccount":"builder","source":{"type":"Git","git":{"uri":"https://github.com/sclorg/nodejs-ex"}},"revision":{"type":"Git","git":{"commit":"7b9f57949786059a3fab03b8493279c945770f [truncated 2573 chars]
I0928 15:10:24.537644   20160 round_trippers.go:420] GET https://api.minmli092746.qe.devcluster.openshift.com:6443/apis/build.openshift.io/v1/namespaces/minmli/builds/nodejs-ex-1/log?follow=true
I0928 15:10:24.537692   20160 round_trippers.go:427] Request Headers:
I0928 15:10:24.537725   20160 round_trippers.go:431]     Accept: application/json, */*
I0928 15:10:24.537752   20160 round_trippers.go:431]     User-Agent: oc/4.5.0 (linux/amd64) kubernetes/9933eb9
I0928 15:10:24.742964   20160 round_trippers.go:446] Response Status: 200 OK in 205 milliseconds
I0928 15:10:24.743015   20160 round_trippers.go:449] Response Headers:
I0928 15:10:24.743047   20160 round_trippers.go:452]     Audit-Id: 31cf04d5-915b-4f5c-af70-8a35b81481fa
I0928 15:10:24.743074   20160 round_trippers.go:452]     Cache-Control: no-cache, private
I0928 15:10:24.743098   20160 round_trippers.go:452]     Cache-Control: no-store
I0928 15:10:24.743127   20160 round_trippers.go:452]     Content-Type: application/octet-stream
I0928 15:10:24.743152   20160 round_trippers.go:452]     Date: Mon, 28 Sep 2020 07:10:24 GMT
Cloning "https://github.com/sclorg/nodejs-ex" ...
	Commit:	7b9f57949786059a3fab03b8493279c945770fb0 (Merge pull request #249 from multi-arch/master)
	Author:	Honza Horak <hhorak@redhat.com>
	Date:	Wed Sep 23 10:52:52 2020 +0200
Caching blobs under "/var/cache/blobs".
Getting image source signatures
Copying blob sha256:c4d668e229cd131e0a8e4f8218dca628d9cf9697572875e355fe4b247b6aa9f0
Copying blob sha256:da1cc572023a942fff15d59aefa5abbb59d2c24a03966db8074ef8f9bab277d4
Copying blob sha256:ec1681b6a383e4ecedbeddd5abc596f3de835aed6db39a735f62395c8edbff30
Copying blob sha256:35ad9b4fba1fa6b00a6f266303348dc0cf9a7c341616e800c2738030c0f64167
Copying blob sha256:5e2ae0c76e83847010202c40d0c7ebac953a6c7871efdea7602b41507b3d11f5
Copying config sha256:8a961c0b3cbcc653bf39713aaf79a36d9921618e2a39fd7e5057cf70c203cf87
Writing manifest to image destination
Storing signatures
Generating dockerfile with builder image image-registry.openshift-image-registry.svc:5000/openshift/nodejs@sha256:1175006938852dff3ac21cdcab0939b65e0499a6ba0c4d1586869b3f31465f3f
STEP 1: FROM image-registry.openshift-image-registry.svc:5000/openshift/nodejs@sha256:1175006938852dff3ac21cdcab0939b65e0499a6ba0c4d1586869b3f31465f3f
STEP 2: LABEL "io.openshift.build.image"="image-registry.openshift-image-registry.svc:5000/openshift/nodejs@sha256:1175006938852dff3ac21cdcab0939b65e0499a6ba0c4d1586869b3f31465f3f"       "io.openshift.build.commit.author"="Honza Horak <hhorak@redhat.com>"       "io.openshift.build.commit.date"="Wed Sep 23 10:52:52 2020 +0200"       "io.openshift.build.commit.id"="7b9f57949786059a3fab03b8493279c945770fb0"       "io.openshift.build.commit.ref"="master"       "io.openshift.build.commit.message"="Merge pull request #249 from multi-arch/master"       "io.openshift.build.source-location"="https://github.com/sclorg/nodejs-ex"
STEP 3: ENV OPENSHIFT_BUILD_NAME="nodejs-ex-1"     OPENSHIFT_BUILD_NAMESPACE="minmli"     OPENSHIFT_BUILD_SOURCE="https://github.com/sclorg/nodejs-ex"     OPENSHIFT_BUILD_COMMIT="7b9f57949786059a3fab03b8493279c945770fb0"
STEP 4: USER root
STEP 5: COPY upload/src /tmp/src
STEP 6: RUN chown -R 1001:0 /tmp/src
STEP 7: USER 1001
STEP 8: RUN /usr/libexec/s2i/assemble
---> Installing application source ...
---> Installing all dependencies
npm WARN deprecated jade@0.26.3: Jade has been renamed to pug, please install the latest version of pug instead of jade
npm WARN deprecated to-iso-string@0.0.2: to-iso-string has been deprecated, use @segment/to-iso-string instead.
npm WARN deprecated mkdirp@0.5.1: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
npm WARN deprecated bson@1.0.9: Fixed a critical issue with BSON serialization documented in CVE-2019-2391, see https://bit.ly/2KcpXdo for more details
npm WARN deprecated mkdirp@0.3.0: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
npm WARN deprecated minimatch@0.3.0: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue

> ejs@2.7.4 postinstall /opt/app-root/src/node_modules/ejs
> node ./postinstall.js

Thank you for installing EJS: built with the Jake JavaScript build tool (https://jakejs.com/)

npm notice created a lockfile as package-lock.json. You should commit this file.
added 121 packages from 342 contributors and audited 121 packages in 4.015s
found 10 vulnerabilities (3 low, 2 moderate, 4 high, 1 critical)
  run `npm audit fix` to fix them, or `npm audit` for details
---> Building in production mode
---> Pruning the development dependencies
audited 121 packages in 0.917s
found 10 vulnerabilities (3 low, 2 moderate, 4 high, 1 critical)
  run `npm audit fix` to fix them, or `npm audit` for details
/opt/app-root/src/.npm is not a mountpoint
---> Cleaning the npm cache /opt/app-root/src/.npm
/tmp is not a mountpoint
---> Cleaning the /tmp/npm-*
STEP 9: CMD /usr/libexec/s2i/run
STEP 10: COMMIT temp.builder.openshift.io/minmli/nodejs-ex-1:9283bd5b
Getting image source signatures
Copying blob sha256:ccf04fbd6e1943f648d1c2980e96038edc02b543c597556098ab2bcaa4fd1fa8
Copying blob sha256:b7b591e3443f17f9d8272b8d118b6c031ca826deb09d4b44f296ba934f1b6e57
Copying blob sha256:511ade7a7dffb350f0e8b72e1a36ed9f1507b5a5caeafe30721d5d1c8b33f1ff
Copying blob sha256:ffdb6cc9bdbbb5e11c5dd0aad9f4eacf35f3adcf5c6df4f4d2c3120cee96dd55
Copying blob sha256:8e38d704f1b812ffefa50f91815ff0f509e72ffc9d3bdbadc8c2b05d2eed1bfc
Copying blob sha256:0e02af8fcd74b7cdff64349d28ee3ed53781d7e546bbae86944aef924cc6faff
Copying config sha256:0ffc6c4be37e02a5aec693743633c58114e097d18b6c4e042837b759fae71949
Writing manifest to image destination
Storing signatures
--> 0ffc6c4be37
0ffc6c4be37e02a5aec693743633c58114e097d18b6c4e042837b759fae71949
Getting image source signatures

Pushing image image-registry.openshift-image-registry.svc:5000/minmli/nodejs-ex:latest ...
Copying blob sha256:0e02af8fcd74b7cdff64349d28ee3ed53781d7e546bbae86944aef924cc6faff
Copying blob sha256:c4d668e229cd131e0a8e4f8218dca628d9cf9697572875e355fe4b247b6aa9f0
Copying blob sha256:35ad9b4fba1fa6b00a6f266303348dc0cf9a7c341616e800c2738030c0f64167
Copying blob sha256:ec1681b6a383e4ecedbeddd5abc596f3de835aed6db39a735f62395c8edbff30
Copying blob sha256:5e2ae0c76e83847010202c40d0c7ebac953a6c7871efdea7602b41507b3d11f5
Copying blob sha256:da1cc572023a942fff15d59aefa5abbb59d2c24a03966db8074ef8f9bab277d4
Copying config sha256:0ffc6c4be37e02a5aec693743633c58114e097d18b6c4e042837b759fae71949
Writing manifest to image destination
Storing signatures
Successfully pushed image-registry.openshift-image-registry.svc:5000/minmli/nodejs-ex@sha256:2d2dafe0a53e304e5bf6adb918e5446c54f924c3a07cf3c8d7e7eaf8f4a74294
Push successful

Comment 40 errata-xmlrpc 2020-10-27 15:54:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196

Comment 41 Oscar Casal Sanchez 2020-10-28 14:37:24 UTC
Hello,

I can see that it was released for OCP 4.6, any change of being this backported to 4.5 or exists a bug for being backported to 4.5?

Thanks in advance,
Oscar

Comment 42 Tom Sweeney 2020-10-28 21:47:36 UTC
Adam, Nalin, would it be possible to backport this to OCP 4.5?

Comment 43 Adam Kaplan 2020-10-29 12:33:43 UTC
This was fixed in buildah v1.14.11, which was included in to 4.5.8 update as part of the fix for Bug #1868401. I'll make the clone BZs to make it clear that this was also fixed in 4.5.z.

https://access.redhat.com/errata/RHBA-2020:3510


Note You need to log in before you can comment on or make changes to this bug.