Bug 1720730

@Nalin - based on the logs, looks like a layer got deleted from underneath buildah...any idea how this could happen?

@adam It's not something we expect, but if there was a problem generating the layer blob file, I'd also have expected an error message.  Any chance we can get a log from a reproducing run with BUILD_LOGLEVEL=6, so that we can more easily follow what's going on during that COMMIT step?

@Justin can you try `oc new-app --loglevel=6 https://github.com/sclorg/nodejs-ex`?

Targeting 4.3 as it is not likely we'll fix this in time for 4.2

I've increased the loglevel in our monitoring code. If it happens again, we will update the BZ.

*** Bug 1770557 has been marked as a duplicate of this bug. ***

We have been having this issue in 4.2 clusters:

time="2020-03-12T15:59:33Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah416993853/mnt/rootfs\": error checking if \"/tmp/buildah416993853/mnt/rootfs/sys/fs/cgroup/hugetlb\" is mounted: no such file or directory"
time="2020-03-12T15:59:33Z" level=debug msg="error cleaning up intermediate chroot bind mounts: <nil>"
time="2020-03-12T15:59:33Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah416993853/mnt/rootfs\": error checking if \"/tmp/buildah416993853/mnt/rootfs/sys/fs/cgroup/hugetlb\" is mounted: no such file or directory"
time="2020-03-12T15:59:33Z" level=debug msg="error cleaning up intermediate mount NS: <nil>"
time="2020-03-12T15:59:33Z" level=debug msg=COMMIT
time="2020-03-12T15:59:33Z" level=debug msg="parsed reference into \"[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.imagestore=/var/lib/shared]docker.io/library/40e402db80b43803dfa961ad8fbf2918e91e49ad7cc99b25a7b3d59be0c8161b-tmp:latest\""
time="2020-03-12T15:59:33Z" level=debug msg="parsed reference into \"[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.imagestore=/var/lib/shared]@021b56a848a78cc8571f81f8ec55f381ca0640588335be8a6dfa833acb82a43d\""
time="2020-03-12T15:59:33Z" level=debug msg="base image \"021b56a848a78cc8571f81f8ec55f381ca0640588335be8a6dfa833acb82a43d\" is already present in local storage, no need to copy its layers"
time="2020-03-12T15:59:33Z" level=debug msg="starting to write to image \"containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.imagestore=/var/lib/shared]docker.io/library/40e402db80b43803dfa961ad8fbf2918e91e49ad7cc99b25a7b3d59be0c8161b-tmp:latest\" using blob cache in \"/var/cache/blobs\""
time="2020-03-12T15:59:33Z" level=debug msg="layer list: [\"2f4129b6dea024391198419be227846bebc4a63d94a5c276c2920a3701f41651\" \"005e12f2f5ea5e2173bf1d412c05792030fbf26c83de504765642fcd1327132f\" \"70b473ffac7a6e0ffc93e96946096851f0ade78b662176be1b4341df3331795c\" \"43ec515f3a59344a3872f435bcafa94b2e5351fcc8cf7f3eff67f12f36491002\" \"14aeefdbf4cb8900aa9bfc76142942d31cab0f58d0ad71ffa9ff9bd801caba7a\"]"
time="2020-03-12T15:59:33Z" level=debug msg="using \"/tmp/buildah481524101\" to hold temporary data"
time="2020-03-12T15:59:33Z" level=debug msg="Tar with options on /var/lib/containers/storage/overlay/14aeefdbf4cb8900aa9bfc76142942d31cab0f58d0ad71ffa9ff9bd801caba7a/diff"
time="2020-03-12T15:59:38Z" level=debug msg="layer \"14aeefdbf4cb8900aa9bfc76142942d31cab0f58d0ad71ffa9ff9bd801caba7a\" size is 595258368 bytes"
time="2020-03-12T15:59:38Z" level=debug msg="OCIv1 config = {\"created\":\"2020-03-12T15:59:33.235872979Z\",\"architecture\":\"amd64\",\"os\":\"linux\",\"config\":{\"User\":\"1001\",\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"container=oci\",\"OPENSHIFT_BUILD_NAME=openjdk-devel-1\",\"OPENSHIFT_BUILD_NAMESPACE=zz-cluster-load-test-0231\",\"BUILD_LOGLEVEL=6\"],\"Cmd\":[\"/bin/bash\"],\"WorkingDir\":\"/builds\",\"Labels\":{\"architecture\":\"x86_64\",\"authoritative-source-url\":\"registry.access.redhat.com\",\"build-date\":\"2020-01-28T10:58:57.851295\",\"com.redhat.build-host\":\"cpt-1004.osbs.prod.upshift.rdu2.redhat.com\",\"com.redhat.component\":\"ubi7-minimal-container\",\"com.redhat.license_terms\":\"https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI\",\"description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"distribution-scope\":\"public\",\"io.k8s.description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"io.k8s.display-name\":\"Red Hat Universal Base Image 7 Minimal\",\"io.openshift.build.name\":\"openjdk-devel-1\",\"io.openshift.build.namespace\":\"zz-cluster-load-test-0231\",\"io.openshift.tags\":\"minimal rhel7\",\"maintainer\":\"Red Hat, Inc.\",\"name\":\"ubi7-minimal\",\"release\":\"303\",\"summary\":\"Provides the latest release of the minimal Red Hat Universal Base Image 7.\",\"url\":\"https://access.redhat.com/containers/#/registry.access.redhat.com/ubi7-minimal/images/7.7-303\",\"vcs-ref\":\"7e9dd8b61a03561c4b5f70b214bc26d19c4f25a4\",\"vcs-type\":\"git\",\"vendor\":\"Red Hat, Inc.\",\"version\":\"7.7\"}},\"rootfs\":{\"type\":\"layers\",\"diff_ids\":[\"sha256:2f4129b6dea024391198419be227846bebc4a63d94a5c276c2920a3701f41651\",\"sha256:a5d78edbc6523a89a2687ed7e2818bc15851a6fd62ac5833a4b12c2b38d07cad\",\"sha256:7772f548ccef4b2910e63af9bc31ab8324f58749436043e5c7ed1d263f0a5aa8\",\"sha256:603a0d3f7ac375cb6d28e4b4495d41524347d96730ad9451923a202ee1ba7634\",\"sha256:6d6ef9321d9777d2be90d5e747f19554f13e29d08bb558daf6e00a644b15f12f\"]},\"history\":[{\"created\":\"2020-01-28T10:59:06.259857804Z\",\"comment\":\"Imported from -\"},{\"created\":\"2020-01-28T10:59:15.548872Z\"},{\"created\":\"2020-03-12T14:58:55.410578153Z\",\"created_by\":\"/bin/sh -c microdnf install which java-11-openjdk-devel \\u0026\\u0026 mkdir /builds \\u0026\\u0026 chgrp -R 0 /builds \\u0026\\u0026 chmod -R g=u /builds\"},{\"created\":\"2020-03-12T14:59:11.113871897Z\",\"created_by\":\"/bin/sh -c #(nop) USER 1001\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:11.89385222Z\",\"created_by\":\"/bin/sh -c #(nop) WORKDIR /builds\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:12.658658208Z\",\"created_by\":\"/bin/sh -c #(nop) ENV \\\"OPENSHIFT_BUILD_NAME\\\"=\\\"openjdk-devel-1\\\" \\\"OPENSHIFT_BUILD_NAMESPACE\\\"=\\\"zz-cluster-load-test-0231\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:13.436671505Z\",\"created_by\":\"/bin/sh -c #(nop) LABEL \\\"io.openshift.build.name\\\"=\\\"openjdk-devel-1\\\" \\\"io.openshift.build.namespace\\\"=\\\"zz-cluster-load-test-0231\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T15:57:15.906697248Z\",\"created_by\":\"/bin/sh -c #(nop) ENV \\\"BUILD_LOGLEVEL\\\"=\\\"6\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T15:57:18.063087881Z\",\"created_by\":\"/bin/sh -c #(nop) COPY . / ./\"},{\"created\":\"2020-03-12T15:59:33.235872979Z\",\"created_by\":\"/bin/sh -c ./gradlew build \\u0026\\u0026 mv build/libs/spring-boot-rest-example-*.jar spring-boot-rest-example.jar\"}]}"
time="2020-03-12T15:59:38Z" level=debug msg="OCIv1 manifest = {\"schemaVersion\":2,\"config\":{\"mediaType\":\"application/vnd.oci.image.config.v1+json\",\"digest\":\"sha256:ea126b0f7200880c6ece6817e92dca1334eab853edd79f349ae072b9af3f929b\",\"size\":3700},\"layers\":[{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:2f4129b6dea024391198419be227846bebc4a63d94a5c276c2920a3701f41651\",\"size\":83005440},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:a5d78edbc6523a89a2687ed7e2818bc15851a6fd62ac5833a4b12c2b38d07cad\",\"size\":20480},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:7772f548ccef4b2910e63af9bc31ab8324f58749436043e5c7ed1d263f0a5aa8\",\"size\":358737408},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:603a0d3f7ac375cb6d28e4b4495d41524347d96730ad9451923a202ee1ba7634\",\"size\":1838080},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:6d6ef9321d9777d2be90d5e747f19554f13e29d08bb558daf6e00a644b15f12f\",\"size\":595258368}]}"
time="2020-03-12T15:59:38Z" level=debug msg="Docker v2s2 config = {\"created\":\"2020-03-12T15:59:33.235872979Z\",\"container_config\":{\"Hostname\":\"27dbfa3c598a\",\"Domainname\":\"\",\"User\":\"1001\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"container=oci\",\"OPENSHIFT_BUILD_NAME=openjdk-devel-1\",\"OPENSHIFT_BUILD_NAMESPACE=zz-cluster-load-test-0231\",\"BUILD_LOGLEVEL=6\"],\"Cmd\":[\"/bin/bash\"],\"ArgsEscaped\":true,\"Image\":\"5b069ae67d4e7eaea54e5001b89fbd5c076bd27f2ba04bfa87745c3c96e687a9\",\"Volumes\":{},\"WorkingDir\":\"/builds\",\"Entrypoint\":[],\"OnBuild\":[],\"Labels\":{\"architecture\":\"x86_64\",\"authoritative-source-url\":\"registry.access.redhat.com\",\"build-date\":\"2020-01-28T10:58:57.851295\",\"com.redhat.build-host\":\"cpt-1004.osbs.prod.upshift.rdu2.redhat.com\",\"com.redhat.component\":\"ubi7-minimal-container\",\"com.redhat.license_terms\":\"https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI\",\"description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"distribution-scope\":\"public\",\"io.k8s.description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"io.k8s.display-name\":\"Red Hat Universal Base Image 7 Minimal\",\"io.openshift.build.name\":\"openjdk-devel-1\",\"io.openshift.build.namespace\":\"zz-cluster-load-test-0231\",\"io.openshift.tags\":\"minimal rhel7\",\"maintainer\":\"Red Hat, Inc.\",\"name\":\"ubi7-minimal\",\"release\":\"303\",\"summary\":\"Provides the latest release of the minimal Red Hat Universal Base Image 7.\",\"url\":\"https://access.redhat.com/containers/#/registry.access.redhat.com/ubi7-minimal/images/7.7-303\",\"vcs-ref\":\"7e9dd8b61a03561c4b5f70b214bc26d19c4f25a4\",\"vcs-type\":\"git\",\"vendor\":\"Red Hat, Inc.\",\"version\":\"7.7\"}},\"config\":{\"Hostname\":\"27dbfa3c598a\",\"Domainname\":\"\",\"User\":\"1001\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"container=oci\",\"OPENSHIFT_BUILD_NAME=openjdk-devel-1\",\"OPENSHIFT_BUILD_NAMESPACE=zz-cluster-load-test-0231\",\"BUILD_LOGLEVEL=6\"],\"Cmd\":[\"/bin/bash\"],\"ArgsEscaped\":true,\"Image\":\"5b069ae67d4e7eaea54e5001b89fbd5c076bd27f2ba04bfa87745c3c96e687a9\",\"Volumes\":{},\"WorkingDir\":\"/builds\",\"Entrypoint\":[],\"OnBuild\":[],\"Labels\":{\"architecture\":\"x86_64\",\"authoritative-source-url\":\"registry.access.redhat.com\",\"build-date\":\"2020-01-28T10:58:57.851295\",\"com.redhat.build-host\":\"cpt-1004.osbs.prod.upshift.rdu2.redhat.com\",\"com.redhat.component\":\"ubi7-minimal-container\",\"com.redhat.license_terms\":\"https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI\",\"description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"distribution-scope\":\"public\",\"io.k8s.description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"io.k8s.display-name\":\"Red Hat Universal Base Image 7 Minimal\",\"io.openshift.build.name\":\"openjdk-devel-1\",\"io.openshift.build.namespace\":\"zz-cluster-load-test-0231\",\"io.openshift.tags\":\"minimal rhel7\",\"maintainer\":\"Red Hat, Inc.\",\"name\":\"ubi7-minimal\",\"release\":\"303\",\"summary\":\"Provides the latest release of the minimal Red Hat Universal Base Image 7.\",\"url\":\"https://access.redhat.com/containers/#/registry.access.redhat.com/ubi7-minimal/images/7.7-303\",\"vcs-ref\":\"7e9dd8b61a03561c4b5f70b214bc26d19c4f25a4\",\"vcs-type\":\"git\",\"vendor\":\"Red Hat, Inc.\",\"version\":\"7.7\"}},\"architecture\":\"amd64\",\"os\":\"linux\",\"parent\":\"sha256:021b56a848a78cc8571f81f8ec55f381ca0640588335be8a6dfa833acb82a43d\",\"rootfs\":{\"type\":\"layers\",\"diff_ids\":[\"sha256:2f4129b6dea024391198419be227846bebc4a63d94a5c276c2920a3701f41651\",\"sha256:a5d78edbc6523a89a2687ed7e2818bc15851a6fd62ac5833a4b12c2b38d07cad\",\"sha256:7772f548ccef4b2910e63af9bc31ab8324f58749436043e5c7ed1d263f0a5aa8\",\"sha256:603a0d3f7ac375cb6d28e4b4495d41524347d96730ad9451923a202ee1ba7634\",\"sha256:6d6ef9321d9777d2be90d5e747f19554f13e29d08bb558daf6e00a644b15f12f\"]},\"history\":[{\"created\":\"2020-01-28T10:59:06.259857804Z\",\"comment\":\"Imported from -\"},{\"created\":\"2020-01-28T10:59:15.548872Z\"},{\"created\":\"2020-03-12T14:58:55.410578153Z\",\"created_by\":\"/bin/sh -c microdnf install which java-11-openjdk-devel \\u0026\\u0026 mkdir /builds \\u0026\\u0026 chgrp -R 0 /builds \\u0026\\u0026 chmod -R g=u /builds\"},{\"created\":\"2020-03-12T14:59:11.113871897Z\",\"created_by\":\"/bin/sh -c #(nop) USER 1001\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:11.89385222Z\",\"created_by\":\"/bin/sh -c #(nop) WORKDIR /builds\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:12.658658208Z\",\"created_by\":\"/bin/sh -c #(nop) ENV \\\"OPENSHIFT_BUILD_NAME\\\"=\\\"openjdk-devel-1\\\" \\\"OPENSHIFT_BUILD_NAMESPACE\\\"=\\\"zz-cluster-load-test-0231\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:13.436671505Z\",\"created_by\":\"/bin/sh -c #(nop) LABEL \\\"io.openshift.build.name\\\"=\\\"openjdk-devel-1\\\" \\\"io.openshift.build.namespace\\\"=\\\"zz-cluster-load-test-0231\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T15:57:15.906697248Z\",\"created_by\":\"/bin/sh -c #(nop) ENV \\\"BUILD_LOGLEVEL\\\"=\\\"6\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T15:57:18.063087881Z\",\"created_by\":\"/bin/sh -c #(nop) COPY . / ./\"},{\"created\":\"2020-03-12T15:59:33.235872979Z\",\"created_by\":\"/bin/sh -c ./gradlew build \\u0026\\u0026 mv build/libs/spring-boot-rest-example-*.jar spring-boot-rest-example.jar\"}]}"
time="2020-03-12T15:59:38Z" level=debug msg="Docker v2s2 manifest = {\"schemaVersion\":2,\"mediaType\":\"application/vnd.docker.distribution.manifest.v2+json\",\"config\":{\"mediaType\":\"application/vnd.docker.container.image.v1+json\",\"size\":6214,\"digest\":\"sha256:2c12cedbbd68807c0864eb3f97e8267b3dba63128333e0831dcb58247b14151d\"},\"layers\":[{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":83005440,\"digest\":\"sha256:2f4129b6dea024391198419be227846bebc4a63d94a5c276c2920a3701f41651\"},{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":20480,\"digest\":\"sha256:a5d78edbc6523a89a2687ed7e2818bc15851a6fd62ac5833a4b12c2b38d07cad\"},{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":358737408,\"digest\":\"sha256:7772f548ccef4b2910e63af9bc31ab8324f58749436043e5c7ed1d263f0a5aa8\"},{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":1838080,\"digest\":\"sha256:603a0d3f7ac375cb6d28e4b4495d41524347d96730ad9451923a202ee1ba7634\"},{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":595258368,\"digest\":\"sha256:6d6ef9321d9777d2be90d5e747f19554f13e29d08bb558daf6e00a644b15f12f\"}]}"
time="2020-03-12T15:59:38Z" level=debug msg="starting to read from image \"containers-storage:021b56a848a78cc8571f81f8ec55f381ca0640588335be8a6dfa833acb82a43d-working-container\" using blob cache in \"/var/cache/blobs\" (compression=0)"
time="2020-03-12T15:59:38Z" level=debug msg="Using blob info cache at /var/lib/containers/storage/cache/blob-info-cache-v1.boltdb"
time="2020-03-12T15:59:38Z" level=debug msg="IsRunningImageAllowed for image containers-storage:"
time="2020-03-12T15:59:38Z" level=debug msg=" Using transport \"containers-storage\" policy section "
time="2020-03-12T15:59:38Z" level=debug msg=" Requirement 0: allowed"
time="2020-03-12T15:59:38Z" level=debug msg="Overall: allowed"
time="2020-03-12T15:59:38Z" level=debug msg="start reading config"
time="2020-03-12T15:59:38Z" level=debug msg="finished reading config"
time="2020-03-12T15:59:38Z" level=debug msg="Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json]"
time="2020-03-12T15:59:38Z" level=debug msg="... will first try using the original manifest unmodified"
time="2020-03-12T15:59:38Z" level=debug msg="layer list: [\"2f4129b6dea024391198419be227846bebc4a63d94a5c276c2920a3701f41651\" \"005e12f2f5ea5e2173bf1d412c05792030fbf26c83de504765642fcd1327132f\" \"70b473ffac7a6e0ffc93e96946096851f0ade78b662176be1b4341df3331795c\" \"43ec515f3a59344a3872f435bcafa94b2e5351fcc8cf7f3eff67f12f36491002\" \"14aeefdbf4cb8900aa9bfc76142942d31cab0f58d0ad71ffa9ff9bd801caba7a\"]"
time="2020-03-12T15:59:38Z" level=debug msg="using \"/tmp/buildah301210912\" to hold temporary data"
time="2020-03-12T15:59:38Z" level=debug msg="Tar with options on /var/lib/containers/storage/overlay/14aeefdbf4cb8900aa9bfc76142942d31cab0f58d0ad71ffa9ff9bd801caba7a/diff"
time="2020-03-12T15:59:43Z" level=debug msg="layer \"14aeefdbf4cb8900aa9bfc76142942d31cab0f58d0ad71ffa9ff9bd801caba7a\" size is 595258880 bytes"
time="2020-03-12T15:59:43Z" level=debug msg="OCIv1 config = {\"created\":\"2020-03-12T15:59:33.235872979Z\",\"architecture\":\"amd64\",\"os\":\"linux\",\"config\":{\"User\":\"1001\",\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"container=oci\",\"OPENSHIFT_BUILD_NAME=openjdk-devel-1\",\"OPENSHIFT_BUILD_NAMESPACE=zz-cluster-load-test-0231\",\"BUILD_LOGLEVEL=6\"],\"Cmd\":[\"/bin/bash\"],\"WorkingDir\":\"/builds\",\"Labels\":{\"architecture\":\"x86_64\",\"authoritative-source-url\":\"registry.access.redhat.com\",\"build-date\":\"2020-01-28T10:58:57.851295\",\"com.redhat.build-host\":\"cpt-1004.osbs.prod.upshift.rdu2.redhat.com\",\"com.redhat.component\":\"ubi7-minimal-container\",\"com.redhat.license_terms\":\"https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI\",\"description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"distribution-scope\":\"public\",\"io.k8s.description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"io.k8s.display-name\":\"Red Hat Universal Base Image 7 Minimal\",\"io.openshift.build.name\":\"openjdk-devel-1\",\"io.openshift.build.namespace\":\"zz-cluster-load-test-0231\",\"io.openshift.tags\":\"minimal rhel7\",\"maintainer\":\"Red Hat, Inc.\",\"name\":\"ubi7-minimal\",\"release\":\"303\",\"summary\":\"Provides the latest release of the minimal Red Hat Universal Base Image 7.\",\"url\":\"https://access.redhat.com/containers/#/registry.access.redhat.com/ubi7-minimal/images/7.7-303\",\"vcs-ref\":\"7e9dd8b61a03561c4b5f70b214bc26d19c4f25a4\",\"vcs-type\":\"git\",\"vendor\":\"Red Hat, Inc.\",\"version\":\"7.7\"}},\"rootfs\":{\"type\":\"layers\",\"diff_ids\":[\"sha256:2f4129b6dea024391198419be227846bebc4a63d94a5c276c2920a3701f41651\",\"sha256:a5d78edbc6523a89a2687ed7e2818bc15851a6fd62ac5833a4b12c2b38d07cad\",\"sha256:7772f548ccef4b2910e63af9bc31ab8324f58749436043e5c7ed1d263f0a5aa8\",\"sha256:603a0d3f7ac375cb6d28e4b4495d41524347d96730ad9451923a202ee1ba7634\",\"sha256:f63167fca0795125dcffccfd0fbd4a2d15d27ea63ff043d5497d0ff3a5fe7461\"]},\"history\":[{\"created\":\"2020-01-28T10:59:06.259857804Z\",\"comment\":\"Imported from -\"},{\"created\":\"2020-01-28T10:59:15.548872Z\"},{\"created\":\"2020-03-12T14:58:55.410578153Z\",\"created_by\":\"/bin/sh -c microdnf install which java-11-openjdk-devel \\u0026\\u0026 mkdir /builds \\u0026\\u0026 chgrp -R 0 /builds \\u0026\\u0026 chmod -R g=u /builds\"},{\"created\":\"2020-03-12T14:59:11.113871897Z\",\"created_by\":\"/bin/sh -c #(nop) USER 1001\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:11.89385222Z\",\"created_by\":\"/bin/sh -c #(nop) WORKDIR /builds\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:12.658658208Z\",\"created_by\":\"/bin/sh -c #(nop) ENV \\\"OPENSHIFT_BUILD_NAME\\\"=\\\"openjdk-devel-1\\\" \\\"OPENSHIFT_BUILD_NAMESPACE\\\"=\\\"zz-cluster-load-test-0231\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:13.436671505Z\",\"created_by\":\"/bin/sh -c #(nop) LABEL \\\"io.openshift.build.name\\\"=\\\"openjdk-devel-1\\\" \\\"io.openshift.build.namespace\\\"=\\\"zz-cluster-load-test-0231\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T15:57:15.906697248Z\",\"created_by\":\"/bin/sh -c #(nop) ENV \\\"BUILD_LOGLEVEL\\\"=\\\"6\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T15:57:18.063087881Z\",\"created_by\":\"/bin/sh -c #(nop) COPY . / ./\"},{\"created\":\"2020-03-12T15:59:33.235872979Z\",\"created_by\":\"/bin/sh -c ./gradlew build \\u0026\\u0026 mv build/libs/spring-boot-rest-example-*.jar spring-boot-rest-example.jar\"}]}"
time="2020-03-12T15:59:43Z" level=debug msg="OCIv1 manifest = {\"schemaVersion\":2,\"config\":{\"mediaType\":\"application/vnd.oci.image.config.v1+json\",\"digest\":\"sha256:be3c7079027d858d63054999e3e41654394a93184acc9abf3ecd17a4e7d4b586\",\"size\":3700},\"layers\":[{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:2f4129b6dea024391198419be227846bebc4a63d94a5c276c2920a3701f41651\",\"size\":83005440},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:a5d78edbc6523a89a2687ed7e2818bc15851a6fd62ac5833a4b12c2b38d07cad\",\"size\":20480},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:7772f548ccef4b2910e63af9bc31ab8324f58749436043e5c7ed1d263f0a5aa8\",\"size\":358737408},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:603a0d3f7ac375cb6d28e4b4495d41524347d96730ad9451923a202ee1ba7634\",\"size\":1838080},{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar\",\"digest\":\"sha256:f63167fca0795125dcffccfd0fbd4a2d15d27ea63ff043d5497d0ff3a5fe7461\",\"size\":595258880}]}"
time="2020-03-12T15:59:43Z" level=debug msg="Docker v2s2 config = {\"created\":\"2020-03-12T15:59:33.235872979Z\",\"container_config\":{\"Hostname\":\"27dbfa3c598a\",\"Domainname\":\"\",\"User\":\"1001\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"container=oci\",\"OPENSHIFT_BUILD_NAME=openjdk-devel-1\",\"OPENSHIFT_BUILD_NAMESPACE=zz-cluster-load-test-0231\",\"BUILD_LOGLEVEL=6\"],\"Cmd\":[\"/bin/bash\"],\"ArgsEscaped\":true,\"Image\":\"5b069ae67d4e7eaea54e5001b89fbd5c076bd27f2ba04bfa87745c3c96e687a9\",\"Volumes\":{},\"WorkingDir\":\"/builds\",\"Entrypoint\":[],\"OnBuild\":[],\"Labels\":{\"architecture\":\"x86_64\",\"authoritative-source-url\":\"registry.access.redhat.com\",\"build-date\":\"2020-01-28T10:58:57.851295\",\"com.redhat.build-host\":\"cpt-1004.osbs.prod.upshift.rdu2.redhat.com\",\"com.redhat.component\":\"ubi7-minimal-container\",\"com.redhat.license_terms\":\"https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI\",\"description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"distribution-scope\":\"public\",\"io.k8s.description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"io.k8s.display-name\":\"Red Hat Universal Base Image 7 Minimal\",\"io.openshift.build.name\":\"openjdk-devel-1\",\"io.openshift.build.namespace\":\"zz-cluster-load-test-0231\",\"io.openshift.tags\":\"minimal rhel7\",\"maintainer\":\"Red Hat, Inc.\",\"name\":\"ubi7-minimal\",\"release\":\"303\",\"summary\":\"Provides the latest release of the minimal Red Hat Universal Base Image 7.\",\"url\":\"https://access.redhat.com/containers/#/registry.access.redhat.com/ubi7-minimal/images/7.7-303\",\"vcs-ref\":\"7e9dd8b61a03561c4b5f70b214bc26d19c4f25a4\",\"vcs-type\":\"git\",\"vendor\":\"Red Hat, Inc.\",\"version\":\"7.7\"}},\"config\":{\"Hostname\":\"27dbfa3c598a\",\"Domainname\":\"\",\"User\":\"1001\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"container=oci\",\"OPENSHIFT_BUILD_NAME=openjdk-devel-1\",\"OPENSHIFT_BUILD_NAMESPACE=zz-cluster-load-test-0231\",\"BUILD_LOGLEVEL=6\"],\"Cmd\":[\"/bin/bash\"],\"ArgsEscaped\":true,\"Image\":\"5b069ae67d4e7eaea54e5001b89fbd5c076bd27f2ba04bfa87745c3c96e687a9\",\"Volumes\":{},\"WorkingDir\":\"/builds\",\"Entrypoint\":[],\"OnBuild\":[],\"Labels\":{\"architecture\":\"x86_64\",\"authoritative-source-url\":\"registry.access.redhat.com\",\"build-date\":\"2020-01-28T10:58:57.851295\",\"com.redhat.build-host\":\"cpt-1004.osbs.prod.upshift.rdu2.redhat.com\",\"com.redhat.component\":\"ubi7-minimal-container\",\"com.redhat.license_terms\":\"https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI\",\"description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"distribution-scope\":\"public\",\"io.k8s.description\":\"The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.\",\"io.k8s.display-name\":\"Red Hat Universal Base Image 7 Minimal\",\"io.openshift.build.name\":\"openjdk-devel-1\",\"io.openshift.build.namespace\":\"zz-cluster-load-test-0231\",\"io.openshift.tags\":\"minimal rhel7\",\"maintainer\":\"Red Hat, Inc.\",\"name\":\"ubi7-minimal\",\"release\":\"303\",\"summary\":\"Provides the latest release of the minimal Red Hat Universal Base Image 7.\",\"url\":\"https://access.redhat.com/containers/#/registry.access.redhat.com/ubi7-minimal/images/7.7-303\",\"vcs-ref\":\"7e9dd8b61a03561c4b5f70b214bc26d19c4f25a4\",\"vcs-type\":\"git\",\"vendor\":\"Red Hat, Inc.\",\"version\":\"7.7\"}},\"architecture\":\"amd64\",\"os\":\"linux\",\"parent\":\"sha256:021b56a848a78cc8571f81f8ec55f381ca0640588335be8a6dfa833acb82a43d\",\"rootfs\":{\"type\":\"layers\",\"diff_ids\":[\"sha256:2f4129b6dea024391198419be227846bebc4a63d94a5c276c2920a3701f41651\",\"sha256:a5d78edbc6523a89a2687ed7e2818bc15851a6fd62ac5833a4b12c2b38d07cad\",\"sha256:7772f548ccef4b2910e63af9bc31ab8324f58749436043e5c7ed1d263f0a5aa8\",\"sha256:603a0d3f7ac375cb6d28e4b4495d41524347d96730ad9451923a202ee1ba7634\",\"sha256:f63167fca0795125dcffccfd0fbd4a2d15d27ea63ff043d5497d0ff3a5fe7461\"]},\"history\":[{\"created\":\"2020-01-28T10:59:06.259857804Z\",\"comment\":\"Imported from -\"},{\"created\":\"2020-01-28T10:59:15.548872Z\"},{\"created\":\"2020-03-12T14:58:55.410578153Z\",\"created_by\":\"/bin/sh -c microdnf install which java-11-openjdk-devel \\u0026\\u0026 mkdir /builds \\u0026\\u0026 chgrp -R 0 /builds \\u0026\\u0026 chmod -R g=u /builds\"},{\"created\":\"2020-03-12T14:59:11.113871897Z\",\"created_by\":\"/bin/sh -c #(nop) USER 1001\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:11.89385222Z\",\"created_by\":\"/bin/sh -c #(nop) WORKDIR /builds\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:12.658658208Z\",\"created_by\":\"/bin/sh -c #(nop) ENV \\\"OPENSHIFT_BUILD_NAME\\\"=\\\"openjdk-devel-1\\\" \\\"OPENSHIFT_BUILD_NAMESPACE\\\"=\\\"zz-cluster-load-test-0231\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T14:59:13.436671505Z\",\"created_by\":\"/bin/sh -c #(nop) LABEL \\\"io.openshift.build.name\\\"=\\\"openjdk-devel-1\\\" \\\"io.openshift.build.namespace\\\"=\\\"zz-cluster-load-test-0231\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T15:57:15.906697248Z\",\"created_by\":\"/bin/sh -c #(nop) ENV \\\"BUILD_LOGLEVEL\\\"=\\\"6\\\"\",\"empty_layer\":true},{\"created\":\"2020-03-12T15:57:18.063087881Z\",\"created_by\":\"/bin/sh -c #(nop) COPY . / ./\"},{\"created\":\"2020-03-12T15:59:33.235872979Z\",\"created_by\":\"/bin/sh -c ./gradlew build \\u0026\\u0026 mv build/libs/spring-boot-rest-example-*.jar spring-boot-rest-example.jar\"}]}"
time="2020-03-12T15:59:43Z" level=debug msg="Docker v2s2 manifest = {\"schemaVersion\":2,\"mediaType\":\"application/vnd.docker.distribution.manifest.v2+json\",\"config\":{\"mediaType\":\"application/vnd.docker.container.image.v1+json\",\"size\":6214,\"digest\":\"sha256:aa1a743c9b4d9390a9e3e1fbd4de7fcd88ffe7b3e1bcbbc085d1f4c6c80ebc13\"},\"layers\":[{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":83005440,\"digest\":\"sha256:2f4129b6dea024391198419be227846bebc4a63d94a5c276c2920a3701f41651\"},{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":20480,\"digest\":\"sha256:a5d78edbc6523a89a2687ed7e2818bc15851a6fd62ac5833a4b12c2b38d07cad\"},{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":358737408,\"digest\":\"sha256:7772f548ccef4b2910e63af9bc31ab8324f58749436043e5c7ed1d263f0a5aa8\"},{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":1838080,\"digest\":\"sha256:603a0d3f7ac375cb6d28e4b4495d41524347d96730ad9451923a202ee1ba7634\"},{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar\",\"size\":595258880,\"digest\":\"sha256:f63167fca0795125dcffccfd0fbd4a2d15d27ea63ff043d5497d0ff3a5fe7461\"}]}"
time="2020-03-12T15:59:43Z" level=debug msg="starting to read from image \"containers-storage:021b56a848a78cc8571f81f8ec55f381ca0640588335be8a6dfa833acb82a43d-working-container\" using blob cache in \"/var/cache/blobs\" (compression=0)"
time="2020-03-12T15:59:43Z" level=debug msg="finished reading from image \"containers-storage:021b56a848a78cc8571f81f8ec55f381ca0640588335be8a6dfa833acb82a43d-working-container\" using blob cache: cache had 0 hits, 1 misses, 0 errors"
time="2020-03-12T15:59:43Z" level=debug msg="Skipping blob sha256:2f4129b6dea024391198419be227846bebc4a63d94a5c276c2920a3701f41651 (already present):"
time="2020-03-12T15:59:43Z" level=debug msg="Skipping blob sha256:a5d78edbc6523a89a2687ed7e2818bc15851a6fd62ac5833a4b12c2b38d07cad (already present):"
time="2020-03-12T15:59:43Z" level=debug msg="Skipping blob sha256:7772f548ccef4b2910e63af9bc31ab8324f58749436043e5c7ed1d263f0a5aa8 (already present):"
time="2020-03-12T15:59:43Z" level=debug msg="Skipping blob sha256:603a0d3f7ac375cb6d28e4b4495d41524347d96730ad9451923a202ee1ba7634 (already present):"
time="2020-03-12T15:59:43Z" level=debug msg="error reading layer \"sha256:f63167fca0795125dcffccfd0fbd4a2d15d27ea63ff043d5497d0ff3a5fe7461\": open /tmp/buildah481524101/sha256:f63167fca0795125dcffccfd0fbd4a2d15d27ea63ff043d5497d0ff3a5fe7461: no such file or directory"
time="2020-03-12T15:59:44Z" level=debug msg="finished reading from image \"containers-storage:021b56a848a78cc8571f81f8ec55f381ca0640588335be8a6dfa833acb82a43d-working-container\" using blob cache: cache had 0 hits, 3 misses, 0 errors"
time="2020-03-12T15:59:44Z" level=debug msg="finished writing to image \"containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.imagestore=/var/lib/shared]docker.io/library/40e402db80b43803dfa961ad8fbf2918e91e49ad7cc99b25a7b3d59be0c8161b-tmp:latest\" using blob cache"
I0312 15:59:45.073275       1 round_trippers.go:438] GET https://172.30.0.1:443/apis/build.openshift.io/v1/namespaces/zz-cluster-load-test-0231/builds/advanced-deployment-builder-2 200 OK in 8 milliseconds
I0312 15:59:45.083682       1 round_trippers.go:438] PUT https://172.30.0.1:443/apis/build.openshift.io/v1/namespaces/zz-cluster-load-test-0231/builds/advanced-deployment-builder-2/details 200 OK in 10 milliseconds
I0312 15:59:45.087652       1 round_trippers.go:438] GET https://172.30.0.1:443/apis/build.openshift.io/v1/namespaces/zz-cluster-load-test-0231/builds/advanced-deployment-builder-2 200 OK in 3 milliseconds
I0312 15:59:45.096922       1 round_trippers.go:438] PUT https://172.30.0.1:443/apis/build.openshift.io/v1/namespaces/zz-cluster-load-test-0231/builds/advanced-deployment-builder-2/details 200 OK in 8 milliseconds
F0312 15:59:45.151449       1 helpers.go:114] error: build error: error committing container for step {Env:[BUILD_LOGLEVEL=6 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=oci OPENSHIFT_BUILD_NAME=openjdk-devel-1 OPENSHIFT_BUILD_NAMESPACE=zz-cluster-load-test-0231  BUILD_LOGLEVEL=6] Command:run Args:[./gradlew build && mv build/libs/spring-boot-rest-example-*.jar spring-boot-rest-example.jar] Flags:[] Attrs:map[] Message:RUN ./gradlew build && mv build/libs/spring-boot-rest-example-*.jar spring-boot-rest-example.jar Original:RUN ./gradlew build && mv build/libs/spring-boot-rest-example-*.jar spring-boot-rest-example.jar}: error copying layers and metadata for container "954b93827620aafe2f6d0b77909f7382e92c6758830da0b50244281ebfd5f29e": Error reading blob sha256:f63167fca0795125dcffccfd0fbd4a2d15d27ea63ff043d5497d0ff3a5fe7461: error reading blob from source image "containers-storage:021b56a848a78cc8571f81f8ec55f381ca0640588335be8a6dfa833acb82a43d-working-container": error opening file "/tmp/buildah481524101/sha256:f63167fca0795125dcffccfd0fbd4a2d15d27ea63ff043d5497d0ff3a5fe7461" to buffer layer blob: open /tmp/buildah481524101/sha256:f63167fca0795125dcffccfd0fbd4a2d15d27ea63ff043d5497d0ff3a5fe7461: no such file or directory

This looks like a bad interaction between the image committing logic and the blob caching logic.  While I can't directly reproduce the failure here, it's telling that when committing every layer, the job of generating a configuration blob and manifest for the to-be-committed image is being run twice, and each time, we cache copies of the blobs that represent any layers that aren't already in the blob cache in a per-job unique temporary directory.  In the log, between the two jobs, the content (and both the size and the digest of the content) of the newest layer is different.  We then appear to be attempting to read the cached copy of the blob for one layer from the wrong directory.  My thinking is that preventing the job from needing to be run a second time will avoid any such mismatch.

Opened https://github.com/containers/buildah/pull/2500 with a candidate patch for the library that we consume.  If/when it's merged, we'll need to update the version of the library that we use in OpenShift to fix it here.

Nalin, the PR is already merged, let us know if you could share next steps about delivering ERRATA with new version of the library.

test many times and trigger no error like 'error reading blob from source image'
verified on version : 4.6.0-0.nightly-2020-09-26-202331

$ oc logs -f build/nodejs-ex-1 --loglevel=8
I0928 15:10:23.979101   20160 loader.go:375] Config loaded from file:  ./kubeconfig_debug
I0928 15:10:23.987991   20160 round_trippers.go:420] GET https://api.minmli092746.qe.devcluster.openshift.com:6443/apis/build.openshift.io/v1/namespaces/minmli/builds/nodejs-ex-1
I0928 15:10:23.988026   20160 round_trippers.go:427] Request Headers:
I0928 15:10:23.988045   20160 round_trippers.go:431]     Accept: application/json, */*
I0928 15:10:23.988062   20160 round_trippers.go:431]     User-Agent: oc/4.5.0 (linux/amd64) kubernetes/9933eb9
I0928 15:10:24.529402   20160 round_trippers.go:446] Response Status: 200 OK in 541 milliseconds
I0928 15:10:24.529449   20160 round_trippers.go:449] Response Headers:
I0928 15:10:24.529494   20160 round_trippers.go:452]     Content-Length: 3597
I0928 15:10:24.529541   20160 round_trippers.go:452]     Audit-Id: 8a72c055-abd3-45b9-b343-63fe35a6a95e
I0928 15:10:24.529567   20160 round_trippers.go:452]     Cache-Control: no-cache, private
I0928 15:10:24.529591   20160 round_trippers.go:452]     Cache-Control: no-store
I0928 15:10:24.529614   20160 round_trippers.go:452]     Content-Type: application/json
I0928 15:10:24.529637   20160 round_trippers.go:452]     Date: Mon, 28 Sep 2020 07:10:24 GMT
I0928 15:10:24.529659   20160 round_trippers.go:452]     X-Kubernetes-Pf-Flowschema-Uid: 5b1f1dcd-b82a-4310-a2c2-5d634c8d1281
I0928 15:10:24.529680   20160 round_trippers.go:452]     X-Kubernetes-Pf-Prioritylevel-Uid: 35be4319-ece5-40cc-891f-4bb32f85874e
I0928 15:10:24.529795   20160 request.go:1068] Response Body: {"kind":"Build","apiVersion":"build.openshift.io/v1","metadata":{"name":"nodejs-ex-1","namespace":"minmli","selfLink":"/apis/build.openshift.io/v1/namespaces/minmli/builds/nodejs-ex-1","uid":"5d9a9787-ce4f-44ca-b4ec-cf41190e466a","resourceVersion":"1387368","creationTimestamp":"2020-09-28T07:05:52Z","labels":{"app":"nodejs-ex","app.kubernetes.io/component":"nodejs-ex","app.kubernetes.io/instance":"nodejs-ex","buildconfig":"nodejs-ex","openshift.io/build-config.name":"nodejs-ex","openshift.io/build.start-policy":"Serial"},"annotations":{"openshift.io/build-config.name":"nodejs-ex","openshift.io/build.number":"1","openshift.io/build.pod-name":"nodejs-ex-1-build"},"ownerReferences":[{"apiVersion":"build.openshift.io/v1","kind":"BuildConfig","name":"nodejs-ex","uid":"e97301f6-327b-4f08-bd8d-34504912bd51","controller":true}]},"spec":{"serviceAccount":"builder","source":{"type":"Git","git":{"uri":"https://github.com/sclorg/nodejs-ex"}},"revision":{"type":"Git","git":{"commit":"7b9f57949786059a3fab03b8493279c945770f [truncated 2573 chars]
I0928 15:10:24.537644   20160 round_trippers.go:420] GET https://api.minmli092746.qe.devcluster.openshift.com:6443/apis/build.openshift.io/v1/namespaces/minmli/builds/nodejs-ex-1/log?follow=true
I0928 15:10:24.537692   20160 round_trippers.go:427] Request Headers:
I0928 15:10:24.537725   20160 round_trippers.go:431]     Accept: application/json, */*
I0928 15:10:24.537752   20160 round_trippers.go:431]     User-Agent: oc/4.5.0 (linux/amd64) kubernetes/9933eb9
I0928 15:10:24.742964   20160 round_trippers.go:446] Response Status: 200 OK in 205 milliseconds
I0928 15:10:24.743015   20160 round_trippers.go:449] Response Headers:
I0928 15:10:24.743047   20160 round_trippers.go:452]     Audit-Id: 31cf04d5-915b-4f5c-af70-8a35b81481fa
I0928 15:10:24.743074   20160 round_trippers.go:452]     Cache-Control: no-cache, private
I0928 15:10:24.743098   20160 round_trippers.go:452]     Cache-Control: no-store
I0928 15:10:24.743127   20160 round_trippers.go:452]     Content-Type: application/octet-stream
I0928 15:10:24.743152   20160 round_trippers.go:452]     Date: Mon, 28 Sep 2020 07:10:24 GMT
Cloning "https://github.com/sclorg/nodejs-ex" ...
	Commit:	7b9f57949786059a3fab03b8493279c945770fb0 (Merge pull request #249 from multi-arch/master)
	Author:	Honza Horak <hhorak>
	Date:	Wed Sep 23 10:52:52 2020 +0200
Caching blobs under "/var/cache/blobs".
Getting image source signatures
Copying blob sha256:c4d668e229cd131e0a8e4f8218dca628d9cf9697572875e355fe4b247b6aa9f0
Copying blob sha256:da1cc572023a942fff15d59aefa5abbb59d2c24a03966db8074ef8f9bab277d4
Copying blob sha256:ec1681b6a383e4ecedbeddd5abc596f3de835aed6db39a735f62395c8edbff30
Copying blob sha256:35ad9b4fba1fa6b00a6f266303348dc0cf9a7c341616e800c2738030c0f64167
Copying blob sha256:5e2ae0c76e83847010202c40d0c7ebac953a6c7871efdea7602b41507b3d11f5
Copying config sha256:8a961c0b3cbcc653bf39713aaf79a36d9921618e2a39fd7e5057cf70c203cf87
Writing manifest to image destination
Storing signatures
Generating dockerfile with builder image image-registry.openshift-image-registry.svc:5000/openshift/nodejs@sha256:1175006938852dff3ac21cdcab0939b65e0499a6ba0c4d1586869b3f31465f3f
STEP 1: FROM image-registry.openshift-image-registry.svc:5000/openshift/nodejs@sha256:1175006938852dff3ac21cdcab0939b65e0499a6ba0c4d1586869b3f31465f3f
STEP 2: LABEL "io.openshift.build.image"="image-registry.openshift-image-registry.svc:5000/openshift/nodejs@sha256:1175006938852dff3ac21cdcab0939b65e0499a6ba0c4d1586869b3f31465f3f"       "io.openshift.build.commit.author"="Honza Horak <hhorak>"       "io.openshift.build.commit.date"="Wed Sep 23 10:52:52 2020 +0200"       "io.openshift.build.commit.id"="7b9f57949786059a3fab03b8493279c945770fb0"       "io.openshift.build.commit.ref"="master"       "io.openshift.build.commit.message"="Merge pull request #249 from multi-arch/master"       "io.openshift.build.source-location"="https://github.com/sclorg/nodejs-ex"
STEP 3: ENV OPENSHIFT_BUILD_NAME="nodejs-ex-1"     OPENSHIFT_BUILD_NAMESPACE="minmli"     OPENSHIFT_BUILD_SOURCE="https://github.com/sclorg/nodejs-ex"     OPENSHIFT_BUILD_COMMIT="7b9f57949786059a3fab03b8493279c945770fb0"
STEP 4: USER root
STEP 5: COPY upload/src /tmp/src
STEP 6: RUN chown -R 1001:0 /tmp/src
STEP 7: USER 1001
STEP 8: RUN /usr/libexec/s2i/assemble
---> Installing application source ...
---> Installing all dependencies
npm WARN deprecated jade.3: Jade has been renamed to pug, please install the latest version of pug instead of jade
npm WARN deprecated to-iso-string.2: to-iso-string has been deprecated, use @segment/to-iso-string instead.
npm WARN deprecated mkdirp.1: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
npm WARN deprecated bson.9: Fixed a critical issue with BSON serialization documented in CVE-2019-2391, see https://bit.ly/2KcpXdo for more details
npm WARN deprecated mkdirp.0: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
npm WARN deprecated minimatch.0: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue

> ejs.4 postinstall /opt/app-root/src/node_modules/ejs
> node ./postinstall.js

Thank you for installing EJS: built with the Jake JavaScript build tool (https://jakejs.com/)

npm notice created a lockfile as package-lock.json. You should commit this file.
added 121 packages from 342 contributors and audited 121 packages in 4.015s
found 10 vulnerabilities (3 low, 2 moderate, 4 high, 1 critical)
  run `npm audit fix` to fix them, or `npm audit` for details
---> Building in production mode
---> Pruning the development dependencies
audited 121 packages in 0.917s
found 10 vulnerabilities (3 low, 2 moderate, 4 high, 1 critical)
  run `npm audit fix` to fix them, or `npm audit` for details
/opt/app-root/src/.npm is not a mountpoint
---> Cleaning the npm cache /opt/app-root/src/.npm
/tmp is not a mountpoint
---> Cleaning the /tmp/npm-*
STEP 9: CMD /usr/libexec/s2i/run
STEP 10: COMMIT temp.builder.openshift.io/minmli/nodejs-ex-1:9283bd5b
Getting image source signatures
Copying blob sha256:ccf04fbd6e1943f648d1c2980e96038edc02b543c597556098ab2bcaa4fd1fa8
Copying blob sha256:b7b591e3443f17f9d8272b8d118b6c031ca826deb09d4b44f296ba934f1b6e57
Copying blob sha256:511ade7a7dffb350f0e8b72e1a36ed9f1507b5a5caeafe30721d5d1c8b33f1ff
Copying blob sha256:ffdb6cc9bdbbb5e11c5dd0aad9f4eacf35f3adcf5c6df4f4d2c3120cee96dd55
Copying blob sha256:8e38d704f1b812ffefa50f91815ff0f509e72ffc9d3bdbadc8c2b05d2eed1bfc
Copying blob sha256:0e02af8fcd74b7cdff64349d28ee3ed53781d7e546bbae86944aef924cc6faff
Copying config sha256:0ffc6c4be37e02a5aec693743633c58114e097d18b6c4e042837b759fae71949
Writing manifest to image destination
Storing signatures
--> 0ffc6c4be37
0ffc6c4be37e02a5aec693743633c58114e097d18b6c4e042837b759fae71949
Getting image source signatures

Pushing image image-registry.openshift-image-registry.svc:5000/minmli/nodejs-ex:latest ...
Copying blob sha256:0e02af8fcd74b7cdff64349d28ee3ed53781d7e546bbae86944aef924cc6faff
Copying blob sha256:c4d668e229cd131e0a8e4f8218dca628d9cf9697572875e355fe4b247b6aa9f0
Copying blob sha256:35ad9b4fba1fa6b00a6f266303348dc0cf9a7c341616e800c2738030c0f64167
Copying blob sha256:ec1681b6a383e4ecedbeddd5abc596f3de835aed6db39a735f62395c8edbff30
Copying blob sha256:5e2ae0c76e83847010202c40d0c7ebac953a6c7871efdea7602b41507b3d11f5
Copying blob sha256:da1cc572023a942fff15d59aefa5abbb59d2c24a03966db8074ef8f9bab277d4
Copying config sha256:0ffc6c4be37e02a5aec693743633c58114e097d18b6c4e042837b759fae71949
Writing manifest to image destination
Storing signatures
Successfully pushed image-registry.openshift-image-registry.svc:5000/minmli/nodejs-ex@sha256:2d2dafe0a53e304e5bf6adb918e5446c54f924c3a07cf3c8d7e7eaf8f4a74294
Push successful

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196

Hello,

I can see that it was released for OCP 4.6, any change of being this backported to 4.5 or exists a bug for being backported to 4.5?

Thanks in advance,
Oscar

Adam, Nalin, would it be possible to backport this to OCP 4.5?

This was fixed in buildah v1.14.11, which was included in to 4.5.8 update as part of the fix for Bug #1868401. I'll make the clone BZs to make it clear that this was also fixed in 4.5.z.

https://access.redhat.com/errata/RHBA-2020:3510

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days