From interaction with openshift devs using 4.x vs. 3.x based CI per agreement with Nalin, opening a bug to track See slack thread https://coreos.slack.com/archives/C014MHHKUSF/p1591216952054700 for the triage / diagnosis
Realized I did the wrong product/component ... fixing now.
Moving to the Buildah component as assigning to Nalin.
I'm able to reproduce this, but only in OpenShift, as the buildah CLI seems to be getting the right result. The incorrect value could be coming from ContainerConfig, or it could be the hardwired default that we use when no value is set, since in the error case they appear to have the same value.
I no longer believe that ContainerConfig/Config confusion is the root cause. https://github.com/openshift/imagebuilder/pull/169 should fix the problem once it's merged and we integrate a version with the change.
We've encountered a regression in the upstream fix (https://github.com/openshift/imagebuilder/issues/173) which I thought would resolve this, so we need to do some more work there.
Regression in imagebuilder should be fixed as of https://github.com/openshift/imagebuilder/pull/174, so we need to tag a release there and then we can pull it in to the builder.
The release in ImageBuilder was just recently created and needs to now be vendored through Buildah and Podman. That won't happen today, so I'm moving to Upcoming Sprint.
https://github.com/containers/buildah/pull/2616
Checked on 4.6.0-0.nightly-2020-09-30-052433 using below build and do not see the issue. $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.6.0-0.nightly-2020-09-30-052433 True False 7h43m Cluster version is 4.6.0-0.nightly-2020-09-30-052433 $ cat build.yaml apiVersion: image.openshift.io/v1 kind: ImageStream metadata: name: pipeline namespace: default --- apiVersion: build.openshift.io/v1 kind: Build metadata: name: root namespace: default spec: serviceAccount: builder source: git: uri: https://github.com/openshift/ocp-release-operator-sdk type: Git strategy: type: Docker dockerStrategy: from: kind: DockerImage name: registry.svc.ci.openshift.org/openshift/release:golang-1.13 dockerfilePath: ci/dockerfiles/builder.Dockerfile $ oc create -f build.yaml imagestream.image.openshift.io/pipeline created build.build.openshift.io/root created $ oc get builds NAME TYPE FROM STATUS STARTED DURATION root Docker Git@6c8ecfc Running 46 seconds ago $ oc get builds NAME TYPE FROM STATUS STARTED DURATION root Docker Git@6c8ecfc Complete 7 minutes ago 4m0s $ oc describe build root Name: root Namespace: default Created: 7 minutes ago Labels: <none> Annotations: openshift.io/build.pod-name=root-build Status: Complete Started: Wed, 30 Sep 2020 12:27:04 IST Duration: 4m0s FetchInputs: 29s PullImages: 40s Build: 2m38s Build Pod: root-build Strategy: Docker URL: https://github.com/openshift/ocp-release-operator-sdk Commit: 6c8ecfc (Merge pull request #74 from openshift-bot/updating-release/helm/dockerfile-bas..) Author/Committer: OpenShift Merge Robot / GitHub From Image: DockerImage registry.svc.ci.openshift.org/openshift/release:golang-1.13 Dockerfile Path: ci/dockerfiles/builder.Dockerfile Build trigger cause: <unknown> Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> Successfully assigned default/root-build to ip-10-0-193-246.us-east-2.compute.internal Normal AddedInterface 7m22s multus Add eth0 [10.129.2.6/23] Normal Pulling 7m22s kubelet, ip-10-0-193-246.us-east-2.compute.internal Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1a27bd55c6ca56e8d2ef2ad694a5b7817eed55777d94c7cae6494ebeef5951d9" Normal Created 7m14s kubelet, ip-10-0-193-246.us-east-2.compute.internal Created container git-clone Normal Started 7m14s kubelet, ip-10-0-193-246.us-east-2.compute.internal Started container git-clone Normal Pulled 7m14s kubelet, ip-10-0-193-246.us-east-2.compute.internal Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1a27bd55c6ca56e8d2ef2ad694a5b7817eed55777d94c7cae6494ebeef5951d9" in 7.836926666s Normal BuildStarted 7m13s build-controller Build default/root is now running Normal Pulled 6m44s kubelet, ip-10-0-193-246.us-east-2.compute.internal Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1a27bd55c6ca56e8d2ef2ad694a5b7817eed55777d94c7cae6494ebeef5951d9" already present on machine Normal Created 6m44s kubelet, ip-10-0-193-246.us-east-2.compute.internal Created container manage-dockerfile Normal Started 6m44s kubelet, ip-10-0-193-246.us-east-2.compute.internal Started container manage-dockerfile Normal Started 6m43s kubelet, ip-10-0-193-246.us-east-2.compute.internal Started container docker-build Normal Pulled 6m43s kubelet, ip-10-0-193-246.us-east-2.compute.internal Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1a27bd55c6ca56e8d2ef2ad694a5b7817eed55777d94c7cae6494ebeef5951d9" already present on machine Normal Created 6m43s kubelet, ip-10-0-193-246.us-east-2.compute.internal Created container docker-build Normal BuildCompleted 3m24s build-controller Build default/root completed successfully [sunilc@system 300946]$ oc get pods NAME READY STATUS RESTARTS AGE root-build 0/1 Completed 0 7m40s $ oc logs root-build Caching blobs under "/var/cache/blobs". Pulling image registry.svc.ci.openshift.org/openshift/release:golang-1.13 ... Getting image source signatures Copying blob sha256:372921d38035c45c718b1c1c66cf8273d5d72ac3159b7108a441411927f08efd Copying blob sha256:75f829a71a1c5277a7abf55495ac8d16759691d980bf1d931795e5eb68a294c0 Copying config sha256:27cdaeb53567c7a78515d9719ceb0a172b819045692df37884367d270be87642 Writing manifest to image destination Storing signatures STEP 1: FROM registry.svc.ci.openshift.org/openshift/release:golang-1.13 STEP 2: WORKDIR /go/src/github.com/operator-framework/operator-sdk --> 838250663f0 STEP 3: ENV GOPATH=/go PATH=/go/src/github.com/operator-framework/operator-sdk/build:$PATH GO111MODULE=on GOFLAGS=-mod=vendor --> f8e5a14ab91 STEP 4: COPY . . --> 0fbf15d9268 STEP 5: RUN echo $PATH ; which go ; make -f ci/prow.Makefile build /go/src/github.com/operator-framework/operator-sdk/build:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/go/bin /usr/local/go/bin/go make -f Makefile build/operator-sdk make[1]: Entering directory `/go/src/github.com/operator-framework/operator-sdk' make[1]: Leaving directory `/go/src/github.com/operator-framework/operator-sdk' --> e2e4c2a24e6 STEP 6: ENV "OPENSHIFT_BUILD_NAME"="root" "OPENSHIFT_BUILD_NAMESPACE"="default" "OPENSHIFT_BUILD_SOURCE"="https://github.com/openshift/ocp-release-operator-sdk" "OPENSHIFT_BUILD_COMMIT"="6c8ecfc6a506ed2d912c9355b50534b827bcdb84" --> ea4b996afd1 STEP 7: LABEL "io.openshift.build.commit.author"="OpenShift Merge Robot \u003copenshift-merge-robot.github.com\u003e" "io.openshift.build.commit.date"="Sat Sep 19 05:08:17 2020 -0400" "io.openshift.build.commit.id"="6c8ecfc6a506ed2d912c9355b50534b827bcdb84" "io.openshift.build.commit.message"="Merge pull request #74 from openshift-bot/updating-release/helm/dockerfile-bas.." "io.openshift.build.commit.ref"="master" "io.openshift.build.name"="root" "io.openshift.build.namespace"="default" "io.openshift.build.source-location"="https://github.com/openshift/ocp-release-operator-sdk" STEP 8: COMMIT temp.builder.openshift.io/default/root:fe79a432 --> e9d278573c5 e9d278573c5a49eeb58f2f62d25add3d4561834af4075d01508fe90dbc6bb762 Build complete, no image push requested
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196