Bug 1843727

Summary: buildah needs to consider envs, etc. from both ContainerConfig and Config from image
Product: OpenShift Container Platform Reporter: Gabe Montero <gmontero>
Component: ContainersAssignee: Nalin Dahyabhai <nalin>
Status: CLOSED ERRATA QA Contact: Sunil Choudhary <schoudha>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.6CC: aos-bugs, dwalsh, jokerman, tsweeney
Target Milestone: ---   
Target Release: 4.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-27 16:04:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Gabe Montero 2020-06-03 23:14:22 UTC 
From interaction with openshift devs using 4.x vs. 3.x based CI

per agreement with Nalin, opening a bug to track

See slack thread https://coreos.slack.com/archives/C014MHHKUSF/p1591216952054700 for the triage / diagnosis
Comment 1 Gabe Montero 2020-06-04 23:45:15 UTC 
Realized I did the wrong product/component ... fixing now.
Comment 2 Tom Sweeney 2020-06-05 14:51:26 UTC 
Moving to the Buildah component as assigning to Nalin.
Comment 4 Nalin Dahyabhai 2020-07-10 20:58:50 UTC 
I'm able to reproduce this, but only in OpenShift, as the buildah CLI seems to be getting the right result.  The incorrect value could be coming from ContainerConfig, or it could be the hardwired default that we use when no value is set, since in the error case they appear to have the same value.
Comment 5 Nalin Dahyabhai 2020-07-14 02:03:45 UTC 
I no longer believe that ContainerConfig/Config confusion is the root cause.  https://github.com/openshift/imagebuilder/pull/169 should fix the problem once it's merged and we integrate a version with the change.
Comment 6 Nalin Dahyabhai 2020-07-30 22:07:51 UTC 
We've encountered a regression in the upstream fix (https://github.com/openshift/imagebuilder/issues/173) which I thought would resolve this, so we need to do some more work there.
Comment 7 Nalin Dahyabhai 2020-08-18 13:47:44 UTC 
Regression in imagebuilder should be fixed as of https://github.com/openshift/imagebuilder/pull/174, so we need to tag a release there and then we can pull it in to the builder.
Comment 8 Tom Sweeney 2020-09-11 19:01:14 UTC 
The release in ImageBuilder was just recently created and needs to now be vendored through Buildah and Podman.  That won't happen today, so I'm moving to Upcoming Sprint.
Comment 10 Daniel Walsh 2020-09-15 13:28:46 UTC 
https://github.com/containers/buildah/pull/2616
Comment 11 Sunil Choudhary 2020-09-30 14:50:17 UTC 
Checked on 4.6.0-0.nightly-2020-09-30-052433 using below build and do not see the issue.

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2020-09-30-052433   True        False         7h43m   Cluster version is 4.6.0-0.nightly-2020-09-30-052433

$ cat build.yaml
apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
  name: pipeline
  namespace: default
---
apiVersion: build.openshift.io/v1
kind: Build
metadata:
  name: root
  namespace: default
spec:
  serviceAccount: builder
  source:
    git:
      uri: https://github.com/openshift/ocp-release-operator-sdk
    type: Git
  strategy:
    type: Docker
    dockerStrategy:
      from:
        kind: DockerImage
        name: registry.svc.ci.openshift.org/openshift/release:golang-1.13
      dockerfilePath: ci/dockerfiles/builder.Dockerfile

$ oc create -f build.yaml 
imagestream.image.openshift.io/pipeline created
build.build.openshift.io/root created

$ oc get builds
NAME   TYPE     FROM          STATUS    STARTED          DURATION
root   Docker   Git@6c8ecfc   Running   46 seconds ago   

$ oc get builds
NAME   TYPE     FROM          STATUS     STARTED         DURATION
root   Docker   Git@6c8ecfc   Complete   7 minutes ago   4m0s

$ oc describe build root
Name:		root
Namespace:	default
Created:	7 minutes ago
Labels:		<none>
Annotations:	openshift.io/build.pod-name=root-build

Status:		Complete
Started:	Wed, 30 Sep 2020 12:27:04 IST
Duration:	4m0s
  FetchInputs:	  29s
  PullImages:	  40s
  Build:	  2m38s

Build Pod:	root-build

Strategy:		Docker
URL:			https://github.com/openshift/ocp-release-operator-sdk
Commit:			6c8ecfc (Merge pull request #74 from openshift-bot/updating-release/helm/dockerfile-bas..)
Author/Committer:	OpenShift Merge Robot / GitHub
From Image:		DockerImage registry.svc.ci.openshift.org/openshift/release:golang-1.13
Dockerfile Path:	ci/dockerfiles/builder.Dockerfile

Build trigger cause:	<unknown>

Events:
  Type		Reason		Age		From							Message
  ----		------		----		----							-------
  Normal	Scheduled	<unknown>								Successfully assigned default/root-build to ip-10-0-193-246.us-east-2.compute.internal
  Normal	AddedInterface	7m22s		multus							Add eth0 [10.129.2.6/23]
  Normal	Pulling		7m22s		kubelet, ip-10-0-193-246.us-east-2.compute.internal	Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1a27bd55c6ca56e8d2ef2ad694a5b7817eed55777d94c7cae6494ebeef5951d9"
  Normal	Created		7m14s		kubelet, ip-10-0-193-246.us-east-2.compute.internal	Created container git-clone
  Normal	Started		7m14s		kubelet, ip-10-0-193-246.us-east-2.compute.internal	Started container git-clone
  Normal	Pulled		7m14s		kubelet, ip-10-0-193-246.us-east-2.compute.internal	Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1a27bd55c6ca56e8d2ef2ad694a5b7817eed55777d94c7cae6494ebeef5951d9" in 7.836926666s
  Normal	BuildStarted	7m13s		build-controller					Build default/root is now running
  Normal	Pulled		6m44s		kubelet, ip-10-0-193-246.us-east-2.compute.internal	Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1a27bd55c6ca56e8d2ef2ad694a5b7817eed55777d94c7cae6494ebeef5951d9" already present on machine
  Normal	Created		6m44s		kubelet, ip-10-0-193-246.us-east-2.compute.internal	Created container manage-dockerfile
  Normal	Started		6m44s		kubelet, ip-10-0-193-246.us-east-2.compute.internal	Started container manage-dockerfile
  Normal	Started		6m43s		kubelet, ip-10-0-193-246.us-east-2.compute.internal	Started container docker-build
  Normal	Pulled		6m43s		kubelet, ip-10-0-193-246.us-east-2.compute.internal	Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1a27bd55c6ca56e8d2ef2ad694a5b7817eed55777d94c7cae6494ebeef5951d9" already present on machine
  Normal	Created		6m43s		kubelet, ip-10-0-193-246.us-east-2.compute.internal	Created container docker-build
  Normal	BuildCompleted	3m24s		build-controller					Build default/root completed successfully
[sunilc@system 300946]$ oc get pods
NAME         READY   STATUS      RESTARTS   AGE
root-build   0/1     Completed   0          7m40s

$ oc logs root-build
Caching blobs under "/var/cache/blobs".

Pulling image registry.svc.ci.openshift.org/openshift/release:golang-1.13 ...
Getting image source signatures
Copying blob sha256:372921d38035c45c718b1c1c66cf8273d5d72ac3159b7108a441411927f08efd
Copying blob sha256:75f829a71a1c5277a7abf55495ac8d16759691d980bf1d931795e5eb68a294c0
Copying config sha256:27cdaeb53567c7a78515d9719ceb0a172b819045692df37884367d270be87642
Writing manifest to image destination
Storing signatures
STEP 1: FROM registry.svc.ci.openshift.org/openshift/release:golang-1.13
STEP 2: WORKDIR /go/src/github.com/operator-framework/operator-sdk
--> 838250663f0
STEP 3: ENV GOPATH=/go     PATH=/go/src/github.com/operator-framework/operator-sdk/build:$PATH     GO111MODULE=on     GOFLAGS=-mod=vendor
--> f8e5a14ab91
STEP 4: COPY . .
--> 0fbf15d9268
STEP 5: RUN echo $PATH ; which go ; make -f ci/prow.Makefile build
/go/src/github.com/operator-framework/operator-sdk/build:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/go/bin
/usr/local/go/bin/go
make -f Makefile build/operator-sdk
make[1]: Entering directory `/go/src/github.com/operator-framework/operator-sdk'
make[1]: Leaving directory `/go/src/github.com/operator-framework/operator-sdk'
--> e2e4c2a24e6
STEP 6: ENV "OPENSHIFT_BUILD_NAME"="root" "OPENSHIFT_BUILD_NAMESPACE"="default" "OPENSHIFT_BUILD_SOURCE"="https://github.com/openshift/ocp-release-operator-sdk" "OPENSHIFT_BUILD_COMMIT"="6c8ecfc6a506ed2d912c9355b50534b827bcdb84"
--> ea4b996afd1
STEP 7: LABEL "io.openshift.build.commit.author"="OpenShift Merge Robot \u003copenshift-merge-robot.github.com\u003e" "io.openshift.build.commit.date"="Sat Sep 19 05:08:17 2020 -0400" "io.openshift.build.commit.id"="6c8ecfc6a506ed2d912c9355b50534b827bcdb84" "io.openshift.build.commit.message"="Merge pull request #74 from openshift-bot/updating-release/helm/dockerfile-bas.." "io.openshift.build.commit.ref"="master" "io.openshift.build.name"="root" "io.openshift.build.namespace"="default" "io.openshift.build.source-location"="https://github.com/openshift/ocp-release-operator-sdk"
STEP 8: COMMIT temp.builder.openshift.io/default/root:fe79a432
--> e9d278573c5
e9d278573c5a49eeb58f2f62d25add3d4561834af4075d01508fe90dbc6bb762
Build complete, no image push requested
Comment 14 errata-xmlrpc 2020-10-27 16:04:47 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196