Bug 1696756

Summary: Source strategy builds using wrong Assemble User
Product: OpenShift Container Platform Reporter: Adam Kaplan <adam.kaplan>
Component: BuildAssignee: Adam Kaplan <adam.kaplan>
Status: CLOSED ERRATA QA Contact: wewang <wewang>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.1.0CC: aos-bugs, gdumplet, jmorales, wzheng
Target Milestone: ---   
Target Release: 4.1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-04 10:47:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Adam Kaplan 2019-04-05 14:58:15 UTC
Description: A docker type build on OpenShift 4.0 fails because it is using a Dockerfile from the wrong directory, or mixing cached image layers from a second build in the same project, when using spec.source.contextDir, and the two builds in the same project are for the same Git repository. The same code base builds fine on 3.11, and also in an automated build on quay.io.

Affects Version: 4.1

oc v3.11.0+0cbc58b
kubernetes v1.11.0+d4cacc0
features: Basic-Auth

Server https://api.cluster-9586.9586.openshiftworkshop.com:6443
kubernetes v1.12.4+0f8e04e

Steps To Reproduce:

In a 4.0 cluster run:

oc new-app https://raw.githubusercontent.com/openshift-labs/workshop-spawner/develop/templates/jumpbox-server-development.json
In addition to some deployments, this creates two build configurations, both using the same Git repository, but different contextDir within the same Git repository. The separate directories used for the build both have a Dockerfile.

Current Result
The build configuration for the custom keycloak image fails with:

Generating dockerfile with builder image jboss/keycloak-openshift:4.0.0.Final
STEP 1: FROM jboss/keycloak-openshift:4.0.0.Final
STEP 2: LABEL "io.openshift.build.source-location"="/tmp/build/inputs" "io.openshift.build.source-context-dir"="keycloak" "io.openshift.build.image"="jboss/keycloak-openshift:4.0.0.Final"
STEP 3: ENV OPENSHIFT_BUILD_NAME="jumpbox-keycloak-2" OPENSHIFT_BUILD_NAMESPACE="workshop"
STEP 4: USER root
STEP 5: COPY upload/scripts /tmp/scripts
STEP 6: COPY upload/src /tmp/src
STEP 7: RUN chown -R 1001:0 /tmp/scripts /tmp/src
time="2019-04-05T01:35:57Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah180432153/mnt/rootfs\": error checking if \"/tmp/buildah180432153/mnt/rootfs/sys/fs/cgroup/blkio\" is mounted: no such file or directory"
time="2019-04-05T01:35:57Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah180432153/mnt/rootfs\": error checking if \"/tmp/buildah180432153/mnt/rootfs/sys/fs/cgroup/blkio\" is mounted: no such file or directory"
STEP 8: USER 1001
STEP 9: RUN /tmp/scripts/assemble
+ '[' -f /tmp/src/jupyterhub-realm.json ']'
+ mv /tmp/src/jupyterhub-realm.json .
mv: cannot move ‘/tmp/src/jupyterhub-realm.json’ to ‘./jupyterhub-realm.json’: Permission denied
subprocess exited with status 1
subprocess exited with status 1
error: build error: error building at step {Env:[OPENSHIFT_BUILD_NAMESPACE=workshop OPENSHIFT_BUILD_NAME=jumpbox-keycloak-2 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin JAVA_HOME=/usr/lib/jvm/java KEYCLOAK_VERSION=4.0.0.Final LAUNCH_JBOSS_IN_BACKGROUND=1 PROXY_ADDRESS_FORWARDING=false JDBC_POSTGRES_VERSION=42.2.2 JDBC_MYSQL_VERSION=5.1.46 JDBC_MARIADB_VERSION=2.2.3 JBOSS_HOME=/opt/jboss/keycloak LANG=en_US.UTF-8   OPENSHIFT_BUILD_NAME=jumpbox-keycloak-2 OPENSHIFT_BUILD_NAMESPACE=workshop] Command:run Args:[/tmp/scripts/assemble] Flags:[] Attrs:map[] Message:RUN /tmp/scripts/assemble Original:RUN /tmp/scripts/assemble}: exit status 1
The keycloak/Dockerfile it was supposed to use contained:

FROM jboss/keycloak-openshift:4.0.0.Final

USER root

COPY . /tmp/src

RUN mv /tmp/src/.s2i/bin /tmp/scripts

RUN rm -rf /tmp/src/.git* && \
    chown -R 1000 /tmp/src && \
    chgrp -R 0 /tmp/src && \
    chmod -R g+w /tmp/src

USER 1000

RUN /tmp/scripts/assemble

CMD [ "/tmp/scripts/run" ]
This Dockerfile uses user ID 1000. The log of the build show it using user ID 1001.

The user ID 1001 appears in a separate jupyterhub/Dockerfile, which is what the other build was using.

It appears to be either using the wrong Dockerfile, or more likely, is using a cached image layer from the wrong build context, with the log messages show what was the original command used when it built the layer it did use.

Not sure if this is specifically tied to fact that two builds use same Git repository, but with different contextDir, or whether it more general problem.

Expected Result
Build should complete okay.

Additional Information
The key part of the build configuration for the custom keycloak image is:

spec:
  failedBuildsHistoryLimit: 5
  nodeSelector: null
  output:
    to:
      kind: ImageStreamTag
      name: jumpbox-keycloak:latest
  postCommit: {}
  resources: {}
  runPolicy: Serial
  source:
    contextDir: keycloak
    git:
      ref: develop
      uri: https://github.com/openshift-labs/workshop-spawner.git
    type: Git
  strategy:
    sourceStrategy:
      from:
        kind: DockerImage
        name: jboss/keycloak-openshift:4.0.0.Final
    type: Source
See the template used in the oc new-app command above for more details of the builds.

Comment 1 Adam Kaplan 2019-04-05 15:27:05 UTC
@Graham (original reporter - CC-ed) - the "different Dockerfile" you are seeing is the new output from S2I builds. In 4.1 Source strategy builds generate a fixed Dockerfile that buildah consumes.

I think the bug here is that we're hard-coding the assemble user to "1001" if the image doesn't have the "io.openshift.s2i.assemble-user" label, but we are not checking the "last" user in the base builder image.

Comment 2 Graham Dumpleton 2019-04-05 20:42:42 UTC
It can't be because of your S2I build behaviour, as this isn't an S2I build, it is a "docker" type build and it should be doing what my Dockerfile tells it to. The Dockerfile just happens to be invoking the "assemble" script after faking up the image to mirror what S2I would do. I cannot see therefore how they are related. If it is for some reason apply S2I builds steps for a "docker" build, it is even more confused.

Comment 3 Graham Dumpleton 2019-04-05 20:52:49 UTC
Haha, I should check my own build configuration. It is setup for "source" build, when it wasn't intended to. Since I fake out the Dockerfile to trigger an S2I build so can build on quay.io, I can just switch to "docker" build to get around the problem. Thanks.

Comment 4 wewang 2019-04-08 02:56:29 UTC
@Adam Kaplan @Graham Dumpleton  I tested it in version:  4.0.0-0.11, builds completes, it works,custom keycloak image use the correct dockerfile with USER 1000

steps:
1. oc new-app https://raw.githubusercontent.com/openshift-labs/workshop-spawner/develop/templates/jumpbox-server-development.json

2. Check build and the logs of jumpbox-keycloak
$ oc get builds
NAME                 TYPE     FROM          STATUS     STARTED         DURATION
jumpbox-hub-1        Docker   Git@c7b2418   Complete   3 minutes ago   2m3s
jumpbox-keycloak-1   Docker   Git@c7b2418   Complete   3 minutes ago   2m5s
$ oc logs -f build/jumpbox-keycloak-1
 http://pastebin.test.redhat.com/750619

3.Get bc of jumpbox-keycloak
  source:
    contextDir: keycloak
    git:
      ref: develop
      uri: https://github.com/openshift-labs/workshop-spawner.git
    type: Git
  strategy:
    dockerStrategy:                ###docker strategy
      from:
        kind: DockerImage
        name: jboss/keycloak-openshift:4.0.0.Final
    type: Docker

Comment 5 Graham Dumpleton 2019-04-08 03:02:24 UTC
If you use:

oc new-app https://raw.githubusercontent.com/openshift-labs/workshop-spawner/develop/templates/jumpbox-server-development.json

just then, it will work because the build configuration was switched to a docker build instead of source build. Instead use:

oc new-app https://raw.githubusercontent.com/openshift-labs/workshop-spawner/3.0.4/templates/jumpbox-server-development.json

if you are trying to replicate it. That is use version 3.0.4, which has setup for source build.

There was confusion on my part initially because I thought it was using a docker build, when it was actually using a source build.

Comment 6 Adam Kaplan 2019-04-09 12:56:19 UTC
Builder PR: https://github.com/openshift/builder/pull/61

Comment 7 Adam Kaplan 2019-04-11 12:55:13 UTC
Additional test cases needed in origin, but this is ready for QE.

Comment 9 wewang 2019-04-12 07:16:06 UTC
Verified in 
4.0.0-0.ci-2019-04-11-185255
payload: registry.svc.ci.openshift.org/ocp/release@sha256:fdeeee0c19bd7b5873744dacf5859ac8adf0850961b7a449db839068f5ce7aef

Comment 11 errata-xmlrpc 2019-06-04 10:47:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758