Created attachment 1622400 [details] build config on OCP 4 Description of problem: As stated in the docs, to have custom labels in the s2i builds, it's required to have the file "image_metadata.json" in this path "/tmp/.s2i/image_metadata.json" This works perfectly on OCP 3.x, but fails on OCP 4.x (no custom label added). It seems that the s2i is ignoring this file. Also works locally with the s2i command line. Version-Release number of selected component (if applicable): 4.1.16 How reproducible: Always Steps to Reproduce: 1. Deploy the reproducer into a OpenShift 4.x cluster: ~~~ oc new-app https://github.com/ricardozanini/tenkichannel --build-env MAVEN_ARGS_APPEND="-pl rain-forecast-process -am" --build-env ARTIFACT_DIR="rain-forecast-process/target" --build-env JAVA_APP_JAR="rain-forecast-process-1.0.0-SNAPSHOT-runner.jar" --strategy=source --docker-image=openshift/openjdk-11-rhel8:latest ~~~ (make sure to use this image: https://access.redhat.com/containers/?tab=overview#/registry.access.redhat.com/openjdk/openjdk-11-rhel8) 2. Wait for the build to finish 3.Inspect the imagestreamtag, in the dockerImageMetadata.Config.Labels field, you won't see the custom label described in the "image_metadata.json" file: ~~~ Labels: org.jboss.container.deployments-dir: /deployments vcs-type: git usage: >- https://access.redhat.com/documentation/en-us/red_hat_jboss_middleware_for_openshift/3/html/red_hat_java_s2i_for_openshift/ io.openshift.build.commit.author: Ricardo Zanini <zanini> io.k8s.display-name: Java Applications com.redhat.component: openjdk-11-rhel8-container summary: Source To Image (S2I) image for Red Hat OpenShift providing OpenJDK 11 release: '13' io.openshift.build.commit.date: 'Thu Oct 3 15:53:10 2019 -0300' io.openshift.s2i.scripts-url: 'image:///usr/local/s2i' vendor: 'Red Hat, Inc.' io.openshift.expose-services: '' io.openshift.build.source-location: 'https://github.com/ricardozanini/tenkichannel' distribution-scope: public name: openjdk/openjdk-11-rhel8 vcs-ref: e742320ec83194f7e012063f664054884c33c3cf io.openshift.s2i.destination: /tmp io.k8s.description: >- Platform for building and running plain Java applications (fat-jar and flat classpath) build-date: '2019-09-19T14:57:20.411924' io.fabric8.s2i.version.maven: '3.5' org.jboss.product: openjdk io.openshift.build.commit.message: minor changes url: >- https://access.redhat.com/containers/#/registry.access.redhat.com/openjdk/openjdk-11-rhel8/images/1.0-13 org.jboss.product.openjdk.version: '11.0' com.redhat.license_terms: 'https://www.redhat.com/en/about/red-hat-end-user-license-agreements' architecture: x86_64 io.openshift.build.commit.id: 7e4ea89b9413003dc294beba56e828b0913247ab io.cekit.version: 3.4.0 io.openshift.tags: 'builder,java' version: '1.0' org.jboss.product.version: '11.0' maintainer: 'Red Hat, Inc.' io.openshift.build.commit.ref: master authoritative-source-url: registry.access.redhat.com description: Source To Image (S2I) image for Red Hat OpenShift providing OpenJDK 11 io.openshift.build.image: >- image-registry.openshift-image-registry.svc:5000/openshift/openjdk-11-rhel8 io.fabric8.s2i.version.jolokia: 1.5.0-redhat-1 com.redhat.build-host: cpt-1002.osbs.prod.upshift.rdu2.redhat.com ~~~ This same steps can be reproduced in a OCP 3.x version to see the label being added. Actual results: No custom labels are added to the final image on OCP 4.1 builds. Expected results: Labels being added Additional info: One can use this repo for testing: https://github.com/ricardozanini/tenkichannel This is the command line for running locally: (clone the repo locally first) ~~~ s2i build . openjdk/openjdk-11-rhel8 quay.io/ricardozanini/rain-forecast-process:1.0.0 -e MAVEN_ARGS_APPEND="-pl rain-forecast-process -am" -e ARTIFACT_DIR="rain-forecast-process/target" -e JAVA_APP_JAR="rain-forecast-process-1.0.0-SNAPSHOT-runner.jar" ~~~ Then use "docker inspect", you should see a custom label named "org.tenkichannel/service":"rain-forecast-process" Attached the build config logs in the OCP 4.x
Forgot to mention the command to run locally with s2i command line tool: ~~~ s2i build . openjdk/openjdk-11-rhel8 rain-forecast-process -e MAVEN_ARGS_APPEND="-pl rain-forecast-process -am" -e ARTIFACT_DIR="rain-forecast-process/target" -e JAVA_APP_JAR="rain-forecast-process-1.0.0-SNAPSHOT-runner.jar" ~~~
Updating the reproducer command: ~~~ oc new-app https://github.com/ricardozanini/tenkichannel --context-dir="rain-forecast-process" --strategy=source --docker-image=openshift/openjdk-11-rhel8:latest ~~~ Locally: ~~~ s2i build rain-forecast-process/ openjdk/openjdk-11-rhel8 rain-forecast-process ~~~
OK confirmed that the "image_metadata.json" file is only used during s2i's commit image post executor step for labelling the image, so it would have been lost in the 3.x to 4.x move off of docker and use of s2i's as-dockerfile support. I was able to reproduce it with an s2i invocation with some tweaks based on Ricardo's oc new-app invocation: gmontero ~/go/src/github.com/openshift/source-to-image (master)$ s2i build https://github.com/ricardozanini/tenkichannel gmontero/openjdk-11-rhel8 --context-dir="rain-forecast-process" --as-dockerfile=./ggm Application dockerfile generated in ./ggm gmontero ~/go/src/github.com/openshift/source-to-image (master)$ cat ./ggm FROM gmontero/openjdk-11-rhel8 LABEL "io.openshift.s2i.build.image"="gmontero/openjdk-11-rhel8" \ "io.openshift.s2i.build.commit.author"="Ricardo Zanini <zanini>" \ "io.openshift.s2i.build.commit.date"="Mon Oct 21 21:06:13 2019 -0300" \ "io.openshift.s2i.build.commit.id"="7be7d8e060f1d8ff6606063283a7ec5c52ac3716" \ "io.openshift.s2i.build.commit.ref"="master" \ "io.openshift.s2i.build.commit.message"="Fix #10" \ "io.openshift.s2i.build.source-location"="https://github.com/ricardozanini/tenkichannel" \ "io.openshift.s2i.build.source-context-dir"="rain-forecast-process" USER root # Copying in source code COPY upload/src /tmp/src # Change file ownership to the assemble user. Builder image must support chown command. RUN chown -R 1001:0 /tmp/src USER 1001 # Assemble script sourced from builder image based on user input or image metadata. # If this file does not exist in the image, the build will fail. RUN /usr/libexec/s2i/assemble # Run script sourced from builder image based on user input or image metadata. # If this file does not exist in the image, the build will fail. CMD /usr/libexec/s2i/run gmontero ~/go/src/github.com/openshift/source-to-image (master)$ I'm hopeful we can add use of it in https://github.com/openshift/source-to-image/blob/master/pkg/util/labels.go for --as-dockerfile
I now get this dockerfile: gmontero ~/go/src/github.com/openshift/source-to-image (image_metadata_labels)$ cat ggm FROM gmontero/openjdk-11-rhel8 LABEL "io.openshift.s2i.build.commit.author"="Ricardo Zanini <zanini>" \ "io.openshift.s2i.build.commit.ref"="master" \ "io.openshift.s2i.build.source-location"="https://github.com/ricardozanini/tenkichannel" \ "org.tenkichannel/service"="rain-forecast-process" \ "io.openshift.s2i.build.image"="gmontero/openjdk-11-rhel8" \ "io.openshift.s2i.build.commit.date"="Mon Oct 21 21:06:13 2019 -0300" \ "io.openshift.s2i.build.commit.id"="7be7d8e060f1d8ff6606063283a7ec5c52ac3716" \ "io.openshift.s2i.build.commit.message"="Fix #10" \ "io.openshift.s2i.build.source-context-dir"="rain-forecast-process" USER root # Copying in source code COPY upload/src /tmp/src # Change file ownership to the assemble user. Builder image must support chown command. RUN chown -R 1001:0 /tmp/src USER 1001 # Assemble script sourced from builder image based on user input or image metadata. # If this file does not exist in the image, the build will fail. RUN /usr/libexec/s2i/assemble # Run script sourced from builder image based on user input or image metadata. # If this file does not exist in the image, the build will fail. CMD /usr/libexec/s2i/run "org.tenkichannel/service"="rain-forecast-process" is from Ricardo's image_metadata.json Note, after the source-to-image PR merges, we'll need a bump of s2i in https://github.com/openshift/builder for the oc new-app induced build to work as expected.
Many thanks for this, Gabe! Please also note that BuildConfigs with SourceStrategy also have to work with this new patch. I assume it will since the container used for sti has the s2i binary installed. Also, in the docs [1] the JSON format file follows this pattern: ``` { "labels": [ {"labelkey1":"value1"}, {"labelkey2":"value2"}, ......... ] } ``` But I see some image_metadata.json in the following format as well (working on 3.11): ``` { "labels": [ { "labelkey1" : "value1", "labelkey2" : "value2" } ] } ``` Could you please confirm if both formats will work (maybe adding a new test case to your PR)? [1] https://github.com/openshift/source-to-image/blob/master/docs/new_labels.md#example
Both formats work @Ricardo. I've added both cases to the new unit tests in the PR noted above. Also, my note above in reference to bumping openshift/builder is what will facilitate this working with Source Strategy build configs.
Thanks for the heads up, @Gabe.
Verified in version: 4.3.0-0.nightly-2019-11-02-092336 s2i v1.2.0-12-g1877e115 $s2i build https://github.com/ricardozanini/tenkichannel gmontero/openjdk-11-rhel8 --context-dir="rain-forecast-process" --as-dockerfile=./ggm FROM gmontero/openjdk-11-rhel8 LABEL "io.openshift.s2i.build.commit.message"="Fix #10" \ "org.tenkichannel/service"="rain-forecast-process" \ "io.openshift.s2i.build.commit.id"="7be7d8e060f1d8ff6606063283a7ec5c52ac3716" \ "io.openshift.s2i.build.commit.ref"="master" \ "io.openshift.s2i.build.source-location"="https://github.com/ricardozanini/tenkichannel" \ "io.openshift.s2i.build.source-context-dir"="rain-forecast-process" \ # "org.tenkichannel/service"="rain-forecast-process" is from Ricardo's image_metadata.json "io.openshift.s2i.build.image"="gmontero/openjdk-11-rhel8" \ "io.openshift.s2i.build.commit.author"="Ricardo Zanini <zanini>" \ "io.openshift.s2i.build.commit.date"="Mon Oct 21 21:06:13 2019 -0300" USER root # Copying in source code COPY upload/src /tmp/src # Change file ownership to the assemble user. Builder image must support chown command. RUN chown -R 1001:0 /tmp/src USER 1001 # Assemble script sourced from builder image based on user input or image metadata. # If this file does not exist in the image, the build will fail. RUN /usr/libexec/s2i/assemble # Run script sourced from builder image based on user input or image metadata. # If this file does not exist in the image, the build will fail. CMD /usr/libexec/s2i/run
I have hit an issue while checking this fix. The reproducer provided by Ricardo works in fixed version, however this reproducer is placing "image_metadata.json" file into wrong folder - "/tmp/src/.s2i/image_metadata.json" According to [1] the correct place for "image_metadata.json" is "/tmp/.s2i/image_metadata.json". If image_metadata.json is placed into this folder then the labels are not processed by S2I build and are missing in built image. To reproduce this issue you can use this command: ~~~ oc new-app https://github.com/kiegroup/kogito-examples --context-dir="onboarding-example/hr" --strategy=source --docker-image=quay.io/kiegroup/kogito-quarkus-ubi8-s2i:0.5.1 ~~~ Expected image label and value is for example "prometheus.io/path" : "/metrics". [1] https://github.com/openshift/source-to-image/blob/master/docs/new_labels.md
I just tried oc new-app https://github.com/ricardozanini/tenkichannel --context-dir="rain-forecast-process" --strategy=source --docker-image=gmontero/openjdk-11-rhel8:latest and see the label from the image_metadata.json file ("org.tenkichannel/service":"rain-forecast-process") added to the generated dockerfile: STEP 2: LABEL "io.openshift.build.image"="gmontero/openjdk-11-rhel8@sha256:8d2177bdfc307bb023ce8b9438f2c1d4bd024fb15049a50e2672aa8c7fa69745" "io.openshift.build.commit.author"="Ricardo Zanini <1538000+ricardozanini.github.com>" "io.openshift.build.commit.id"="712bb849ceb6df820c2f6d88015f4dd529a8b6a4" "io.openshift.build.source-location"="https://github.com/ricardozanini/tenkichannel" "org.tenkichannel/service"="rain-forecast-process" "io.openshift.build.commit.date"="Wed Nov 27 12:01:31 2019 -0300" "io.openshift.build.commit.ref"="master" "io.openshift.build.commit.message"="bump quarkus, kogito and camel / fixing yahoo weather bug (#13)" "io.openshift.build.source-context-dir"="rain-forecast-process" where the image_metadata.json file is located directly off of the context root (i.e. no "src" dir): https://github.com/ricardozanini/tenkichannel/blob/master/rain-forecast-process/.s2i/image_metadata.json I git cloned https://github.com/kiegroup/kogito-examples and see no image_metadata.json file We are continuing to work this in the bug I had Karel open: https://bugzilla.redhat.com/show_bug.cgi?id=1780622
Per https://bugzilla.redhat.com/show_bug.cgi?id=1780622#c5 Ben wants us to revert the changes done with this bug. Starting that process. Subsequently, doc updates in the interim noting the lack of 4.x support will be tracked with this bug. Open Jira feature requests for further debate on implementing the feature. Note: output labels can also be set via https://docs.openshift.com/container-platform/4.2/builds/managing-build-output.html#builds-output-image-labels_managing-build-output
*** Bug 1780622 has been marked as a duplicate of this bug. ***
So we need to update https://docs.openshift.com/container-platform/4.2/builds/managing-build-output.html#builds-output-image-labels_managing-build-output to note that the s2i command line feature noted at https://github.com/openshift/source-to-image/blob/master/docs/new_labels.md is currently not supported in 4.x See Ben's comments in https://bugzilla.redhat.com/show_bug.cgi?id=1780622 if you want details.
Gabe, Ben Could you guys please clarify what would be the official way to set custom labels at build time in a similar way we have for ocp 3.11? If my understanding is correct, by using buildconfig.spec.output.imageLabels we can't update it during build time (assemble) automatically, it needs to be updated by a operator or manually, is that correct? Or will be the buildconfig.spec.output.imageLabels label the only supported option?
> Could you guys please clarify what would be the official way to set custom labels at build time in a similar way we have for ocp 3.11? unfortunately it's currently a gap in what is possible with v4. > Or will be the buildconfig.spec.output.imageLabels label the only supported option? for now it's the only way, but i think we'd strongly consider an RFE to bring the "dynamic" behavior back if you have a use case. Just be aware that it's not a trivial rfe unfortunately.
So, Gabe, the fix of this bug will be add a doc to say buildconfig.spec.output.imageLabels label is the only supported option?
Yes Wenjing, to reiterate what Ben noted #Comment 20
*** Bug 1823362 has been marked as a duplicate of this bug. ***