Description of problem: S2i Incremental build does not behave incrementally. Our OCP 3.11 test stop working on OCP 4.1. Second build of incremental build config still download all dependencies. On OCP 3.11 build was skipped. Version-Release number of selected component (if applicable): OCP 4.1 How reproducible: Steps to Reproduce: oc login url oc project mchoma oc create -f ./build_config.yaml # Build first time oc start-build --from-dir=./sti_app sti-bc # wait until Status: Completed # maven is downloading dependencies as building app oc describe build.build.openshift.io/sti-bc-2 # -o offline mode means maven should not contact repository online. When incremental build work correctly ehating maven repo should be not necessary echo "MAVEN_ARGS_APPEND=-o" >> ./stii_app/.s2i/environment # build again # there should be no dependency downloading oc start-build --from-dir=./sti_app sti-bc oc describe build.build.openshift.io/sti-bc-3 | grep Status: # Return echo "" >> ./stii_app/.s2i/environment Actual results: Status: Failed (Generic Build failure - check logs for details.) Expected results: Status: Completed Additional info:
Please provide the BuildConfig used for your test, along the verbose build logs from the build (oc start-build --build-loglevel=5)
Created attachment 1560300 [details] reproducer
Created attachment 1560301 [details] incremental-build-OCP-4-1.log
In the registry.access.redhat.com/jboss-eap-7/eap72-openshift:1.0 image, I looked at the /usr/local/s2i/assemble script, which calls the maven_s2i_build() function from /opt/jboss/container/maven/s2i/maven-s2i, which I have listed below. To me, it looks like the function that restores the artifacts is (s2i_core_copy_artifacts) is after the function(s) that perform the maven build. Is the version of the image that you are using with 4.1 different than the one you were using with 3.11? If so, could you please point me towards the version that you were using with 3.11? It might also be useful if you could provide your build logs with a loglevel of 5 which will provide more details about what is going on. # main entry point, perform the build function maven_s2i_build() { maven_s2i_init if [ -f "${S2I_SOURCE_DIR}/pom.xml" ]; then # maven build maven_s2i_maven_build else # binary build maven_s2i_binary_build fi s2i_core_copy_artifacts "${S2I_SOURCE_DIR}" s2i_core_process_image_mounts s2i_core_cleanup # Remove java tmp perf data dir owned by 185 rm -rf /tmp/hsperfdata_jboss }
From what I understand, it is the responsibility of the scripts in the image to check for and restore artifacts that are saved by the save-artifacts script. Example: https://github.com/openshift/source-to-image/blob/master/examples/nginx-centos7/s2i/bin/assemble
Created attachment 1561320 [details] incremental-build-OCP-3-11.log
Created attachment 1561338 [details] incremental-build-ocp-3-11-level-5.log
Created attachment 1561490 [details] incremental-build-ocp-4-1-level-5.log
Images are same for OCP 3.11 and OCP 4.1. I have attached logs of loglevel 5 for both OCPs
What have you learned about this so far, Corey?
On 4.1 this error is coming up (also coming up for me with a simpler test case) I am working on tracking down the cause, but this is what is not allowing the incremental build. Explicitly pulling image image-registry.openshift-image-registry.svc:5000/mchoma/sti:latest Asked to pull fresh copy of "image-registry.openshift-image-registry.svc:5000/mchoma/sti:latest". Warning: Pull failed, retrying in 5s ... Asked to pull fresh copy of "image-registry.openshift-image-registry.svc:5000/mchoma/sti:latest". Warning: Pull failed, retrying in 5s ... Asked to pull fresh copy of "image-registry.openshift-image-registry.svc:5000/mchoma/sti:latest". Warning: Pull failed, retrying in 5s ... Failed to pull incremental builder image image-registry.openshift-image-registry.svc:5000/mchoma/sti:latest - executing normal s2i build instead.
I am currently testing a fix for this and should have a pull request up soon.
Submitted pull request https://github.com/openshift/builder/pull/70 and am working on an update to the s2i_incremental extended test in origin to prevent this regression in the future.
moving to post since it appears the PR has been submitted and reviewed
Submitted pull request https://github.com/openshift/origin/pull/22793 to update the s2i incremental test to more accurately reflect how an incremental build should work.
Verified with payload 4.1.0-0.nightly-2019-05-07-233329. The incremental build don't download dependencies, and don't meet auth error. $ oc get builds NAME TYPE FROM STATUS STARTED DURATION sti-bc-2 Source Binary Complete 24 minutes ago 2m37s sti-bc-3 Source Binary Complete 16 minutes ago 1m39s $oc build-logs sti-bc-3 ===============snip================ [INFO] Error stacktraces are turned on. [INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] JBoss EAP Quickstart: ejb-in-ear [INFO] JBoss EAP Quickstart: ejb-in-ear - ejb [INFO] JBoss EAP Quickstart: ejb-in-ear - web [INFO] JBoss EAP Quickstart: ejb-in-ear - ear [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building JBoss EAP Quickstart: ejb-in-ear 1.0.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building JBoss EAP Quickstart: ejb-in-ear - ejb 1.0.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ eap-sti-ejb --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /tmp/src/ejb/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ eap-sti-ejb --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 1 source file to /tmp/src/ejb/target/classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ eap-sti-ejb --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /tmp/src/ejb/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.3:testCompile (default-testCompile) @ eap-sti-ejb --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 1 source file to /tmp/src/ejb/target/test-classes ===============snip================
I have tried 4.1.0-rc.1, but seems fix is not there.
I can confirm I dont see the issue in oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.1.0-rc.3 True False 4h36m Cluster version is 4.1.0-rc.3
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758