Bug 1704767 - s2i incremental build always download dependencies
Summary: s2i incremental build always download dependencies
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.1.0
Assignee: Corey Daley
QA Contact: wewang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-30 13:12 UTC by mchoma
Modified: 2019-06-04 10:48 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-04 10:48:18 UTC
Target Upstream Version:


Attachments (Terms of Use)
reproducer (20.02 KB, application/zip)
2019-04-30 13:24 UTC, mchoma
no flags Details
incremental-build-OCP-4-1.log (71.24 KB, application/octet-stream)
2019-04-30 13:33 UTC, mchoma
no flags Details
incremental-build-OCP-3-11.log (6.58 KB, application/octet-stream)
2019-05-02 05:57 UTC, mchoma
no flags Details
incremental-build-ocp-3-11-level-5.log (1.20 MB, text/plain)
2019-05-02 06:42 UTC, mchoma
no flags Details
incremental-build-ocp-4-1-level-5.log (149.29 KB, text/plain)
2019-05-02 10:17 UTC, mchoma
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 None None None 2019-06-04 10:48:24 UTC

Description mchoma 2019-04-30 13:12:12 UTC
Description of problem:
S2i Incremental build does not behave incrementally. Our OCP 3.11 test stop working on OCP 4.1. Second build of incremental build config still download all dependencies. On OCP 3.11 build was skipped.

Version-Release number of selected component (if applicable):
OCP 4.1

How reproducible:


Steps to Reproduce:
oc login url
oc project mchoma
oc create -f ./build_config.yaml

# Build first time
oc start-build --from-dir=./sti_app sti-bc
# wait until Status: Completed
# maven is downloading dependencies as building app
oc describe build.build.openshift.io/sti-bc-2

# -o offline mode means maven should not contact repository online. When incremental build work correctly ehating maven repo  should be not necessary
echo "MAVEN_ARGS_APPEND=-o" >> ./stii_app/.s2i/environment

# build again
# there should be no dependency downloading
oc start-build --from-dir=./sti_app sti-bc

oc describe build.build.openshift.io/sti-bc-3 | grep Status:

# Return 
echo "" >> ./stii_app/.s2i/environment

Actual results:
Status:         Failed (Generic Build failure - check logs for details.)

Expected results:
Status: Completed

Additional info:

Comment 1 Adam Kaplan 2019-04-30 13:21:05 UTC
Please provide the BuildConfig used for your test, along the verbose build logs from the build (oc start-build --build-loglevel=5)

Comment 2 mchoma 2019-04-30 13:24:50 UTC
Created attachment 1560300 [details]
reproducer

Comment 3 mchoma 2019-04-30 13:33:17 UTC
Created attachment 1560301 [details]
incremental-build-OCP-4-1.log

Comment 4 Corey Daley 2019-05-01 14:37:35 UTC
In the registry.access.redhat.com/jboss-eap-7/eap72-openshift:1.0 image, I looked at the /usr/local/s2i/assemble script, which calls the maven_s2i_build() function from /opt/jboss/container/maven/s2i/maven-s2i, which I have listed below.  To me, it looks like the function that restores the artifacts is (s2i_core_copy_artifacts) is after the function(s) that perform the maven build.

Is the version of the image that you are using with 4.1 different than the one you were using with 3.11?
If so, could you please point me towards the version that you were using with 3.11?

It might also be useful if you could provide your build logs with a loglevel of 5 which will provide more details about what is going on.

# main entry point, perform the build
function maven_s2i_build() {
  maven_s2i_init
  if [ -f "${S2I_SOURCE_DIR}/pom.xml" ]; then
    # maven build
    maven_s2i_maven_build
  else
    # binary build
    maven_s2i_binary_build
  fi
  s2i_core_copy_artifacts "${S2I_SOURCE_DIR}"

  s2i_core_process_image_mounts

  s2i_core_cleanup

  # Remove java tmp perf data dir owned by 185
  rm -rf /tmp/hsperfdata_jboss
}

Comment 5 Corey Daley 2019-05-01 14:41:01 UTC
From what I understand, it is the responsibility of the scripts in the image to check for and restore artifacts that are saved by the save-artifacts script.  
Example: https://github.com/openshift/source-to-image/blob/master/examples/nginx-centos7/s2i/bin/assemble

Comment 6 mchoma 2019-05-02 05:57:59 UTC
Created attachment 1561320 [details]
incremental-build-OCP-3-11.log

Comment 7 mchoma 2019-05-02 06:42:05 UTC
Created attachment 1561338 [details]
incremental-build-ocp-3-11-level-5.log

Comment 8 mchoma 2019-05-02 10:17:25 UTC
Created attachment 1561490 [details]
incremental-build-ocp-4-1-level-5.log

Comment 9 mchoma 2019-05-02 10:18:41 UTC
Images are same for OCP 3.11 and OCP 4.1. I have attached logs of loglevel 5 for both OCPs

Comment 10 Ben Parees 2019-05-02 21:38:54 UTC
What have you learned about this so far, Corey?

Comment 11 Corey Daley 2019-05-03 19:57:44 UTC
On 4.1 this error is coming up (also coming up for me with a simpler test case)
I am working on tracking down the cause, but this is what is not allowing the incremental build.


Explicitly pulling image image-registry.openshift-image-registry.svc:5000/mchoma/sti:latest
Asked to pull fresh copy of "image-registry.openshift-image-registry.svc:5000/mchoma/sti:latest".
Warning: Pull failed, retrying in 5s ...
Asked to pull fresh copy of "image-registry.openshift-image-registry.svc:5000/mchoma/sti:latest".
Warning: Pull failed, retrying in 5s ...
Asked to pull fresh copy of "image-registry.openshift-image-registry.svc:5000/mchoma/sti:latest".
Warning: Pull failed, retrying in 5s ...
Failed to pull incremental builder image image-registry.openshift-image-registry.svc:5000/mchoma/sti:latest - executing normal s2i build instead.

Comment 12 Corey Daley 2019-05-07 15:41:08 UTC
I am currently testing a fix for this and should have a pull request up soon.

Comment 13 Corey Daley 2019-05-07 16:10:35 UTC
Submitted pull request https://github.com/openshift/builder/pull/70 and am working on an update to the s2i_incremental extended test in origin to prevent this regression in the future.

Comment 14 Eric Paris 2019-05-07 17:07:53 UTC
moving to post since it appears the PR has been submitted and reviewed

Comment 15 Corey Daley 2019-05-07 17:29:44 UTC
Submitted pull request https://github.com/openshift/origin/pull/22793 to update the s2i incremental test to more accurately reflect how an incremental build should work.

Comment 17 XiuJuan Wang 2019-05-08 09:03:55 UTC
Verified with payload 4.1.0-0.nightly-2019-05-07-233329.
The incremental build don't download dependencies, and don't meet auth error.
$ oc get builds 
NAME       TYPE     FROM     STATUS     STARTED          DURATION
sti-bc-2   Source   Binary   Complete   24 minutes ago   2m37s
sti-bc-3   Source   Binary   Complete   16 minutes ago   1m39s
 
$oc build-logs sti-bc-3
===============snip================
[INFO] Error stacktraces are turned on.
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] JBoss EAP Quickstart: ejb-in-ear
[INFO] JBoss EAP Quickstart: ejb-in-ear - ejb
[INFO] JBoss EAP Quickstart: ejb-in-ear - web
[INFO] JBoss EAP Quickstart: ejb-in-ear - ear
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Building JBoss EAP Quickstart: ejb-in-ear 1.0.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Building JBoss EAP Quickstart: ejb-in-ear - ejb 1.0.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ eap-sti-ejb ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /tmp/src/ejb/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ eap-sti-ejb ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /tmp/src/ejb/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ eap-sti-ejb ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /tmp/src/ejb/src/test/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.3:testCompile (default-testCompile) @ eap-sti-ejb ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /tmp/src/ejb/target/test-classes
===============snip================

Comment 18 mchoma 2019-05-10 05:28:53 UTC
I have tried 4.1.0-rc.1, but seems fix is not there.

Comment 20 mchoma 2019-05-13 11:57:31 UTC
I can confirm I dont see the issue in 

oc get clusterversion
NAME      VERSION      AVAILABLE   PROGRESSING   SINCE     STATUS
version   4.1.0-rc.3   True        False         4h36m     Cluster version is 4.1.0-rc.3

Comment 22 errata-xmlrpc 2019-06-04 10:48:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.