Bug 1788982

Summary: OCP 4.3 Copying directory in DockerFile BuildConfig raise from 1 second to 10 minutes.
Product: OpenShift Container Platform Reporter: mchoma
Component: ContainersAssignee: Nalin Dahyabhai <nalin>
Status: CLOSED ERRATA QA Contact: weiwei jiang <wjiang>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.3.0CC: aos-bugs, ccoleman, dwalsh, jokerman, mokumar, nalin, pkremens, pthomas, skuznets, tsweeney, wking, wzheng
Target Milestone: ---Keywords: Regression
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1790525 (view as bug list) Environment:
Last Closed: 2020-05-04 11:23:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1790525    

Description mchoma 2020-01-08 14:00:11 UTC
Description of problem:
In EAP templates there is 2 step build process. In first step s2i build prepare application and server and in the second step in DockerFile BuilConfig server with application is copied into runtime image.

Now with OCP 4.3 this "copy server" step takes 10 minutes. Which raise time for our testsuite from ~11 hours to ~20 hours. We consider this for regression and blocker for us.

copying took 10 minutes on OCP 4.3
2020-01-08T12:51:17.424137378Z STEP 2: COPY /server $JBOSS_HOME
2020-01-08T13:01:19.831516593Z STEP 3: USER root

copying took 1 second on OCP 4.2
2020-01-08T13:25:03.029938746Z STEP 2: COPY /server $JBOSS_HOME
2020-01-08T13:25:04.628606937Z STEP 3: USER root


Version-Release number of selected component (if applicable):
OCP 4.3 version 4.3.0-0.nightly-2020-01-06-101556


How reproducible:
Always


Steps to Reproduce:
1. Install template to use
oc replace --force -f https://raw.githubusercontent.com/jboss-container-images/jboss-eap-7-openshift-image/eap-cd-dev/templates/eap-cd-basic-s2i.json

2. Create App using this template

oc new-app --template=eap-cd-basic-s2i \
 -p IMAGE_STREAM_NAMESPACE=mchoma \
 -p SOURCE_REPOSITORY_URL=https://github.com/jboss-developer/jboss-eap-quickstarts \
 -p SOURCE_REPOSITORY_REF=openshift \
 -p CONTEXT_DIR=kitchensink

3. Check eap-app-build-artifacts-1 and eap-app-2 to be in Status Complete

watch oc get buildsActual results:


Actual results:
Duration of eap-app-build-artifacts-1 and eap-app-2 is ~15 minutes

Expected results:
Duration of eap-app-build-artifacts-1 and eap-app-2 should be ~5 minutes

Comment 2 Tom Sweeney 2020-01-08 15:57:17 UTC
Nalin might this be related to https://github.com/containers/buildah/issues/2067 and/or https://github.com/containers/buildah/issues/2047?

Comment 4 Nalin Dahyabhai 2020-01-10 21:02:18 UTC
aOpened #126 (linked above) to incorporate an buildah which should address this for the 4.4 branch, and https://github.com/openshift/builder/pull/127 for the same changes to the 4.3 branch.

Comment 7 Clayton Coleman 2020-02-07 19:07:33 UTC
Note that this is an urgent product regression, setting severity to urgent.  This should take priority over feature work.

Comment 8 Steve Kuznetsov 2020-02-07 19:13:21 UTC
Adding some colour to this on the impact it has on builds...


Before:
src                   Docker   Dockerfile   Complete   36 minutes ago   10m2s
installer             Docker                Complete   23 minutes ago   6m25s
libvirt-installer     Docker                Complete   22 minutes ago   11m30s
openstack-installer   Docker                Complete   19 minutes ago   6m43s
ovirt-installer       Docker                Complete   21 minutes ago   11m4s
baremetal-installer   Docker                Complete   22 minutes ago   6m14s
upi-installer         Docker                Complete   23 minutes ago   10m52s
installer-artifacts   Docker                Complete   14 minutes ago   11m58s
--------------------------------------------------------------------------------
After:
NAME                  TYPE     FROM         STATUS     STARTED             DURATION
src                   Docker   Dockerfile   Complete   About an hour ago   1m41s
installer             Docker                Complete   About an hour ago   34m43s
libvirt-installer     Docker                Complete   About an hour ago   35m36s
openstack-installer   Docker                Complete   About an hour ago   34m42s
ovirt-installer       Docker                Complete   About an hour ago   34m39s
baremetal-installer   Docker                Complete   About an hour ago   36m12s
upi-installer         Docker                Complete   About an hour ago   36m28s
installer-artifacts   Docker                Running    16 minutes ago 


==============

Before:
src                      Docker   Dockerfile   Complete   2 hours ago   4m56s
bin                      Docker   Dockerfile   Complete   2 hours ago   3m49s
tests                    Docker                Complete   2 hours ago   5m39s
hyperkube                Docker                Complete   2 hours ago   3m4s
rpms                     Docker   Dockerfile   Complete   2 hours ago   3m8s
base-machine-with-rpms   Docker   Dockerfile   Complete   2 hours ago   28s
machine-os-content       Docker                Complete   2 hours ago   6m53s
--------------------------------------------------------------------------------
After:
src                      Docker   Dockerfile   Complete   20 hours ago   5m25s
bin                      Docker   Dockerfile   Complete   20 hours ago   1m37s
rpms                     Docker   Dockerfile   Complete   20 hours ago   4m27s
bin-cross                Docker   Dockerfile   Complete   20 hours ago   4m16s
hyperkube                Docker                Complete   20 hours ago   1h57m36s
tests                    Docker                Complete   20 hours ago   1h59m49s
artifacts                Docker                Complete   20 hours ago   4m27s
base-machine-with-rpms   Docker   Dockerfile   Complete   20 hours ago   56s
machine-os-content       Docker                Complete   3 hours ago    1h25m8s

Regressing from a 3min build to a 2h build renders Builds basically unsuable. This blocks migration onto 4.x for the CI infrastructure.

Comment 11 mchoma 2020-02-10 07:10:06 UTC
Matching Image stream is here https://raw.githubusercontent.com/jboss-container-images/jboss-eap-7-openshift-image/eap-cd-dev/templates/eap-cd-image-stream.json

4.4 we will try during standard Layered product testing.

Comment 12 weiwei jiang 2020-02-11 03:33:52 UTC
Checked with 4.4.0-0.nightly-2020-02-09-220310, all the Duration are under 5 minutes, verified.

$ oc get build         
NAME                        TYPE     FROM          STATUS     STARTED         DURATION
eap-app-build-artifacts-1   Source   Git@6eaf5ee   Complete   7 minutes ago   4m54s
eap-app-2                   Docker   Dockerfile    Complete   2 minutes ago   2m49s

$ oc new-app --template=eap-cd-basic-s2i \                                                                                                                                                                       
 -p IMAGE_STREAM_NAMESPACE=default \                                                                                                                                                                               
 -p SOURCE_REPOSITORY_URL=https://github.com/jboss-developer/jboss-eap-quickstarts \                                                                                                                               
 -p SOURCE_REPOSITORY_REF=openshift \                                                                                                                                                                              
 -p CONTEXT_DIR=kitchensink                                                                                                                                                                                        
                                                                                                                                                                                                                   
--> Deploying template "default/eap-cd-basic-s2i" to project default                                                                                                                                               
                                                                                                                                                                                                                   
     JBoss EAP CD 18                                                                                                                                                                                               
     ---------                                                                                                                                                                                                     
     An example JBoss Enterprise Application Platform continuous delivery application. For more information about using this template, see https://github.com/jboss-container-images/jboss-eap-7-openshift-image/bl
ob/eap-cd/README.adoc                                                                                                                                                                                              
                                                                                                                                                                                                                   
     A new JBoss EAP CD based application has been created in your project.                                                                                                                                        
                                                                                                                                                                                                                   
     * With parameters:                                                                                                                                                                                            
        * Application Name=eap-app                                                                                                                                                                                 
        * Git Repository URL=https://github.com/jboss-developer/jboss-eap-quickstarts                                                                                                                              
        * Git Reference=openshift                                                                                                                                                                                  
        * Context Directory=kitchensink                                                                                                                                                                            
        * Galleon layers=                                                                                                                                                                                          
        * Enable ExampleDS datasource=false                                                                                                                                                                        
        * Queues=                                                                                                                                                                                                  
        * Topics=                                                                                                                                                                                                  
        * AMQ cluster password=QPyGOL7f # generated                                                                                                                                                                
        * Github Webhook Secret=YKTrbYVi # generated                                                                                                                                                               
        * Generic Webhook Secret=xgJ2SCcl # generated                                                                                                                                                              
        * ImageStream Namespace=default                                                                                                                                                                            
        * JGroups Cluster Password=nqJmC1uf # generated
        * Deploy Exploded Archives=false
        * Maven mirror URL=
        * Maven Additional Arguments=-Dcom.redhat.xpaas.repo.jbossorg
        * ARTIFACT_DIR=
        * MEMORY_LIMIT=1Gi

--> Creating resources ...
    service "eap-app" created
    service "eap-app-ping" created
    route.route.openshift.io "eap-app" created
    imagestream.image.openshift.io "eap-app" created 
    imagestream.image.openshift.io "eap-app-build-artifacts" created
    buildconfig.build.openshift.io "eap-app-build-artifacts" created
    buildconfig.build.openshift.io "eap-app" created 
    deploymentconfig.apps.openshift.io "eap-app" created
--> Success
    Access your application via route 'eap-app-default.apps.wjaws0210.qe.devcluster.openshift.com' 
    Build scheduled, use 'oc logs -f bc/eap-app-build-artifacts' to track its progress.
    Build scheduled, use 'oc logs -f bc/eap-app' to track its progress.
    Run 'oc status' to view your app.

Comment 14 errata-xmlrpc 2020-05-04 11:23:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581