Bug 1799674

Summary: Build time increased on OCP 4.3.0
Product: OpenShift Container Platform Reporter: Hongkai Liu <hongkliu>
Component: ContainersAssignee: Nalin Dahyabhai <nalin>
Status: CLOSED DUPLICATE QA Contact: weiwei jiang <wjiang>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.3.0CC: aos-bugs, dwalsh, jokerman, nagrawal, nalin, tsweeney, wzheng
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-06 19:17:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
imagestream
none
build
none
build log 430
none
build log 311 none

Description Hongkai Liu 2020-02-06 18:43:29 UTC 
Running the same build on 3.11 cluster and 4.3.0 cluster.
Comparing the build time:

#430
oc get build --context build01 -n ttt
NAME                 TYPE     FROM   STATUS     STARTED             DURATION
machine-os-content   Docker          Complete   About an hour ago   1h22m0s
#311
machine.os.content.debug oc get build
NAME                 TYPE     FROM   STATUS     STARTED             DURATION
machine-os-content   Docker          Complete   About an hour ago   8m36s

Reprod. steps:

oc new-project ttt
oc apply -f ttt.is.yaml
oc apply -f machine-os-content.build.yaml


The yamls and logs are attached.

Related to https://bugzilla.redhat.com/show_bug.cgi?id=1784163
Comment 1 Hongkai Liu 2020-02-06 18:44:43 UTC 
Created attachment 1659787 [details]
imagestream
Comment 2 Hongkai Liu 2020-02-06 18:45:17 UTC 
Created attachment 1659802 [details]
build
Comment 3 Hongkai Liu 2020-02-06 18:46:04 UTC 
Created attachment 1659825 [details]
build log 430
Comment 4 Hongkai Liu 2020-02-06 18:46:46 UTC 
Created attachment 1659846 [details]
build log 311
Comment 6 Adam Kaplan 2020-02-07 02:14:14 UTC 
Moving to the Containers team - root issues are with buildah. Needs to be moved back to Builds when ready to be vendored in.
Comment 7 Tom Sweeney 2020-02-10 23:21:12 UTC 
Assigning to Nalin.  I suspect this is related if not a duplicate to https://bugzilla.redhat.com/show_bug.cgi?id=1790525
Comment 9 Nalin Dahyabhai 2020-03-06 19:08:59 UTC 
(In reply to Tom Sweeney from comment #7)
> Assigning to Nalin.  I suspect this is related if not a duplicate to
> https://bugzilla.redhat.com/show_bug.cgi?id=1790525

The big offenders in the 4.3 log compared to 3.11 happen during COPY instructions, and while one of them requires 4.3 to download an image, the majority of the time we spend on that instruction happens after the builder logs that the image was successfully pulled, so I do believe this is the same problem that we see in #1790525.  The COPY implementation is getting an overhaul, but that work is still in progress, so I'm moving this to 4.5.
Comment 10 Nalin Dahyabhai 2020-03-06 19:17:40 UTC 
Taking a closer look at default-route-openshift-image-registry.apps.build01.ci.devcluster.openshift.com/test-image/pipeline@sha256:9ca3e5979408a90873226d80e4d2952715bd2a4cc9db647c9d8533c588409064, this should be fixed when #1790525's fix goes out.  It doesn't require the rework I mentioned in comment 9, which targets cases where the source uses a .dockerignore file.  Marking this as a duplicate instead.

*** This bug has been marked as a duplicate of bug 1790525 ***