Bug 1657997

Summary: 4.0 s2i quickstart builds are pushing all layers on incremental builds
Product: OpenShift Container Platform Reporter: Mike Fiedler <mifiedle>
Component: ContainersAssignee: Nalin Dahyabhai <nalin>
Status: CLOSED ERRATA QA Contact: weiwei jiang <wjiang>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.1.0CC: aos-bugs, dwalsh, hongkliu, jokerman, mmccomas, mpatel, tsweeney
Target Milestone: ---   
Target Release: 4.1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-04 10:41:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1682978    
Bug Blocks:    
Attachments:
Description Flags
build logs, registry log, imageregistry crd none

Description Mike Fiedler 2018-12-11 00:18:51 UTC
Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Mike Fiedler 2018-12-11 00:38:28 UTC
Sorry for empty description.   New bugzilla saved the bz before I could fill it out.

Description of problem:

In 4.0, the second and subsequent builds of an s2i application are pushing all layers, not just the changed top layer.   This is resulting in much longer incremental builds times than previous releases.   Example, django-psql-example normally builds in ~3 minutes for the initial build and then ~20 seconds thereafter when re-built in the same project.   In 4.0, each build in a project takes ~3 minutes, whether initial or subsequent.

Starting this bz with Image Registry, but maybe it is s2i?


Version-Release number of selected component (if applicable): 4.0 as installed by next-gen installer on 12/10


How reproducible: Always


Steps to Reproduce:
1. oc new-project test
2. oc new-app centos/ruby-25-centos7~https://github.com/sclorg/ruby-ex.git 
3. oc get builds

ruby-ex-1   Source    Git@887f01c   Complete   2 minutes ago   2m43s

4. oc start-build ruby-ex
5. oc get builds

Actual results:

ruby-ex-1   Source    Git@887f01c   Complete   7 minutes ago   2m43s
ruby-ex-2   Source    Git@887f01c   Complete   3 minutes ago   2m30s

Build log (attached) shows all layers pushed for the second build.

Expected results:

The second build should be much faster and only the modified application layer is pushed.


Additional info:

Build and registry logs attached

Comment 2 Mike Fiedler 2018-12-11 00:40:02 UTC
Created attachment 1513220 [details]
build logs, registry log, imageregistry crd

Comment 3 Ben Parees 2018-12-11 02:19:08 UTC
This is an issue w/ buildah's management of image layers and how it pushes layers, transferring to that team.  I know they are working on some aspects of this already, but i'm glad to hear the performance team is monitoring the issue as well.

Comment 4 Daniel Walsh 2019-01-21 18:47:36 UTC
Ben isn't this fixed now with the latest buildah?

Comment 5 Tom Sweeney 2019-01-22 22:41:26 UTC
Dan, I think it is, but Nalin would know for sure or how close it is if not.

Comment 6 Daniel Walsh 2019-01-29 19:21:04 UTC
Fixed in the current release.

Comment 8 weiwei jiang 2019-02-26 02:20:47 UTC
https://bugzilla.redhat.com/show_bug.cgi?id=1682978 is blocking this issue to move to verified

Comment 9 Mike Fiedler 2019-03-06 15:22:22 UTC
Verified on 4.0.0-0.nightly-2019-03-06-074438 

Registry pushes are now faster in 4.0 than in 3.11

Comment 12 errata-xmlrpc 2019-06-04 10:41:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758