Bug 1657997 - 4.0 s2i quickstart builds are pushing all layers on incremental builds
Summary: 4.0 s2i quickstart builds are pushing all layers on incremental builds
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.1.0
Assignee: Nalin Dahyabhai
QA Contact: weiwei jiang
Depends On: 1682978
TreeView+ depends on / blocked
Reported: 2018-12-11 00:18 UTC by Mike Fiedler
Modified: 2019-06-04 10:41 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2019-06-04 10:41:14 UTC
Target Upstream Version:

Attachments (Terms of Use)
build logs, registry log, imageregistry crd (98.72 KB, application/gzip)
2018-12-11 00:40 UTC, Mike Fiedler
no flags Details

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 0 None None None 2019-06-04 10:41:20 UTC

Description Mike Fiedler 2018-12-11 00:18:51 UTC
Description of problem:

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

Actual results:

Expected results:

Additional info:

Comment 1 Mike Fiedler 2018-12-11 00:38:28 UTC
Sorry for empty description.   New bugzilla saved the bz before I could fill it out.

Description of problem:

In 4.0, the second and subsequent builds of an s2i application are pushing all layers, not just the changed top layer.   This is resulting in much longer incremental builds times than previous releases.   Example, django-psql-example normally builds in ~3 minutes for the initial build and then ~20 seconds thereafter when re-built in the same project.   In 4.0, each build in a project takes ~3 minutes, whether initial or subsequent.

Starting this bz with Image Registry, but maybe it is s2i?

Version-Release number of selected component (if applicable): 4.0 as installed by next-gen installer on 12/10

How reproducible: Always

Steps to Reproduce:
1. oc new-project test
2. oc new-app centos/ruby-25-centos7~https://github.com/sclorg/ruby-ex.git 
3. oc get builds

ruby-ex-1   Source    Git@887f01c   Complete   2 minutes ago   2m43s

4. oc start-build ruby-ex
5. oc get builds

Actual results:

ruby-ex-1   Source    Git@887f01c   Complete   7 minutes ago   2m43s
ruby-ex-2   Source    Git@887f01c   Complete   3 minutes ago   2m30s

Build log (attached) shows all layers pushed for the second build.

Expected results:

The second build should be much faster and only the modified application layer is pushed.

Additional info:

Build and registry logs attached

Comment 2 Mike Fiedler 2018-12-11 00:40:02 UTC
Created attachment 1513220 [details]
build logs, registry log, imageregistry crd

Comment 3 Ben Parees 2018-12-11 02:19:08 UTC
This is an issue w/ buildah's management of image layers and how it pushes layers, transferring to that team.  I know they are working on some aspects of this already, but i'm glad to hear the performance team is monitoring the issue as well.

Comment 4 Daniel Walsh 2019-01-21 18:47:36 UTC
Ben isn't this fixed now with the latest buildah?

Comment 5 Tom Sweeney 2019-01-22 22:41:26 UTC
Dan, I think it is, but Nalin would know for sure or how close it is if not.

Comment 6 Daniel Walsh 2019-01-29 19:21:04 UTC
Fixed in the current release.

Comment 8 weiwei jiang 2019-02-26 02:20:47 UTC
https://bugzilla.redhat.com/show_bug.cgi?id=1682978 is blocking this issue to move to verified

Comment 9 Mike Fiedler 2019-03-06 15:22:22 UTC
Verified on 4.0.0-0.nightly-2019-03-06-074438 

Registry pushes are now faster in 4.0 than in 3.11

Comment 12 errata-xmlrpc 2019-06-04 10:41:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.