Bug 1828119 - OpenShift Build Image - COPY Command is Very Slow
Summary: OpenShift Build Image - COPY Command is Very Slow
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers
Version: 4.3.z
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: 4.6.0
Assignee: Nalin Dahyabhai
QA Contact: Weinan Liu
URL:
Whiteboard:
Depends On:
Blocks: 1952467
TreeView+ depends on / blocked
 
Reported: 2020-04-27 04:44 UTC by Michael Washer
Modified: 2023-12-15 17:45 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: the logic that handled COPY or ADD instructions for copying content from the build context did not efficiently filter what they were copying when a .dockerignore file was present, and always behaved as though a .dockerignore file was present. Consequence: COPY and ADD would be noticeably slowed by the cumulative overhead of evaluating whether each item in the source location should be copied to the destination, even when there was no need to perform this evaluation. Fix: that logic was rewritten. Result: This slowdown should no longer occur.
Clone Of:
: 1833328 1952467 (view as bug list)
Environment:
Last Closed: 2020-10-27 15:58:27 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift builder pull 172 0 None closed Bug 1833328: bump buildah to v1.16.0 2021-02-09 08:18:39 UTC
Red Hat Knowledge Base (Solution) 5565271 0 None None None 2020-11-09 21:49:36 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 15:58:46 UTC

Description Michael Washer 2020-04-27 04:44:53 UTC
Description of problem:

Copy operation is very slow during a build.
Since the update to OCP 4.3.12 the build process for customers image takes upwards of 10 times longer.
There is no .dockerignore file present in the repository and S2I is being used.

Customer has since upgraded to 4.3.13 and still experiencing issues.


Version-Release number of selected component (if applicable):
$ oc version -o yaml
openshiftVersion: 4.3.13

$ oc adm release info
docker-builder                                sha256:909e63e5bfb45ceb804a8389d19ad0dc1a7a7d58d624786391a818bfda904311

How reproducible:
On customers deployment using both Debian and RH NodeJS10 images.

Steps to Reproduce:
1. Perform build of Image

Actual results:
Image takes a large amount of time to build, waiting on `COPY` commands. 

Expected results:
Image is deployed and started within a reasonable amount of time.

Additional info:

Appears to be an issue with Buildah and potential clone of 1813258, however my customer is not having the error logs provided in 1813258. 
https://github.com/containers/buildah/issues/2090
https://github.com/containers/buildah/issues/1714

Only errors in logs are:
```
cfg.go:163] error reading file: open /var/run/secrets/openshift.io/pull/config.json: no such file or directory
daemonless.go:61] looking for .dockerconfigjson at /var/run/secrets/openshift.io/pull/.dockerconfigjson
cfg.go:163] error reading file: open /var/run/secrets/openshift.io/pull/.dockerconfigjson: no such file or directory
daemonless.go:61] looking for .dockercfg at /var/run/secrets/openshift.io/pull/.dockercfg
daemonless.go:61] found valid .dockercfg at /var/run/secrets/openshift.io/pull/.dockercfg
```

Is there a way to force roll-back the OCP-builder image or rush this through as it's blocking development.

Comment 2 Adam Kaplan 2020-04-28 18:58:50 UTC
There are no code changes in the builder image released in 4.3.13, nor any of the core components of the build system (openshift-controller-manager, image-registry, cluster-openshift-controller-manager-operator) [1].

Can you please provide the following:

1. An example BuildConfig that reproduces the slow build times - preferably with publicly accessible source code.
2. Output build logs with the build log level set to 6 and the `--timestamps=true` option:

```
$ oc start-build mybuild --build-loglevel=6
$ oc logs build/mybuild-1 --timestamps=true
```

[1] https://openshift-release.svc.ci.openshift.org/releasestream/4-stable/release/4.3.13

Comment 14 Tom Sweeney 2020-06-09 17:55:37 UTC
I don't have anything further, but I do know that Nalin has been devoting the majority of his time to this issue and has been making progress.  Nalin do you have anything more specific to add?

Comment 15 Nalin Dahyabhai 2020-06-18 17:56:41 UTC
Nothing meaningful to add to that.  We've made more progress, and the newer implementation is showing to be an improvement, but it won't be ready for merging during the current sprint.

Comment 16 Michael Washer 2020-06-25 00:58:22 UTC
Is there anything further I can tell the customer at this stage?

Comment 17 Tom Sweeney 2020-06-25 20:14:04 UTC
I know there's been more progress and there was an pre release demo the other day.  I don't have a specific date for completion, but I'm hoping in the next few weeks.  Nalin anything further to report?

Comment 18 Nalin Dahyabhai 2020-07-08 14:09:35 UTC
The newer implementation is still under development, and at this point the focus has moved to adding more conformance tests to exercise corner cases.  I expect this effort to continue past this sprint.

Comment 19 Nalin Dahyabhai 2020-07-30 21:58:29 UTC
The newer implementation is still under development, but portions of that work have been merged into the library where that logic lives.  Once it's all merged and the tests look good there, we can pull in the updated version, but I don't expect that all to be finished during this sprint.

Comment 20 Michael Washer 2020-08-17 00:59:07 UTC
Are there any progress, plans, updates for this?

Comment 21 Tom Sweeney 2020-08-17 18:44:45 UTC
There's been incremental progress and Nalin has been steadily working on this.  I don't' know how to quantify the progress to date, but I do know he's continuing with this work and has been for many weeks now.   I'll ask Nalin to comment if he has further info.

Comment 22 Nalin Dahyabhai 2020-08-21 17:00:20 UTC
The last large piece is now up for review for inclusion in the library that we use.  Once it's merged, we'll be able to tag a release there and then pull it into the builder image.

Comment 24 Nalin Dahyabhai 2020-09-08 15:27:10 UTC
Opened https://github.com/openshift/builder/pull/172 for merging the changes.

Comment 29 errata-xmlrpc 2020-10-27 15:58:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.