Description of problem: Copy operation is very slow during a build. Since the update to OCP 4.3.12 the build process for customers image takes upwards of 10 times longer. There is no .dockerignore file present in the repository and S2I is being used. Customer has since upgraded to 4.3.13 and still experiencing issues. Version-Release number of selected component (if applicable): $ oc version -o yaml openshiftVersion: 4.3.13 $ oc adm release info docker-builder sha256:909e63e5bfb45ceb804a8389d19ad0dc1a7a7d58d624786391a818bfda904311 How reproducible: On customers deployment using both Debian and RH NodeJS10 images. Steps to Reproduce: 1. Perform build of Image Actual results: Image takes a large amount of time to build, waiting on `COPY` commands. Expected results: Image is deployed and started within a reasonable amount of time. Additional info: Appears to be an issue with Buildah and potential clone of 1813258, however my customer is not having the error logs provided in 1813258. https://github.com/containers/buildah/issues/2090 https://github.com/containers/buildah/issues/1714 Only errors in logs are: ``` cfg.go:163] error reading file: open /var/run/secrets/openshift.io/pull/config.json: no such file or directory daemonless.go:61] looking for .dockerconfigjson at /var/run/secrets/openshift.io/pull/.dockerconfigjson cfg.go:163] error reading file: open /var/run/secrets/openshift.io/pull/.dockerconfigjson: no such file or directory daemonless.go:61] looking for .dockercfg at /var/run/secrets/openshift.io/pull/.dockercfg daemonless.go:61] found valid .dockercfg at /var/run/secrets/openshift.io/pull/.dockercfg ``` Is there a way to force roll-back the OCP-builder image or rush this through as it's blocking development.
There are no code changes in the builder image released in 4.3.13, nor any of the core components of the build system (openshift-controller-manager, image-registry, cluster-openshift-controller-manager-operator) [1]. Can you please provide the following: 1. An example BuildConfig that reproduces the slow build times - preferably with publicly accessible source code. 2. Output build logs with the build log level set to 6 and the `--timestamps=true` option: ``` $ oc start-build mybuild --build-loglevel=6 $ oc logs build/mybuild-1 --timestamps=true ``` [1] https://openshift-release.svc.ci.openshift.org/releasestream/4-stable/release/4.3.13
I don't have anything further, but I do know that Nalin has been devoting the majority of his time to this issue and has been making progress. Nalin do you have anything more specific to add?
Nothing meaningful to add to that. We've made more progress, and the newer implementation is showing to be an improvement, but it won't be ready for merging during the current sprint.
Is there anything further I can tell the customer at this stage?
I know there's been more progress and there was an pre release demo the other day. I don't have a specific date for completion, but I'm hoping in the next few weeks. Nalin anything further to report?
The newer implementation is still under development, and at this point the focus has moved to adding more conformance tests to exercise corner cases. I expect this effort to continue past this sprint.
The newer implementation is still under development, but portions of that work have been merged into the library where that logic lives. Once it's all merged and the tests look good there, we can pull in the updated version, but I don't expect that all to be finished during this sprint.
Are there any progress, plans, updates for this?
There's been incremental progress and Nalin has been steadily working on this. I don't' know how to quantify the progress to date, but I do know he's continuing with this work and has been for many weeks now. I'll ask Nalin to comment if he has further info.
The last large piece is now up for review for inclusion in the library that we use. Once it's merged, we'll be able to tag a release there and then pull it into the builder image.
Opened https://github.com/openshift/builder/pull/172 for merging the changes.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196