Bug 1807214

Summary: storage-untar process extremely slow in 4.3.0 vs 4.2.16 when performing Dockerfile COPY
Product: OpenShift Container Platform Reporter: Nick Curry <ncurry>
Component: BuildAssignee: Adam Kaplan <adam.kaplan>
Status: CLOSED DUPLICATE QA Contact: wewang <wewang>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.3.0CC: aos-bugs, ncurry, wzheng
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-02-26 16:02:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Nick Curry 2020-02-25 20:20:07 UTC
Description of problem:

When performing a Binary BuildConfig in 4.3.0 the build hangs on a single Dockerfile COPY operation. The storage-untar process inside build container works exteremely slowly in 4.3.0.

The same operation in 4.2.16 is nearly instantaneous.


Version-Release number of selected component (if applicable):


Build container images in 4.3.0:
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b4d7ccfae31e068acba8065848d9c4ff206c0b4cf25e7792946a8752565a417


Comparable images in 4.2.16:
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:42e116e130c36e1e2926797879ec91c3e0a3fa2f70172a2be6ea1bdb0978e213


How reproducible: 100%


Steps to Reproduce:
1. Deploy Binary BuildConfig 
2. Attempt to perform Dockerfile COPY on a 1.1 GB directory containing 8728 subdirectories and 56397 files into container image
3. storage-untar process copies data exceptionally slow

Actual results:

Dockerfile rendered useless


Expected results:

Dockerfile suceeds like it does in 4.2.16


Additional info:


The FROM image in the Dockerfile is a HCL Commerce 9.0.1.7 application image.


Example definitions for reproduction:


BuildConfig:

kind: BuildConfig
apiVersion: build.openshift.io/v1
metadata:
  name: test
spec:
  source:
    type:
      Binary
  strategy:
    type: Docker
    dockerStrategy:
      from:
        kind: "ImageStreamTag"
        name: "ts-app:9.0.1.7"
  output:
    to:
      kind: ImageStreamTag
      name: test:latest


ImageStream:

kind: ImageStream
apiVersion: image.openshift.io/v1
metadata:
  name: test


Dockerfile:


FROM registry.redhat.io/ubi8:latest

COPY CusDeploy /SETUP/Cus/

Comment 1 Nick Curry 2020-02-25 20:24:33 UTC
Possibly related?
https://bugzilla.redhat.com/show_bug.cgi?id=1790525

Comment 2 Nick Curry 2020-02-25 20:37:09 UTC
Reproduced on a centos:latest image
https://hub.docker.com/_/centos

vs the HCL commerce ts-app:9.0.1.7 image

Comment 3 Nick Curry 2020-02-25 20:49:27 UTC
The storage-untar command in each build 4.2 vs 4.3 is slightly different:



4.3
storage-untar / /var/lib/containers/storage/overlay/5069cad46ede6e5810f772797ebea565d338a8024b1d27ed766b4ad62a668ea2/merged

4.2
storage-untar / /var/lib/containers/storage/overlay/d53487b1df2a7050b0e20f02323ec077adebd35fb4a7990d6bd520bae52b74be/merged/SETUP/Cus


Not sure if that is significant.

Comment 4 Adam Kaplan 2020-02-26 16:02:36 UTC
This regression has been fixed in Bug 1790525 (currently VERIFIED, awaiting release).

*** This bug has been marked as a duplicate of bug 1790525 ***