Bug 1642350 - Failure to restore cache produces corrupted files passed to incremental builds
Summary: Failure to restore cache produces corrupted files passed to incremental builds
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: ---
: 3.11.z
Assignee: Ben Parees
QA Contact: wewang
URL:
Whiteboard:
Depends On:
Blocks: 1644343 1644344
TreeView+ depends on / blocked
 
Reported: 2018-10-24 08:33 UTC by Sergio G.
Modified: 2018-11-20 03:12 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Unnecessarily short timeout resulted in a failure to reuse artifacts from a previous build when incremental builds were selected with s2i. This could occur when the size of the artifacts being reused was particularly large or the host system was running particularly slowly. Consequence: Invalid artifacts could be used in a subsequent build, or artifacts would be recreated instead of reused resulting in performance degradation. Fix: Timeout has been increased to a sufficiently large value as to avoid this problem. Result: Artifact reuse should no longer timeout.
Clone Of:
: 1644343 1644344 (view as bug list)
Environment:
Last Closed: 2018-11-20 03:11:52 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github https://github.com/openshift source-to-image issues 929 None None None 2020-02-20 07:49:23 UTC
Red Hat Product Errata RHBA-2018:3537 None None None 2018-11-20 03:12:09 UTC

Description Sergio G. 2018-10-24 08:33:50 UTC
This problem is described in source-to-image upstream tracker.

Seems to be merged and I'd like to know when/if sti-build image will be updated with that fix.

References:
 [1] https://github.com/openshift/source-to-image/issues/929
 [2] https://access.redhat.com/containers/?tab=overview#/registry.access.redhat.com/openshift3/ose-sti-builder

Comment 1 Ben Parees 2018-10-24 14:39:40 UTC
https://github.com/openshift/origin/pull/21345

Comment 2 Sergio G. 2018-10-29 08:25:24 UTC
Will these new s2i images be usable from older releases than 3.11.z once they're released? Is there any known issue to not being able to use them in 3.9?

Comment 3 Ben Parees 2018-10-29 16:00:34 UTC
This is not a fix to s2i images, this is a fix to the core product code, we are delivering the fix for 3.11.z, we have no current plans to backport the fix to an older release.

Comment 4 Sergio G. 2018-10-30 13:53:12 UTC
Then a note in the documentation in previous releases should be added to avoid using incremental builds if the resulting tar is big enough to time out (the extraction will time out and the partially written files are still being passed into subsequent builds)?

Comment 5 Ben Parees 2018-10-30 13:55:04 UTC
> Then a note in the documentation in previous releases should be added to avoid using incremental builds if the resulting tar is big enough to time out (the extraction will time out and the partially written files are still being passed into subsequent builds)?

This is the first incident of this i'm aware of.  If customers run into it on older versions we will revisit backporting the change as needed.

Comment 10 wewang 2018-11-05 10:50:58 UTC
@Ben, Download S2I version "v1.1.12"
I created tar file with 1001M to want to test extract timeout, but seems time is enough to extract now. it's hard to let error happen, should I create a bigger tar file to test it?

steps:
1.  Create a tar file with 1G
$ ls -lh
total 1001M
-rw-r--r--. 1 wewang docker 1001M Nov  5 18:34 ruby.tar

2. S2i build with the file 
$ ./s2i  build     --pull-policy=never --incremental-pull-policy=never     --incremental    --copy /home/wewang/Downloads/ruby  centos/ruby-22-centos7 ruby-hello-app --loglevel=5

3. Logs: https://url.corp.redhat.com/3afeb05

And when I get source from https://github.com/openshift/source-to-image/tree/release-3.11 and install  s2i binary manually, seems it's older, its version is v1.1.11.

Comment 11 Ben Parees 2018-11-05 15:43:45 UTC
v1.1.12 contains the fix, so you're going to have a hard time hitting the timeout with that version.


and the sourcecode in https://github.com/openshift/source-to-image/tree/release-3.11 has also already been patched.

If you want a version of the code where you can easily recreate the original error, you'd need to use the v1.1.11 release, which was created before the fix was introduced.

Comment 12 wewang 2018-11-06 02:26:38 UTC
YES,tested older S2I version "v1.1.11" failed in tar part, so verified the bug, thanks

[wewang@wen-local sourcetoimage]$ ./s2i  build     --pull-policy=never --incremental-pull-policy=never    --copy /home/wewang/Downloads/ruby/  centos/ruby-22-centos7 ruby-hello-app1 --loglevel=5I1106 10:16:11.689919   17419 build.go:50] Running S2I version "v1.1.11"
I1106 10:16:11.690141   17419 util.go:57] Getting docker credentials for centos/ruby-22-centos7
I1106 10:16:11.690182   17419 util.go:73] Using  credentials for pulling centos/ruby-22-centos7
I1106 10:16:11.691204   17419 util.go:269] Checking if image "centos/ruby-22-centos7" is available locally ...
I1106 10:16:11.696912   17419 build.go:165] 
Builder Name:			Ruby 2.2
Builder Image:			centos/ruby-22-centos7
Builder Image Version:		"c159276"
Source:				/home/wewang/Downloads/ruby/
Output Image Tag:		ruby-hello-app1
Environment:			
Labels:				
Incremental Build:		disabled
Remove Old Build:		disabled
Builder Pull Policy:		never
Previous Image Pull Policy:	never
Quiet:				disabled
Layered Build:			disabled
Docker Endpoint:		unix:///var/run/docker.sock
Docker Pull Config:		/home/wewang/.docker/config.json
Docker Pull User:		wewang58

I1106 10:16:11.696952   17419 util.go:269] Checking if image "centos/ruby-22-centos7" is available locally ...
I1106 10:16:11.702077   17419 docker.go:510] Using locally available image "centos/ruby-22-centos7:latest"
I1106 10:16:11.702098   17419 docker.go:741] Image sha256:e42d0dccf073123561d83ea8bbc9f0cc5e491cfd07130a464a416cdb99ced387 contains io.openshift.s2i.scripts-url set to "image:///usr/libexec/s2i"
I1106 10:16:11.702116   17419 scm.go:20] DownloadForSource /home/wewang/Downloads/ruby/
I1106 10:16:11.702170   17419 sti.go:198] Preparing to build ruby-hello-app1
I1106 10:16:11.702418   17419 download.go:30] Copying sources from "/home/wewang/Downloads/ruby/" to "/tmp/s2i361273747/upload/src"
I1106 10:16:11.702537   17419 fs.go:236] F "/home/wewang/Downloads/ruby/ruby.tar" -> "/tmp/s2i361273747/upload/src/ruby.tar"
I1106 10:16:26.488150   17419 install.go:260] Using "assemble" installed from "image:///usr/libexec/s2i/assemble"
I1106 10:16:26.488196   17419 install.go:260] Using "run" installed from "image:///usr/libexec/s2i/run"
I1106 10:16:26.488216   17419 install.go:260] Using "save-artifacts" installed from "image:///usr/libexec/s2i/save-artifacts"
I1106 10:16:26.488239   17419 ignore.go:63] .s2iignore file does not exist
I1106 10:16:26.488248   17419 sti.go:207] Clean build will be performed
I1106 10:16:26.488261   17419 sti.go:210] Performing source build from /home/wewang/Downloads/ruby/
I1106 10:16:26.488268   17419 sti.go:221] Running "assemble" in "ruby-hello-app1"
I1106 10:16:26.488273   17419 sti.go:559] Using image name centos/ruby-22-centos7
I1106 10:16:26.634042   17419 docker.go:510] Using locally available image "centos/ruby-22-centos7:latest"
I1106 10:16:26.634106   17419 sti.go:445] No user environment provided (no environment file found in application sources)
I1106 10:16:26.634157   17419 sti.go:673] starting the source uploading ...
I1106 10:16:26.634177   17419 tar.go:217] Adding "/tmp/s2i361273747/upload" to tar ...
I1106 10:16:26.634299   17419 tar.go:312] Adding to tar: /tmp/s2i361273747/upload/scripts as scripts
I1106 10:16:26.700442   17419 docker.go:741] Image sha256:e42d0dccf073123561d83ea8bbc9f0cc5e491cfd07130a464a416cdb99ced387 contains io.openshift.s2i.scripts-url set to "image:///usr/libexec/s2i"
I1106 10:16:26.700480   17419 docker.go:816] Base directory for S2I scripts is '/usr/libexec/s2i'. Untarring destination is '/tmp'.
I1106 10:16:26.700514   17419 docker.go:972] Setting "/bin/sh -c tar -C /tmp -xf - && /usr/libexec/s2i/assemble" command for container ...
I1106 10:16:26.700838   17419 docker.go:981] Creating container with options {Name:"s2i_centos_ruby_22_centos7_6b976a51" Config:{Hostname: Domainname: User: AttachStdin:false AttachStdout:true AttachStderr:false ExposedPorts:map[] Tty:false OpenStdin:true StdinOnce:true Env:[] Cmd:[/bin/sh -c tar -C /tmp -xf - && /usr/libexec/s2i/assemble] Healthcheck:<nil> ArgsEscaped:false Image:centos/ruby-22-centos7:latest Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:&{Binds:[] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode: PortBindings:map[] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] CapAdd:[] CapDrop:[] DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:false PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[] UTSMode: UsernsMode: ShmSize:67108864 Sysctls:map[] Runtime: ConsoleSize:[0 0] Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DiskQuota:0 KernelMemory:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:0 Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] Init:<nil>}} ...
E1106 10:18:26.701248   17419 tar.go:245] Error writing header for "scripts": io: read/write on closed pipe
E1106 10:18:26.701277   17419 tar.go:270] Error writing tar: io: read/write on closed pipe
I1106 10:18:26.701389   17419 cleanup.go:33] Removing temporary directory /tmp/s2i361273747
I1106 10:18:26.701414   17419 fs.go:278] Removing directory '/tmp/s2i361273747'
I1106 10:18:26.789832   17419 build.go:171] Build failed
E1106 10:18:26.789882   17419 errors.go:276] An error occurred: context deadline exceeded

Comment 14 errata-xmlrpc 2018-11-20 03:11:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3537


Note You need to log in before you can comment on or make changes to this bug.