Bug 1642350
| Summary: | Failure to restore cache produces corrupted files passed to incremental builds | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Sergio G. <sgarciam> | |
| Component: | Build | Assignee: | Ben Parees <bparees> | |
| Status: | CLOSED ERRATA | QA Contact: | wewang <wewang> | |
| Severity: | low | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 3.11.0 | CC: | aos-bugs, bparees, wzheng | |
| Target Milestone: | --- | |||
| Target Release: | 3.11.z | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | Bug Fix | ||
| Doc Text: |
Cause: Unnecessarily short timeout resulted in a failure to reuse artifacts from a previous build when incremental builds were selected with s2i. This could occur when the size of the artifacts being reused was particularly large or the host system was running particularly slowly.
Consequence: Invalid artifacts could be used in a subsequent build, or artifacts would be recreated instead of reused resulting in performance degradation.
Fix: Timeout has been increased to a sufficiently large value as to avoid this problem.
Result: Artifact reuse should no longer timeout.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1644343 1644344 (view as bug list) | Environment: | ||
| Last Closed: | 2018-11-20 03:11:52 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1644343, 1644344 | |||
|
Description
Sergio G.
2018-10-24 08:33:50 UTC
Will these new s2i images be usable from older releases than 3.11.z once they're released? Is there any known issue to not being able to use them in 3.9? This is not a fix to s2i images, this is a fix to the core product code, we are delivering the fix for 3.11.z, we have no current plans to backport the fix to an older release. Then a note in the documentation in previous releases should be added to avoid using incremental builds if the resulting tar is big enough to time out (the extraction will time out and the partially written files are still being passed into subsequent builds)? > Then a note in the documentation in previous releases should be added to avoid using incremental builds if the resulting tar is big enough to time out (the extraction will time out and the partially written files are still being passed into subsequent builds)?
This is the first incident of this i'm aware of. If customers run into it on older versions we will revisit backporting the change as needed.
@Ben, Download S2I version "v1.1.12" I created tar file with 1001M to want to test extract timeout, but seems time is enough to extract now. it's hard to let error happen, should I create a bigger tar file to test it? steps: 1. Create a tar file with 1G $ ls -lh total 1001M -rw-r--r--. 1 wewang docker 1001M Nov 5 18:34 ruby.tar 2. S2i build with the file $ ./s2i build --pull-policy=never --incremental-pull-policy=never --incremental --copy /home/wewang/Downloads/ruby centos/ruby-22-centos7 ruby-hello-app --loglevel=5 3. Logs: https://url.corp.redhat.com/3afeb05 And when I get source from https://github.com/openshift/source-to-image/tree/release-3.11 and install s2i binary manually, seems it's older, its version is v1.1.11. v1.1.12 contains the fix, so you're going to have a hard time hitting the timeout with that version. and the sourcecode in https://github.com/openshift/source-to-image/tree/release-3.11 has also already been patched. If you want a version of the code where you can easily recreate the original error, you'd need to use the v1.1.11 release, which was created before the fix was introduced. YES，tested older S2I version "v1.1.11" failed in tar part, so verified the bug, thanks
[wewang@wen-local sourcetoimage]$ ./s2i build --pull-policy=never --incremental-pull-policy=never --copy /home/wewang/Downloads/ruby/ centos/ruby-22-centos7 ruby-hello-app1 --loglevel=5I1106 10:16:11.689919 17419 build.go:50] Running S2I version "v1.1.11"
I1106 10:16:11.690141 17419 util.go:57] Getting docker credentials for centos/ruby-22-centos7
I1106 10:16:11.690182 17419 util.go:73] Using credentials for pulling centos/ruby-22-centos7
I1106 10:16:11.691204 17419 util.go:269] Checking if image "centos/ruby-22-centos7" is available locally ...
I1106 10:16:11.696912 17419 build.go:165]
Builder Name: Ruby 2.2
Builder Image: centos/ruby-22-centos7
Builder Image Version: "c159276"
Source: /home/wewang/Downloads/ruby/
Output Image Tag: ruby-hello-app1
Environment:
Labels:
Incremental Build: disabled
Remove Old Build: disabled
Builder Pull Policy: never
Previous Image Pull Policy: never
Quiet: disabled
Layered Build: disabled
Docker Endpoint: unix:///var/run/docker.sock
Docker Pull Config: /home/wewang/.docker/config.json
Docker Pull User: wewang58
I1106 10:16:11.696952 17419 util.go:269] Checking if image "centos/ruby-22-centos7" is available locally ...
I1106 10:16:11.702077 17419 docker.go:510] Using locally available image "centos/ruby-22-centos7:latest"
I1106 10:16:11.702098 17419 docker.go:741] Image sha256:e42d0dccf073123561d83ea8bbc9f0cc5e491cfd07130a464a416cdb99ced387 contains io.openshift.s2i.scripts-url set to "image:///usr/libexec/s2i"
I1106 10:16:11.702116 17419 scm.go:20] DownloadForSource /home/wewang/Downloads/ruby/
I1106 10:16:11.702170 17419 sti.go:198] Preparing to build ruby-hello-app1
I1106 10:16:11.702418 17419 download.go:30] Copying sources from "/home/wewang/Downloads/ruby/" to "/tmp/s2i361273747/upload/src"
I1106 10:16:11.702537 17419 fs.go:236] F "/home/wewang/Downloads/ruby/ruby.tar" -> "/tmp/s2i361273747/upload/src/ruby.tar"
I1106 10:16:26.488150 17419 install.go:260] Using "assemble" installed from "image:///usr/libexec/s2i/assemble"
I1106 10:16:26.488196 17419 install.go:260] Using "run" installed from "image:///usr/libexec/s2i/run"
I1106 10:16:26.488216 17419 install.go:260] Using "save-artifacts" installed from "image:///usr/libexec/s2i/save-artifacts"
I1106 10:16:26.488239 17419 ignore.go:63] .s2iignore file does not exist
I1106 10:16:26.488248 17419 sti.go:207] Clean build will be performed
I1106 10:16:26.488261 17419 sti.go:210] Performing source build from /home/wewang/Downloads/ruby/
I1106 10:16:26.488268 17419 sti.go:221] Running "assemble" in "ruby-hello-app1"
I1106 10:16:26.488273 17419 sti.go:559] Using image name centos/ruby-22-centos7
I1106 10:16:26.634042 17419 docker.go:510] Using locally available image "centos/ruby-22-centos7:latest"
I1106 10:16:26.634106 17419 sti.go:445] No user environment provided (no environment file found in application sources)
I1106 10:16:26.634157 17419 sti.go:673] starting the source uploading ...
I1106 10:16:26.634177 17419 tar.go:217] Adding "/tmp/s2i361273747/upload" to tar ...
I1106 10:16:26.634299 17419 tar.go:312] Adding to tar: /tmp/s2i361273747/upload/scripts as scripts
I1106 10:16:26.700442 17419 docker.go:741] Image sha256:e42d0dccf073123561d83ea8bbc9f0cc5e491cfd07130a464a416cdb99ced387 contains io.openshift.s2i.scripts-url set to "image:///usr/libexec/s2i"
I1106 10:16:26.700480 17419 docker.go:816] Base directory for S2I scripts is '/usr/libexec/s2i'. Untarring destination is '/tmp'.
I1106 10:16:26.700514 17419 docker.go:972] Setting "/bin/sh -c tar -C /tmp -xf - && /usr/libexec/s2i/assemble" command for container ...
I1106 10:16:26.700838 17419 docker.go:981] Creating container with options {Name:"s2i_centos_ruby_22_centos7_6b976a51" Config:{Hostname: Domainname: User: AttachStdin:false AttachStdout:true AttachStderr:false ExposedPorts:map[] Tty:false OpenStdin:true StdinOnce:true Env:[] Cmd:[/bin/sh -c tar -C /tmp -xf - && /usr/libexec/s2i/assemble] Healthcheck:<nil> ArgsEscaped:false Image:centos/ruby-22-centos7:latest Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:&{Binds:[] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode: PortBindings:map[] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] CapAdd:[] CapDrop:[] DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:false PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[] UTSMode: UsernsMode: ShmSize:67108864 Sysctls:map[] Runtime: ConsoleSize:[0 0] Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DiskQuota:0 KernelMemory:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:0 Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] Init:<nil>}} ...
E1106 10:18:26.701248 17419 tar.go:245] Error writing header for "scripts": io: read/write on closed pipe
E1106 10:18:26.701277 17419 tar.go:270] Error writing tar: io: read/write on closed pipe
I1106 10:18:26.701389 17419 cleanup.go:33] Removing temporary directory /tmp/s2i361273747
I1106 10:18:26.701414 17419 fs.go:278] Removing directory '/tmp/s2i361273747'
I1106 10:18:26.789832 17419 build.go:171] Build failed
E1106 10:18:26.789882 17419 errors.go:276] An error occurred: context deadline exceeded
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:3537 |