Bug 1716550
| Summary: | oc client tool randomly gives "panic: runtime error: slice bounds out of range" | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Tim Bielawa <tbielawa> | ||||||
| Component: | Installer | Assignee: | Stephen Benjamin <stbenjam> | ||||||
| Installer sub component: | openshift-installer | QA Contact: | Johnny Liu <jialiu> | ||||||
| Status: | CLOSED ERRATA | Docs Contact: | |||||||
| Severity: | high | ||||||||
| Priority: | high | CC: | aos-bugs, bleanhar, ccoleman, jokerman, jupierce, mmccomas, sponnaga, stbenjam, wking | ||||||
| Version: | 4.1.z | ||||||||
| Target Milestone: | --- | ||||||||
| Target Release: | 4.3.0 | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | No Doc Update | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2020-01-23 11:04:11 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
|
Description
Tim Bielawa
2019-06-03 15:03:29 UTC
This should be solved in https://github.com/openshift/origin/pull/23255 This should be fixed at this point in time, since the PR from comment #1 should be included in oc by now. Moving to qa. Using OC as new as this
> [tbielawa@buildvm ~]$ rpm -q openshift-clients
> openshift-clients-4.2.0-201908150219.git.0.f6120d9.el7.x86_64
We still find it to be throwing random panics
New attachment is full trace from today's explosion
Created attachment 1608651 [details]
new explosion
boom!
I have updated our installed version to openshift-clients-4.2.0-201908261819.git.0.b985ea3.el7.x86_64 Will report back if this happens again Not sure how Tim can reproduce it. In my try with commands in comment 0 and comment 5, cannot reproduce it with oc of either above openshift-clients-4.2.0-201908261819.git.0.b985ea3.el7.x86_64 or latest 4.2.0-201908272219.git.0.1904cc5.el7: for i in {1..100} do echo "trying order: $i =========" rm -rf ./mnt GOTRACEBACK=all oc adm release extract --tools '--command-os=*' quay.io/openshift-release-dev/ocp-release-nightly:4.2.0-0.nightly-2019-08-27-072819 --to=./mnt/workpace/jenkins/working/aos-cd-builds/build%2Foc_sync/tools/4.2.0-0.nightly-2019-08-27-072819 sleep 1 done Now that Tim still reproduces it and comment 3 (comment 1) PR was closed instead of merged, assigning back Hmm, tried comment 7 command and wait, hits the panic several times: http://file.rdu.redhat.com/~xxia/bug-1716550-recreation.txt This area belongs to the installer, I'll let them deal with it. It looks to me like a timeout, maybe due to very frequent pulls. We feel the severity isn't high enough to fix this in 4.2. We rely heavily on this command in CI and don't see this problem happening often. We rely heavily on this command in OCP releases and it happens at least once a week. The result is that we have to re-run release jobs for advisories. Not ideal. Still seeing this. New one today while trying to extract the clients for the pre-release content. Adding new attachment log. [tbielawa@buildvm ~]$ uname -r 3.10.0-1062.1.1.el7.x86_64 There is a race condition in docker, see https://github.com/moby/moby/issues/39859 There's a workaround in http://github.com/openshift/oc/pull/104, as the version of docker kubectl vendors is ancient (even in k8s 1.16). If you can't get a newer oc, run `export MOBY_DISABLE_PIGZ=true` before running oc. Verified this bug with openshift-clinets-4.3.0-201910141917.git.1.7327846.el7 and PASS. Thanks for xxia's reproduce steps. I can reproduce it with openshift-clients-4.2.0-201909221318.git.1.bc66c02.el7.x86_64, after upgrade oc client to openshift-clinets-4.3.0-201910141917.git.1.7327846.el7, the issue is fixed. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0062 |