Bug 1690149

Summary:	oc cli returned `Resource temporarily unavailable` in concurrent build test
Product:	OpenShift Container Platform	Reporter:	Hongkai Liu <hongkliu>
Component:	oc	Assignee:	Maciej Szulik <maszulik>
Status:	CLOSED NOTABUG	QA Contact:	Xingxing Xia <xxia>
Severity:	low	Docs Contact:
Priority:	low
Version:	4.1.0	CC:	aos-bugs, hongkliu, jokerman, mfojtik, mifiedle, mmccomas, wzheng
Target Milestone:	---
Target Release:	4.1.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	aos-scalability-41
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-04-30 15:17:55 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Hongkai Liu 2019-03-18 21:23:19 UTC

Description of problem:
concurrent builds on scale-lab: 25 2xlarge workers.
750 projects for nodejs app.

Durring the test, we called many `oc start-build -n ...` in 50 threads for 750 projects and the return has many `Resource temporarily unavailable`: about 50% of them.


Version-Release number of selected component (if applicable):
4.0.0-0.nightly-2019-03-04-234414

How reproducible: always


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
log file from python build test script:
http://pbench.perf.lab.eng.bos.redhat.com/results/EC2::ip-10-0-14-191/ocp4_conc_build_201903/pbench-user-benchmark_buildlog_2019.03.18T20.27.50/1/reference-result/build_test.log

Comment 1 Mike Fiedler 2019-03-18 23:31:08 UTC

Here's the crash info for the CLI:

SIGABRT: abort
PC=0x7f6012004207 m=8 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7f6012004207
stack: frame={sp:0x7f600a7fb9c8, fp:0x0} stack=[0x7f6009ffc2b0,0x7f600a7fbeb0)
00007f600a7fb8c8:  0000000000000000  0000000000000000 
00007f600a7fb8d8:  0000000000000000  0000000000000000 
00007f600a7fb8e8:  0000000000000000  0000000000000000 
00007f600a7fb8f8:  0000000000000000  0000000000000000 
00007f600a7fb908:  0000000000000000  0000000000000000 
00007f600a7fb918:  0000000000000000  0000000000000000 
00007f600a7fb928:  0000000000000000  0000000000000000 
00007f600a7fb938:  0000000000000000  0000000000000000 
00007f600a7fb948:  0000000000000000  0000000000000000 
00007f600a7fb958:  0000000000000000  0000000000000000 
00007f600a7fb968:  0000000000000000  0000000000000000 
00007f600a7fb978:  0000000000000000  0000000000000000 
00007f600a7fb988:  0000000000000000  0000000000000000 
00007f600a7fb998:  00007f6012395868  0000000002d95bf5 
00007f600a7fb9a8:  00007f5ffc0008c0  00000000000000f1 
00007f600a7fb9b8:  0000000000000011  0000000000000000 
00007f600a7fb9c8: <00007f60120058f8  0000000000000020 
00007f600a7fb9d8:  0000000000000000  0000000000000000 
00007f600a7fb9e8:  0000000000000000  0000000000000000 
00007f600a7fb9f8:  0000000000000000  0000000000000000 
00007f600a7fba08:  0000000000000000  0000000000000000 
00007f600a7fba18:  0000000000000000  0000000000000000 
00007f600a7fba28:  0000000000000000  0000000000000000 
00007f600a7fba38:  0000000000000000  0000000000000000 
00007f600a7fba48:  0000000000000000  000000000000000d 
00007f600a7fba58:  00007f60120bcff9  0000000000000000 
00007f600a7fba68:  00007f60120480b3  000000000000000d 
00007f600a7fba78:  00007f60123951c0  0000000000000001 
00007f600a7fba88:  00007f6012395243  00007f60123913a0 
00007f600a7fba98:  00007f60120498ce  00007f60123951c0 
00007f600a7fbaa8:  000000000000000a  00007f5ffc0008c0 
00007f600a7fbab8:  00000000000000f1  0000000000000011 
runtime: unknown pc 0x7f6012004207
stack: frame={sp:0x7f600a7fb9c8, fp:0x0} stack=[0x7f6009ffc2b0,0x7f600a7fbeb0)
00007f600a7fb8c8:  0000000000000000  0000000000000000 
00007f600a7fb8d8:  0000000000000000  0000000000000000 
00007f600a7fb8e8:  0000000000000000  0000000000000000 
00007f600a7fb8f8:  0000000000000000  0000000000000000 
00007f600a7fb908:  0000000000000000  0000000000000000 
00007f600a7fb918:  0000000000000000  0000000000000000 
00007f600a7fb928:  0000000000000000  0000000000000000 
00007f600a7fb938:  0000000000000000  0000000000000000 
00007f600a7fb948:  0000000000000000  0000000000000000 
00007f600a7fb958:  0000000000000000  0000000000000000 
00007f600a7fb968:  0000000000000000  0000000000000000 
00007f600a7fb978:  0000000000000000  0000000000000000 
00007f600a7fb988:  0000000000000000  0000000000000000 
00007f600a7fb998:  00007f6012395868  0000000002d95bf5 
00007f600a7fb9a8:  00007f5ffc0008c0  00000000000000f1 
00007f600a7fb9b8:  0000000000000011  0000000000000000 
00007f600a7fb9c8: <00007f60120058f8  0000000000000020 
00007f600a7fb9d8:  0000000000000000  0000000000000000 
00007f600a7fb9e8:  0000000000000000  0000000000000000 
00007f600a7fb9f8:  0000000000000000  0000000000000000 
00007f600a7fba08:  0000000000000000  0000000000000000 
00007f600a7fba18:  0000000000000000  0000000000000000 
00007f600a7fba28:  0000000000000000  0000000000000000 
00007f600a7fba38:  0000000000000000  0000000000000000 
00007f600a7fba48:  0000000000000000  000000000000000d 
00007f600a7fba58:  00007f60120bcff9  0000000000000000 
00007f600a7fba68:  00007f60120480b3  000000000000000d 
00007f600a7fba78:  00007f60123951c0  0000000000000001 
00007f600a7fba88:  00007f6012395243  00007f60123913a0 
00007f600a7fba98:  00007f60120498ce  00007f60123951c0 
00007f600a7fbaa8:  000000000000000a  00007f5ffc0008c0 
00007f600a7fbab8:  00000000000000f1  0000000000000011 

goroutine 18 [running]:
runtime.systemstack_switch()
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:363 fp=0xc42008c550 sp=0xc42008c548 pc=0x45fac0
runtime.gcMarkTermination(0x3fe4959eb5700413)
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/mgc.go:1647 +0x407 fp=0xc42008c720 sp=0xc42008c550 pc=0x4208b7
runtime.gcMarkDone()
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/mgc.go:1513 +0x22c fp=0xc42008c748 sp=0xc42008c720 pc=0x42044c
runtime.gcBgMarkWorker(0xc420064500)
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/mgc.go:1912 +0x2e7 fp=0xc42008c7d8 sp=0xc42008c748 pc=0x4213c7
runtime.goexit()
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42008c7e0 sp=0xc42008c7d8 pc=0x462651
created by runtime.gcBgMarkStartWorkers
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/mgc.go:1723 +0x79

goroutine 1 [semacquire, locked to thread]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/conversion.ConversionFuncs.AddUntyped(0xc4207e6210, 0xc4207e6240, 0x26c83e0, 0x0, 0x24bb0e0, 0x0, 0x2a6f3c0, 0x0, 0x0)
	/builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/conversion/converter.go:192 +0x159
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/conversion.(*Converter).RegisterGeneratedUntypedConversionFunc(0xc4209d2b40, 0x26c83e0, 0x0, 0x24bb0e0, 0x0, 0x2a6f3c0, 0x0, 0x0)
	/builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/conversion/converter.go:397 +0x71
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime.(*Scheme).AddGeneratedConversionFunc(0xc420833180, 0x26c83e0, 0x0, 0x24bb0e0, 0x0, 0x2a6f3c0, 0x0, 0x0)
	/builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:361 +0x61
github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/apis/core/v1.RegisterConversions(0xc420833180, 0x0, 0x0)
	/builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/apis/core/v1/zz_generated.conversion.go:1083 +0x4bfb
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme(0x43ffc90, 0xc420833180, 0x2ccb3a0, 0xc420833ab0)
	/builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime/scheme_builder.go:29 +0x6d
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme-fm(0xc420833180, 0x0, 0x0)
	/builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/github.com/openshift/api/apps/install.go:17 +0x34
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme(0x43ff730, 0xc420833180, 0x0, 0x0)
	/builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime/scheme_builder.go:29 +0x6d
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme-fm(0xc420833180, 0x0, 0x0)
	/builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/github.com/openshift/api/apps/install.go:17 +0x34
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme(0x43feef0, 0xc420833180, 0xc420277ac0, 0x2ccf620)
	/builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime/scheme_builder.go:29 +0x6d
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme-fm(0xc420833180, 0x2ccf8a0, 0xc420bfba80)
	/builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/github.com/openshift/api/apps/install.go:17 +0x34
github.com/openshift/origin/pkg/security/apis/security/install.Install(0xc420833180)
	/builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/pkg/security/apis/security/install/install.go:18 +0x36
github.com/openshift/origin/pkg/security/generated/internalclientset/scheme.Install(0xc420833180)
	/builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/pkg/security/generated/internalclientset/scheme/register.go:24 +0x2b
github.com/openshift/origin/pkg/security/generated/internalclientset/scheme.init.0()
	/builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/pkg/security/generated/internalclientset/scheme/register.go:19 +0x5a

goroutine 5 [chan receive]:
github.com/openshift/origin/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x4402e40)
	/builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/github.com/golang/glog/glog.go:879 +0x8b
created by github.com/openshift/origin/vendor/github.com/golang/glog.init.0
	/builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/github.com/golang/glog/glog.go:410 +0x203

goroutine 15 [syscall]:
os/signal.signal_recv(0x0)
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/os/signal/signal_unix.go:28 +0x41

rax    0x0
rbx    0x7f6012395868
rcx    0xffffffffffffffff
rdx    0x6
rdi    0x723d
rsi    0x727e
rbp    0x2d95bf5
rsp    0x7f600a7fb9c8
r8     0xa
r9     0x7f600a7fc700
r10    0x8
r11    0x202
r12    0x7f5ffc0008c0
r13    0xf1
r14    0x11
r15    0x0
rip    0x7f6012004207
rflags 0x202
cs     0x33
fs     0x0
gs     0x0

Comment 2 Hongkai Liu 2019-04-30 15:17:55 UTC

This seems that it is an OS issue on the client side.
With all the other fixes in on the cli and running the test with a VM instead of the pbench-controller pod, clients returns no failures in the current run (4.1.0-0.nightly-2019-04-22-005054).
See more https://bugzilla.redhat.com/show_bug.cgi?id=1704722