Description of problem: concurrent builds on scale-lab: 25 2xlarge workers. 750 projects for nodejs app. Durring the test, we called many `oc start-build -n ...` in 50 threads for 750 projects and the return has many `Resource temporarily unavailable`: about 50% of them. Version-Release number of selected component (if applicable): 4.0.0-0.nightly-2019-03-04-234414 How reproducible: always Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: log file from python build test script: http://pbench.perf.lab.eng.bos.redhat.com/results/EC2::ip-10-0-14-191/ocp4_conc_build_201903/pbench-user-benchmark_buildlog_2019.03.18T20.27.50/1/reference-result/build_test.log
Here's the crash info for the CLI: SIGABRT: abort PC=0x7f6012004207 m=8 sigcode=18446744073709551610 goroutine 0 [idle]: runtime: unknown pc 0x7f6012004207 stack: frame={sp:0x7f600a7fb9c8, fp:0x0} stack=[0x7f6009ffc2b0,0x7f600a7fbeb0) 00007f600a7fb8c8: 0000000000000000 0000000000000000 00007f600a7fb8d8: 0000000000000000 0000000000000000 00007f600a7fb8e8: 0000000000000000 0000000000000000 00007f600a7fb8f8: 0000000000000000 0000000000000000 00007f600a7fb908: 0000000000000000 0000000000000000 00007f600a7fb918: 0000000000000000 0000000000000000 00007f600a7fb928: 0000000000000000 0000000000000000 00007f600a7fb938: 0000000000000000 0000000000000000 00007f600a7fb948: 0000000000000000 0000000000000000 00007f600a7fb958: 0000000000000000 0000000000000000 00007f600a7fb968: 0000000000000000 0000000000000000 00007f600a7fb978: 0000000000000000 0000000000000000 00007f600a7fb988: 0000000000000000 0000000000000000 00007f600a7fb998: 00007f6012395868 0000000002d95bf5 00007f600a7fb9a8: 00007f5ffc0008c0 00000000000000f1 00007f600a7fb9b8: 0000000000000011 0000000000000000 00007f600a7fb9c8: <00007f60120058f8 0000000000000020 00007f600a7fb9d8: 0000000000000000 0000000000000000 00007f600a7fb9e8: 0000000000000000 0000000000000000 00007f600a7fb9f8: 0000000000000000 0000000000000000 00007f600a7fba08: 0000000000000000 0000000000000000 00007f600a7fba18: 0000000000000000 0000000000000000 00007f600a7fba28: 0000000000000000 0000000000000000 00007f600a7fba38: 0000000000000000 0000000000000000 00007f600a7fba48: 0000000000000000 000000000000000d 00007f600a7fba58: 00007f60120bcff9 0000000000000000 00007f600a7fba68: 00007f60120480b3 000000000000000d 00007f600a7fba78: 00007f60123951c0 0000000000000001 00007f600a7fba88: 00007f6012395243 00007f60123913a0 00007f600a7fba98: 00007f60120498ce 00007f60123951c0 00007f600a7fbaa8: 000000000000000a 00007f5ffc0008c0 00007f600a7fbab8: 00000000000000f1 0000000000000011 runtime: unknown pc 0x7f6012004207 stack: frame={sp:0x7f600a7fb9c8, fp:0x0} stack=[0x7f6009ffc2b0,0x7f600a7fbeb0) 00007f600a7fb8c8: 0000000000000000 0000000000000000 00007f600a7fb8d8: 0000000000000000 0000000000000000 00007f600a7fb8e8: 0000000000000000 0000000000000000 00007f600a7fb8f8: 0000000000000000 0000000000000000 00007f600a7fb908: 0000000000000000 0000000000000000 00007f600a7fb918: 0000000000000000 0000000000000000 00007f600a7fb928: 0000000000000000 0000000000000000 00007f600a7fb938: 0000000000000000 0000000000000000 00007f600a7fb948: 0000000000000000 0000000000000000 00007f600a7fb958: 0000000000000000 0000000000000000 00007f600a7fb968: 0000000000000000 0000000000000000 00007f600a7fb978: 0000000000000000 0000000000000000 00007f600a7fb988: 0000000000000000 0000000000000000 00007f600a7fb998: 00007f6012395868 0000000002d95bf5 00007f600a7fb9a8: 00007f5ffc0008c0 00000000000000f1 00007f600a7fb9b8: 0000000000000011 0000000000000000 00007f600a7fb9c8: <00007f60120058f8 0000000000000020 00007f600a7fb9d8: 0000000000000000 0000000000000000 00007f600a7fb9e8: 0000000000000000 0000000000000000 00007f600a7fb9f8: 0000000000000000 0000000000000000 00007f600a7fba08: 0000000000000000 0000000000000000 00007f600a7fba18: 0000000000000000 0000000000000000 00007f600a7fba28: 0000000000000000 0000000000000000 00007f600a7fba38: 0000000000000000 0000000000000000 00007f600a7fba48: 0000000000000000 000000000000000d 00007f600a7fba58: 00007f60120bcff9 0000000000000000 00007f600a7fba68: 00007f60120480b3 000000000000000d 00007f600a7fba78: 00007f60123951c0 0000000000000001 00007f600a7fba88: 00007f6012395243 00007f60123913a0 00007f600a7fba98: 00007f60120498ce 00007f60123951c0 00007f600a7fbaa8: 000000000000000a 00007f5ffc0008c0 00007f600a7fbab8: 00000000000000f1 0000000000000011 goroutine 18 [running]: runtime.systemstack_switch() /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:363 fp=0xc42008c550 sp=0xc42008c548 pc=0x45fac0 runtime.gcMarkTermination(0x3fe4959eb5700413) /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/mgc.go:1647 +0x407 fp=0xc42008c720 sp=0xc42008c550 pc=0x4208b7 runtime.gcMarkDone() /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/mgc.go:1513 +0x22c fp=0xc42008c748 sp=0xc42008c720 pc=0x42044c runtime.gcBgMarkWorker(0xc420064500) /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/mgc.go:1912 +0x2e7 fp=0xc42008c7d8 sp=0xc42008c748 pc=0x4213c7 runtime.goexit() /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42008c7e0 sp=0xc42008c7d8 pc=0x462651 created by runtime.gcBgMarkStartWorkers /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/mgc.go:1723 +0x79 goroutine 1 [semacquire, locked to thread]: github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/conversion.ConversionFuncs.AddUntyped(0xc4207e6210, 0xc4207e6240, 0x26c83e0, 0x0, 0x24bb0e0, 0x0, 0x2a6f3c0, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/conversion/converter.go:192 +0x159 github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/conversion.(*Converter).RegisterGeneratedUntypedConversionFunc(0xc4209d2b40, 0x26c83e0, 0x0, 0x24bb0e0, 0x0, 0x2a6f3c0, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/conversion/converter.go:397 +0x71 github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime.(*Scheme).AddGeneratedConversionFunc(0xc420833180, 0x26c83e0, 0x0, 0x24bb0e0, 0x0, 0x2a6f3c0, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:361 +0x61 github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/apis/core/v1.RegisterConversions(0xc420833180, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/apis/core/v1/zz_generated.conversion.go:1083 +0x4bfb github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme(0x43ffc90, 0xc420833180, 0x2ccb3a0, 0xc420833ab0) /builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime/scheme_builder.go:29 +0x6d github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme-fm(0xc420833180, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/github.com/openshift/api/apps/install.go:17 +0x34 github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme(0x43ff730, 0xc420833180, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime/scheme_builder.go:29 +0x6d github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme-fm(0xc420833180, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/github.com/openshift/api/apps/install.go:17 +0x34 github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme(0x43feef0, 0xc420833180, 0xc420277ac0, 0x2ccf620) /builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime/scheme_builder.go:29 +0x6d github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme-fm(0xc420833180, 0x2ccf8a0, 0xc420bfba80) /builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/github.com/openshift/api/apps/install.go:17 +0x34 github.com/openshift/origin/pkg/security/apis/security/install.Install(0xc420833180) /builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/pkg/security/apis/security/install/install.go:18 +0x36 github.com/openshift/origin/pkg/security/generated/internalclientset/scheme.Install(0xc420833180) /builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/pkg/security/generated/internalclientset/scheme/register.go:24 +0x2b github.com/openshift/origin/pkg/security/generated/internalclientset/scheme.init.0() /builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/pkg/security/generated/internalclientset/scheme/register.go:19 +0x5a goroutine 5 [chan receive]: github.com/openshift/origin/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x4402e40) /builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/github.com/golang/glog/glog.go:879 +0x8b created by github.com/openshift/origin/vendor/github.com/golang/glog.init.0 /builddir/build/BUILD/atomic-openshift-git-0.85336f8/_output/local/go/src/github.com/openshift/origin/vendor/github.com/golang/glog/glog.go:410 +0x203 goroutine 15 [syscall]: os/signal.signal_recv(0x0) /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/sigqueue.go:139 +0xa6 os/signal.loop() /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/os/signal/signal_unix.go:22 +0x22 created by os/signal.init.0 /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/os/signal/signal_unix.go:28 +0x41 rax 0x0 rbx 0x7f6012395868 rcx 0xffffffffffffffff rdx 0x6 rdi 0x723d rsi 0x727e rbp 0x2d95bf5 rsp 0x7f600a7fb9c8 r8 0xa r9 0x7f600a7fc700 r10 0x8 r11 0x202 r12 0x7f5ffc0008c0 r13 0xf1 r14 0x11 r15 0x0 rip 0x7f6012004207 rflags 0x202 cs 0x33 fs 0x0 gs 0x0
This seems that it is an OS issue on the client side. With all the other fixes in on the cli and running the test with a VM instead of the pbench-controller pod, clients returns no failures in the current run (4.1.0-0.nightly-2019-04-22-005054). See more https://bugzilla.redhat.com/show_bug.cgi?id=1704722