1473777 – Hang during oadm drain node

Bug 1473777 - Hang during oadm drain node

Summary: Hang during oadm drain node

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Node
Sub Component:
Version:	3.6.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	urgent
Target Milestone:	---
Target Release:	3.7.0
Assignee:	Derek Carr
QA Contact:	weiwei jiang
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1473953
TreeView+	depends on / blocked

Reported:	2017-07-21 15:54 UTC by Justin Pierce
Modified:	2017-11-28 22:05 UTC (History)
CC List:	10 users (show)
Fixed In Version:
Doc Type:	No Doc Update
Doc Text:	undefined
Clone Of:
Clones:	1473953 (view as bug list)
Environment:
Last Closed:	2017-11-28 22:05:05 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	kubernetes kubernetes pull 49444	0	None	None	None	2017-07-22 17:45:41 UTC
Red Hat Product Errata	RHSA-2017:3188	0	normal	SHIPPED_LIVE	Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update	2017-11-29 02:34:54 UTC

Description Justin Pierce 2017-07-21 15:54:29 UTC

Description of problem:
During an invocation of oadm drain, the command hung in one of our online clusters (free-stg).

Version-Release number of selected component (if applicable):
oc v3.6.152.0
kubernetes v1.6.1+5115d708d7
features: Basic-Auth GSSAPI Kerberos SPNEGO

How reproducible:
Intermittent (oadm drain had run correctly several times on this node before hanging)

Steps to Reproduce:
1. oadm manage-node $x --schedulable=false
2. oadm drain $x --force --delete-local-data --ignore-daemonsets

Actual results:
oadm drain hangs

Expected results:
oadm should succeed in draining the node.

Additional info:

The output from oadm drain:
node "ip-172-31-71-195.us-east-2.compute.internal" already cordoned
WARNING: Deleting pods with local storage: hello-statefulset-1, mongodb-2-zpqv6, oso-clamd-0c5cm, mysql-8-tzmh1; Ignoring DaemonSet-managed pods: oso-clamd-0c5cm; Deleting pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: pull-07201920z-5c-1-deploy
pod "mongodb-6-kbqg3" evicted
pod "hello-statefulset-1" evicted
pod "nodejs-mongodb-example-1-l2sc7" evicted
pod "mongodb-6-rwgn0" evicted
pod "dancer-mysql-persistent-1-w6dfl" evicted
pod "nodejs-mongodb-example-1-lbvjl" evicted
pod "mongodb-2-zpqv6" evicted
pod "cakephp-mysql-persistent-5-pjpzq" evicted
pod "nodejs-mongodb-example-1-xp36w" evicted
pod "mongodb-6-644rz" evicted
pod "postgresql-1-xtm2x" evicted
pod "rails-pgsql-persistent-1-3kph0" evicted
pod "myphp-4-804j8" evicted
pod "mysql-8-tzmh1" evicted
pod "django-psql-persistent-1-629jk" evicted
pod "rails-pgsql-persistent-4-0lwtn" evicted
pod "pull-06282010z-mt-1-c0pn2" evicted
pod "nodejs-mongodb-example-1-p3gnb" evicted
pod "postgresql-1-zvpzs" evicted
pod "frontend-1-mhkvq" evicted
pod "testdc-1-v2rfq" evicted
pod "rails-ex-3-571g2" evicted
pod "nodejs-mongo-persistent-2-sr7hj" evicted
pod "mysql-8-kq8lw" evicted
pod "nodejs-mongodb-example-1-8006d" evicted
pod "mongodb-6-nvxsp" evicted
pod "cakephp-example-5-vbx0d" evicted
pod "hello-idle-4wqb9" evicted
pod "deploymentconfig2v1-1-s8q2j" evicted
pod "nodejs-mongodb-example-1-l3qlb" evicted
pod "nodejs-mongodb-example-1-rg6fh" evicted
pod "nodejs-mongodb-example-1-56kdf" evicted
pod "cakephp-example-3-6jwnb" evicted
pod "cakephp-example-4-n6fft" evicted
pod "njs-3-1r2bq" evicted
pod "nodejs-mongodb-example-1-0f14l" evicted
pod "pull-06130440z-15-1-3qwj3" evicted
pod "cakephp-example-4-x1g1f" evicted
pod "jerms-eap7-4-sqgxh" evicted
pod "cakephp-example-4-9cmsg" evicted
pod "test-1-q39cd" evicted







The stack trace for the oadm process:
SIGABRT: abort
PC=0x463551 m=0

goroutine 0 [idle]:
runtime.futex(0xc029130, 0x0, 0x0, 0x0, 0xc400000000, 0x0, 0x0, 0x0, 0x7fff4b936d80, 0x411f82, ...)
        /usr/lib/golang/src/runtime/sys_linux_amd64.s:387 +0x21
runtime.futexsleep(0xc029130, 0x0, 0xffffffffffffffff)
        /usr/lib/golang/src/runtime/os_linux.go:45 +0x62
runtime.notesleep(0xc029130)
        /usr/lib/golang/src/runtime/lock_futex.go:145 +0x82
runtime.stopm()
        /usr/lib/golang/src/runtime/proc.go:1594 +0xad
runtime.findrunnable(0xc42001f500, 0x0)
        /usr/lib/golang/src/runtime/proc.go:2021 +0x228
runtime.schedule()
        /usr/lib/golang/src/runtime/proc.go:2120 +0x14c
runtime.park_m(0xc421061380)
        /usr/lib/golang/src/runtime/proc.go:2183 +0x123
runtime.mcall(0x7fff4b936f20)
        /usr/lib/golang/src/runtime/asm_amd64.s:240 +0x5b

goroutine 1 [select, 1214 minutes]:
github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd.(*DrainOptions).evictPods(0xc4210b2bb0, 0xc42181a000, 0x2a, 0x45, 0xc4212ae690, 0xe, 0xc421341740, 0x0, 0x0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd/drain.go:523 +0x44d
github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd.(*DrainOptions).deleteOrEvictPods(0xc4210b2bb0, 0xc42181a000, 0x2a, 0x45, 0x0, 0x0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd/drain.go:480 +0xfd
github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd.(*DrainOptions).deleteOrEvictPodsSimple(0xc4210b2bb0, 0x1, 0x0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd/drain.go:255 +0x85
github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd.(*DrainOptions).RunDrain(0xc4210b2bb0, 0x0, 0xc42113ec40)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd/drain.go:242 +0x5b
github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd.NewCmdDrain.func1(0xc420170900, 0xc42113ec40, 0x1, 0x4)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd/drain.go:183 +0x81
github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).execute(0xc420170900, 0xc42113ec00, 0x4, 0x4, 0xc420170900, 0xc42113ec00)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:603 +0x439
github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4201c2fc0, 0xc42002e008, 0xc42002e018, 0xc4201c2fc0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:689 +0x367
github.com/openshift/origin/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4201c2fc0, 0x4, 0xc4201c2fc0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/github.com/spf13/cobra/command.go:648 +0x2b
main.main()
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/cmd/openshift/openshift.go:36 +0x196

goroutine 17 [syscall, 1216 minutes, locked to thread]:
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:2086 +0x1

goroutine 5 [chan receive]:
github.com/openshift/origin/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0xc027860)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/github.com/golang/glog/glog.go:879 +0x7a
created by github.com/openshift/origin/vendor/github.com/golang/glog.init.1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/github.com/golang/glog/glog.go:410 +0x21d

goroutine 292 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc421199860, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 19 [syscall, 1216 minutes]:
os/signal.signal_recv(0x68b6)
        /usr/lib/golang/src/runtime/sigqueue.go:116 +0x157
os/signal.loop()
        /usr/lib/golang/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
        /usr/lib/golang/src/os/signal/signal_unix.go:28 +0x41

goroutine 98 [chan receive]:
github.com/openshift/origin/vendor/github.com/coreos/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc420fcd240)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/pkg/logutil/merge_logger.go:174 +0x94
created by github.com/openshift/origin/vendor/github.com/coreos/etcd/pkg/logutil.NewMergeLogger
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/pkg/logutil/merge_logger.go:92 +0xd2

goroutine 76 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x7c52d58, 0x12a05f200, 0x0, 0x1803a01, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:108 +0x16e
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0x7c52d58, 0x12a05f200, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x4d
created by github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/logs.InitLogs
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/logs/logs.go:50 +0xa8

goroutine 112 [chan receive]:
github.com/openshift/origin/vendor/github.com/coreos/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc420fcd9c0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/pkg/logutil/merge_logger.go:174 +0x94
created by github.com/openshift/origin/vendor/github.com/coreos/etcd/pkg/logutil.NewMergeLogger
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/pkg/logutil/merge_logger.go:92 +0xd2

goroutine 463 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc421999200, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 484 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc4219d8240, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 449 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc42058bb60, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 216 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc4212271a0, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 321 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc421165b00, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 476 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc42192acc0, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 405 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc420b8b140, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 254 [sleep]:
time.Sleep(0xb780095)
        /usr/lib/golang/src/runtime/time.go:59 +0xe1
github.com/openshift/origin/vendor/github.com/juju/ratelimit.realClock.Sleep(0xb780095)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/github.com/juju/ratelimit/ratelimit.go:53 +0x2b
github.com/openshift/origin/vendor/github.com/juju/ratelimit.(*realClock).Sleep(0xc049ac0, 0xb780095)
        <autogenerated>:4 +0x56
github.com/openshift/origin/vendor/github.com/juju/ratelimit.(*Bucket).Wait(0xc4212041e0, 0x1)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/github.com/juju/ratelimit/ratelimit.go:143 +0x5b
github.com/openshift/origin/vendor/k8s.io/client-go/util/flowcontrol.(*tokenBucketRateLimiter).Accept(0xc4211d33c0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/util/flowcontrol/throttle.go:88 +0x37
github.com/openshift/origin/vendor/k8s.io/client-go/rest.(*Request).tryThrottle(0xc42049d380)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/rest/request.go:635 +0x219
github.com/openshift/origin/vendor/k8s.io/client-go/rest.(*Request).Do(0xc42049d380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/rest/request.go:890 +0x62
github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset/typed/policy/internalversion.(*evictions).Evict(0xc421745dc0, 0xc42118cc60, 0x15, 0xa384fe0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset/typed/policy/internalversion/eviction_expansion.go:36 +0x15c
github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd.(*DrainOptions).evictPod(0xc4210b2bb0, 0x0, 0x0, 0x0, 0x0, 0xc420803080, 0x1a, 0x0, 0x0, 0xc4208030c0, ...)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd/drain.go:461 +0x189
github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd.(*DrainOptions).evictPods.func1(0xc4210b2bb0, 0xc4212ae690, 0xe, 0xc421341740, 0x0, 0x0, 0x0, 0x0, 0xc420803080, 0x1a, ...)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd/drain.go:494 +0x90
created by github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd.(*DrainOptions).evictPods
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubectl/cmd/drain.go:511 +0x19b

goroutine 343 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc420b8a180, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 297 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc420fe40c0, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 288 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc42110f8c0, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 307 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc421164720, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 270 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc421205f80, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 294 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc421199a40, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 311 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc421164cc0, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 222 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc421227b60, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 301 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc420fe49c0, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 303 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc420fe4cc0, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 339 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc42110fc80, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 322 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc420fe4de0, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 349 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc420b8a900, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 325 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc420f90120, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 341 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc420b8a060, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 328 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc420f90360, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 370 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc420f91560, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 346 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc420b8a540, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 353 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc420b8ad80, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 376 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc420f91b00, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 409 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc420b8b9e0, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 397 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc4208a0780, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 395 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc4208a0360, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 435 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc42050f140, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 487 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc4219d8ae0, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 446 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc42058b860, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 460 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc421998f00, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 470 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc421021c20, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 493 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc421998000, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 479 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc42192afc0, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 502 [select]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc42192bc20, 0x3b9aca00, 0x7fffffffffffffff, 0xc42000c1e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x1fd
created by github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:328 +0x91

goroutine 662132 [IO wait]:
net.runtime_pollWait(0x7fe2efd70a58, 0x72, 0x6)
        /usr/lib/golang/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc421129170, 0x72, 0xc42003d808, 0xc4200140c8)
        /usr/lib/golang/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc421129170, 0xa394a20, 0xc4200140c8)
        /usr/lib/golang/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc421129110, 0xc420fb0800, 0x800, 0x800, 0x0, 0xa394a20, 0xc4200140c8)
        /usr/lib/golang/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc420088ba8, 0xc420fb0800, 0x800, 0x800, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/net/net.go:173 +0x70
crypto/tls.(*block).readFromUntil(0xc42162e450, 0xa38aaa0, 0xc420088ba8, 0x5, 0xc420088ba8, 0x0)
        /usr/lib/golang/src/crypto/tls/conn.go:476 +0x91
crypto/tls.(*Conn).readRecord(0xc421368000, 0x7c61617, 0xc421368108, 0x0)
        /usr/lib/golang/src/crypto/tls/conn.go:578 +0xc4
crypto/tls.(*Conn).Read(0xc421368000, 0xc420f83000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/crypto/tls/conn.go:1113 +0x116
bufio.(*Reader).fill(0xc421ccda40)
        /usr/lib/golang/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Read(0xc421ccda40, 0xc421197238, 0x9, 0x9, 0xc42003dcc0, 0x441ae0, 0x42e5cc)
        /usr/lib/golang/src/bufio/bufio.go:209 +0x1bc
io.ReadAtLeast(0xa369060, 0xc421ccda40, 0xc421197238, 0x9, 0x9, 0x9, 0xc4219e82c0, 0xc42003dd10, 0x1edbe15)
        /usr/lib/golang/src/io/io.go:307 +0xa4
io.ReadFull(0xa369060, 0xc421ccda40, 0xc421197238, 0x9, 0x9, 0xc4216aa8a0, 0x50c40ae, 0xa)
        /usr/lib/golang/src/io/io.go:325 +0x58
github.com/openshift/origin/vendor/golang.org/x/net/http2.readFrameHeader(0xc421197238, 0x9, 0x9, 0xa369060, 0xc421ccda40, 0x0, 0xc400000000, 0x28bfe57, 0xc42003df80)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/openshift/origin/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc421197200, 0xc4216aaa20, 0x0, 0x0, 0x0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/golang.org/x/net/http2/frame.go:464 +0xa4
github.com/openshift/origin/vendor/golang.org/x/net/http2.(*clientConnReadLoop).run(0xc42003df80, 0x7c55988, 0xc420a89790)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/golang.org/x/net/http2/transport.go:1233 +0x97
github.com/openshift/origin/vendor/golang.org/x/net/http2.(*ClientConn).readLoop(0xc4213531e0)
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/golang.org/x/net/http2/transport.go:1170 +0xa6
created by github.com/openshift/origin/vendor/golang.org/x/net/http2.(*Transport).NewClientConn
        /builddir/build/BUILD/atomic-openshift-git-0.2618a70/_output/local/go/src/github.com/openshift/origin/vendor/golang.org/x/net/http2/transport.go:492 +0x89b

rax    0xca
rbx    0xc027d90
rcx    0xffffffffffffffff
rdx    0x0
rdi    0xc029130
rsi    0x0
rbp    0x7fff4b936d50
rsp    0x7fff4b936d08
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0xc4201055a8
r13    0x0
r14    0xc420105500
r15    0x0
rip    0x463551
rflags 0x286
cs     0x33
fs     0x0
gs     0x0


The output of docker ps (after oadm was killed with signal; node still unschedulable):
[root@free-stg-node-compute-d76a8 ~]# docker ps
CONTAINER ID        IMAGE                                                                                                                                COMMAND                  CREATED             STATUS              PORTS               NAMES
49f069f9d9fd        172.30.44.192:5000/management-infra/oso-rhel7-clamd-update@sha256:8bf75d87ffe3b31cd9c591590d17bd6a8d29855a5e796a1cb1a8c374fdad2ea9   "/bin/sh -c /usr/loca"   18 hours ago        Up 18 hours                             k8s_oso-clamd-update_oso-clamd-g7fnf_management-infra_5d05106b-6d91-11e7-91e2-02306c0cdc4b_0
a2026861d271        registry.ops.openshift.com/openshift3/ose-pod:v3.6.152.0                                                                             "/usr/bin/pod"           18 hours ago        Up 18 hours                             k8s_POD_oso-clamd-g7fnf_management-infra_5d05106b-6d91-11e7-91e2-02306c0cdc4b_0
ffa53a9f8a19        registry.ops.openshift.com/ops/oso-rhel7-host-monitoring:stg                                                                         "/bin/sh -c /usr/loca"   40 hours ago        Up 40 hours                             oso-rhel7-host-monitoring

Comment 1 Eric Paris 2017-07-22 17:45:41 UTC

I think this would be fixed by the upstream PR https://github.com/kubernetes/kubernetes/pull/49444

Comment 2 Jeff Peeler 2017-07-24 17:34:29 UTC

https://github.com/openshift/origin/pull/15435

Comment 4 weiwei jiang 2017-09-27 05:40:29 UTC

Checked with 
# openshift version 
openshift v3.7.0-0.127.0
kubernetes v1.7.0+80709908fd
etcd 3.2.1

and can not reproduce this issue.

Comment 7 errata-xmlrpc 2017-11-28 22:05:05 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188

Note You need to log in before you can comment on or make changes to this bug.