Bug 1593635

Summary:	After update the master-config.yaml, the api and controller pod will gone
Product:	OpenShift Container Platform	Reporter:	zhou ying <yinzhou>
Component:	Test Infrastructure	Assignee:	Ryan Phillips <rphillips>
Status:	CLOSED CURRENTRELEASE	QA Contact:	libra bugs <libra-bugs>
Severity:	high	Docs Contact:
Priority:	high
Version:	3.10.0	CC:	aos-bugs, avagarwa, dyan, haowang, jokerman, mmccomas, rphillips, sjenning, tcarlin, wsun, xtian, xxia, yinzhou
Target Milestone:	---
Target Release:	3.10.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:	undefined	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-12-20 21:42:13 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description zhou ying 2018-06-21 09:29:42 UTC

Description of problem:
add 'masterConfig.ImagePolicyConfig.ExternalRegistryHostname' for the master-config.yaml, the restart the api and controller service, the api pod will always recreate, the controllers pod will 'CrashLoopBackOff'

Version-Release number of selected component (if applicable):
v3.10.2

How reproducible:
30%

Steps to Reproduce:
1. Update the master-config.yaml with :
imagePolicyConfig:
  externalRegistryHostname: host of the registry's route

2. Try to restart the api and controller service
3. Even recover the master-config.yaml, the api and controller still not be running.

Actual results:
2. The api pod will always recreate and the controller pod will 'CrashLoopBackOff'

Expected results:
2. The api and controller pod restart succeed

Additional info:
docker version: docker-1.13.1-63.git94f4240.el7.x86_64

The api pod didn't have any error info, only could get some error info from the docker log:

[root@ip-172-18-8-18 cron.hourly]# docker ps -a |grep master-api
f2262dec097e        22afa152a447                                                                                                                                        "/bin/bash -c '#!/..."   About a minute ago   Exited (137) 25 seconds ago                           k8s_api_master-api-ip-172-18-8-18.ec2.internal_kube-system_f8ddfa563d27a7f9ca95644fe2f8cdd0_11
6be5580e806b        registry.reg-aws.openshift.com:443/openshift3/ose-pod:v3.10.2     

[root@ip-172-18-8-18 master]# journalctl -f -u docker |grep f2262dec097e44
Jun 21 05:02:17 ip-172-18-8-18.ec2.internal dockerd-current[4118]: time="2018-06-21T05:02:17.490491984-04:00" level=warning msg="Unknown healthcheck type 'NONE' (expected 'CMD') in container f2262dec097e445eadc788bb54600533744aeaf3e73a5b4583c549d059a7c240"
Jun 21 05:03:47 ip-172-18-8-18.ec2.internal dockerd-current[4118]: time="2018-06-21T05:03:47.976889423-04:00" level=info msg="Container f2262dec097e445eadc788bb54600533744aeaf3e73a5b4583c549d059a7c240 failed to exit within 30 seconds of signal 15 - using the force"
Jun 21 05:03:48 ip-172-18-8-18.ec2.internal dockerd-current[4118]: time="2018-06-21T05:03:48.209765256-04:00" level=error msg="containerd: deleting container" error="exit status 1: \"container f2262dec097e445eadc788bb54600533744aeaf3e73a5b4583c549d059a7c240 does not exist\\none or more of the container deletions failed\\n\""
Jun 21 05:03:48 ip-172-18-8-18.ec2.internal dockerd-current[4118]: time="2018-06-21T05:03:48.223903073-04:00" level=error msg="Error closing logger: close /var/lib/docker/containers/f2262dec097e445eadc788bb54600533744aeaf3e73a5b4583c549d059a7c240/f2262dec097e445eadc788bb54600533744aeaf3e73a5b4583c549d059a7c240-json.log: file already closed"
Jun 21 05:03:48 ip-172-18-8-18.ec2.internal dockerd-current[4118]: time="2018-06-21T05:03:48.306809354-04:00" level=warning msg="f2262dec097e445eadc788bb54600533744aeaf3e73a5b4583c549d059a7c240 cleanup: failed to unmount secrets: invalid argument"

Logs from the controller pod:

E0621 09:10:42.742822       1 reflector.go:205] github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:125: Failed to list <nil>: Get https://ip-172-18-8-18.ec2.internal:8443/apis/automationbroker.io/v1alpha1/bundlebindings?limit=500&resourceVersion=0: dial tcp 172.18.8.18:8443: getsockopt: connection refused
I0621 09:10:42.746636       1 reconciler.go:112] Skipping reconciling attached volumes still attached since it is disabled via the command line.
I0621 09:10:42.846752       1 reconciler.go:112] Skipping reconciling attached volumes still attached since it is disabled via the command line.
I0621 09:10:42.931206       1 reflector.go:240] Listing and watching <nil> from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:125
I0621 09:10:42.933086       1 reflector.go:240] Listing and watching *v1beta1.VolumeAttachment from github.com/openshift/origin/vendor/k8s.io/client-go/informers/factory.go:87
E0621 09:10:42.933526       1 reflector.go:205] github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:125: Failed to list <nil>: Get https://ip-172-18-8-18.ec2.internal:8443/apis/servicecatalog.k8s.io/v1beta1/clusterserviceclasses?limit=500&resourceVersion=0: dial tcp 172.18.8.18:8443: getsockopt: connection refused
E0621 09:10:42.933591       1 leaderelection.go:224] error retrieving resource lock kube-system/kube-controller-manager: Get https://ip-172-18-8-18.ec2.internal:8443/api/v1/namespaces/kube-system/configmaps/kube-controller-manager: dial tcp 172.18.8.18:8443: getsockopt: connection refused
E0621 09:10:42.933622       1 event.go:260] Could not construct reference to: '&v1.ConfigMap{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Data:map[string]string(nil), BinaryData:map[string][]uint8(nil)}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'LeaderElection' 'ip-172-18-8-18.ec2.internal_8c58a1a1-7532-11e8-95da-0e77881bc7be stopped leading'
I0621 09:10:42.933675       1 leaderelection.go:203] failed to renew lease kube-system/kube-controller-manager: timed out waiting for the condition
F0621 09:10:42.933712       1 controllermanager.go:216] leaderelection lost

Comment 2 Michal Fojtik 2018-06-21 09:57:03 UTC

Strange, it does not seems this is kubelet deleting the container because it fails health checks (the healthz probe succeed after ~20s after start). 

The error in Docker logs suggests that the container did not died on its own (segfault or panic) and also there are no logs in api container that proving it.

The Docker log seems to suggests that "something" killed that container. Moving to Pod team to investigate (but I haven't seen a kill due to liveness probe executed from kubelet).

Comment 3 Avesh Agarwal 2018-06-21 17:35:31 UTC

I am looking into it.

Comment 4 Avesh Agarwal 2018-06-21 17:47:18 UTC

Not related to this, but just curious but wondering why docker is setup with devicemapper when the default is overlay2?

Comment 5 Wang Haoran 2018-06-22 07:53:14 UTC

(In reply to Avesh Agarwal from comment #4)
> Not related to this, but just curious but wondering why docker is setup with
> devicemapper when the default is overlay2?

We are testing different martrix, uppper the priority, as this should be an release blocker

Comment 6 Avesh Agarwal 2018-06-22 15:27:45 UTC

Zhou,

Could you share what exact value you gave to ExternalRegistryHostname? I am having hard time reproducing it on my cluster.

Comment 7 Avesh Agarwal 2018-06-22 15:31:17 UTC

(In reply to Avesh Agarwal from comment #6)
> Zhou,
> 
> Could you share what exact value you gave to ExternalRegistryHostname? I am
> having hard time reproducing it on my cluster.

Or if you could share your exact master-config.yaml that would be helpful too.

Comment 9 Avesh Agarwal 2018-06-22 17:51:42 UTC

I looked into logs from one instance of exited api server pod, and I noticed following stack strace:

docker logs 865bfa8feee1:

--------------------------
Trace[1707801746]: [1.959416ms] [1.959416ms] Recorded the audit event
Trace[1707801746]: [2.128801ms] [169.385µs] About to apply patch
Trace[1707801746]: [735.820788ms] [733.691987ms] About to check admission control
Trace[1707801746]: [10.251151633s] [9.515330845s] END
I0622 17:41:35.243070       1 wrap.go:42] PATCH /api/v1/nodes/ip-172-18-8-18.ec2.internal/status?timeout=10s: (10.251825797s) 504
goroutine 2903 [running]:
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc4248112d0, 0x1f8)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc4248112d0, 0x1f8)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc43139d980, 0x1f8)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:188 +0xac
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics.(*ResponseWriterDelegator).WriteHeader(0xc430e8fa10, 0x1f8)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:281 +0x45
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.SerializeObject(0x5aef814, 0x23, 0x7fde84c598a8, 0xc4308d8a80, 0xacc5e40, 0xc433def4e0, 0xc430c60800, 0x1f8, 0xac9df80, 0xc437871e60)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:95 +0x8d
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.WriteObjectNegotiated(0x7fde80a1e320, 0xc431aaff20, 0xacce140, 0xc42926b5c0, 0x0, 0x0, 0x5a48e11, 0x2, 0xacc5e40, 0xc433def4e0, ...)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:123 +0x32b
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.ErrorNegotiated(0x7fde80a1e320, 0xc431aaff20, 0xac7e5c0, 0xc437871dd0, 0xacce140, 0xc42926b5c0, 0x0, 0x0, 0x5a48e11, 0x2, ...)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:142 +0x163
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers.(*RequestScope).err(0xc423bdf400, 0xac7e5c0, 0xc437871dd0, 0xacc5e40, 0xc433def4e0, 0xc430c60800)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:67 +0x10c
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers.PatchResource.func1(0xacc5e40, 0xc433def4e0, 0xc430c60800)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/patch.go:130 +0x1205
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints.restfulPatchResource.func1(0xc430e8f950, 0xc43285a840)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1052 +0x12c
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc430e8f950, 0xc43285a840)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:199 +0x208
github.com/openshift/origin/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc427078750, 0x7fde80381330, 0xc433def4d0, 0xc430c60800)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/github.com/emicklei/go-restful/container.go:277 +0xb18
github.com/openshift/origin/vendor/github.com/emicklei/go-restful.(*Container).Dispatch(0xc427078750, 0x7fde80381330, 0xc433def4d0, 0xc430c60800)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/github.com/emicklei/go-restful/container.go:199 +0x57
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5a68967, 0xe, 0xc427078750, 0xc4250a72d0, 0x7fde80381330, 0xc433def4d0, 0xc430c60800)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:152 +0x4e0
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc4298228c0, 0x7fde80381330, 0xc433def4d0, 0xc430c60800)
	<autogenerated>:1 +0x75
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc4307716c0, 0x7fde80381330, 0xc433def4d0, 0xc430c60800)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc42d4bb0a0, 0x7fde80381330, 0xc433def4d0, 0xc430c60800)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5a6d554, 0xf, 0xc427078360, 0xc42d4bb0a0, 0x7fde80381330, 0xc433def4d0, 0xc430c60800)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc4234bec80, 0x7fde80381330, 0xc433def4d0, 0xc430c60800)
	<autogenerated>:1 +0x75
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7fde80381330, 0xc433def4d0, 0xc430c60800)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:52 +0x37d
net/http.HandlerFunc.ServeHTTP(0xc424618cd0, 0x7fde80381330, 0xc433def4d0, 0xc430c60800)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7fde80381330, 0xc433def4d0, 0xc430c60800)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:165 +0x42a
net/http.HandlerFunc.ServeHTTP(0xc42bcbc8c0, 0x7fde80381330, 0xc433def4d0, 0xc430c60800)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7fde80381330, 0xc433def4d0, 0xc430c60800)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a
net/http.HandlerFunc.ServeHTTP(0xc424618d70, 0x7fde80381330, 0xc433def4d0, 0xc430c60800)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7fde80381330, 0xc433def4d0, 0xc430c60800)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:78 +0x2b1
net/http.HandlerFunc.ServeHTTP(0xc424619090, 0x7fde80381330, 0xc433def4d0, 0xc430c60800)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1(0x7fde80381330, 0xc433def4d0, 0xc430c60800)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb
net/http.HandlerFunc.ServeHTTP(0xc4234bed60, 0x7fde80381330, 0xc433def4d0, 0xc430c60800)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1(0x7fde80381330, 0xc433def4d0, 0xc430c60800)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x17b
net/http.HandlerFunc.ServeHTTP(0xc42afdac00, 0x7fde80381330, 0xc433def4d0, 0xc430c60800)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc4234bf1c0, 0xacd1e40, 0xc433def4d0, 0xc430c60800, 0xc42a8862a0)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d
created by github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab

logging error output: "k8s\x00\n\f\n\x02v1\x12\x06Status\x12d\n\x06\n\x00\x12\x00\x1a\x00\x12\aFailure\x1a9Timeout: request did not complete within allowed duration\"\aTimeout*\n\n\x00\x12\x00\x1a\x00(\x002\x000\xf8\x03\x1a\x00\"\x00"
 [[hyperkube/v1.10.0+b81c8f8 (linux/amd64) kubernetes/b81c8f8] 172.18.8.18:52014]
I0622 17:41:35.269237       1 handler.go:159] kube-aggregator: GET "/api/v1/nodes/ip-172-18-13-29.ec2.internal" satisfied by nonGoRestful
I0622 17:41:35.269270       1 pathrecorder.go:247] kube-aggregator: "/api/v1/nodes/ip-172-18-13-29.ec2.internal" satisfied by prefix /api/
I0622 17:41:35.269289       1 handler.go:149] kube-apiserver: GET "/api/v1/nodes/ip-172-18-13-29.ec2.internal" satisfied by gorestful with webservice /api/v1
I0622 17:41:35.844442       1 rbac.go:118] RBAC DENY: user "system:anonymous" groups ["system:unauthenticated"] cannot "get" nonResourceURL "/healthz/ready" cluster-wide
I0622 17:41:35.844555       1 authorization.go:60] Forbidden: "/healthz/ready", Reason: "User \"system:anonymous\" cannot \"get\" on \"/healthz/ready\": [clusterrole.rbac.authorization.k8s.io \"system:discovery\" not found, clusterrole.rbac.authorization.k8s.io \"system:oauth-token-deleter\" not found, clusterrole.rbac.authorization.k8s.io \"system:discovery\" not found, clusterrole.rbac.authorization.k8s.io \"system:webhook\" not found, clusterrole.rbac.authorization.k8s.io \"system:basic-user\" not found, clusterrole.rbac.authorization.k8s.io \"cluster-status\" not found, clusterrole.rbac.authorization.k8s.io \"system:scope-impersonation\" not found, clusterrole.rbac.authorization.k8s.io \"self-access-reviewer\" not found]"
I0622 17:41:35.844654       1 wrap.go:42] GET /healthz/ready: (412.582µs) 403 [[kube-probe/1.10+] 172.18.8.18:52184]
I0622 17:41:35.909034       1 trace.go:76] Trace[1501879407]: "Patch /api/v1/nodes/ip-172-18-13-29.ec2.internal/status" (started: 2018-06-22 17:41:25.865717345 +0000 UTC m=+76.562399803) (total time: 10.043278754s):
Trace[1501879407]: [75.721µs] [75.721µs] Recorded the audit event
Trace[1501879407]: [301.515µs] [225.794µs] About to apply patch
Trace[1501879407]: [1.398615ms] [1.0971ms] About to check admission control
Trace[1501879407]: [10.043278754s] [10.041880139s] END
I0622 17:41:35.909116       1 wrap.go:42] PATCH /api/v1/nodes/ip-172-18-13-29.ec2.internal/status?timeout=10s: (10.043763946s) 504
goroutine 3201 [running]:
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc42761c8c0, 0x1f8)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc42761c8c0, 0x1f8)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc432288ca0, 0x1f8)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:188 +0xac
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics.(*ResponseWriterDelegator).WriteHeader(0xc42b994de0, 0x1f8)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:281 +0x45
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.SerializeObject(0x5aef814, 0x23, 0x7fde84c598a8, 0xc4323e9f00, 0xacc5e40, 0xc42ccb69a0, 0xc42ba6cc00, 0x1f8, 0xac9df80, 0xc422d5ed80)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:95 +0x8d
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.WriteObjectNegotiated(0x7fde80a1e320, 0xc431a7da10, 0xacce140, 0xc42926b5c0, 0x0, 0x0, 0x5a48e11, 0x2, 0xacc5e40, 0xc42ccb69a0, ...)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:123 +0x32b
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.ErrorNegotiated(0x7fde80a1e320, 0xc431a7da10, 0xac7e5c0, 0xc422d5eab0, 0xacce140, 0xc42926b5c0, 0x0, 0x0, 0x5a48e11, 0x2, ...)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:142 +0x163
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers.(*RequestScope).err(0xc423ccd540, 0xac7e5c0, 0xc422d5eab0, 0xacc5e40, 0xc42ccb69a0, 0xc42ba6cc00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:67 +0x10c
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers.PatchResource.func1(0xacc5e40, 0xc42ccb69a0, 0xc42ba6cc00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/patch.go:130 +0x1205
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints.restfulPatchResource.func1(0xc42b994d50, 0xc42cf05020)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1052 +0x12c
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc42b994d50, 0xc42cf05020)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:199 +0x208
github.com/openshift/origin/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc427078750, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/github.com/emicklei/go-restful/container.go:277 +0xb18
github.com/openshift/origin/vendor/github.com/emicklei/go-restful.(*Container).Dispatch(0xc427078750, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/github.com/emicklei/go-restful/container.go:199 +0x57
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5a68967, 0xe, 0xc427078750, 0xc4250a72d0, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:152 +0x4e0
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc4298228c0, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00)
	<autogenerated>:1 +0x75
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc4307716c0, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc42d4bb0a0, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5a6d554, 0xf, 0xc427078360, 0xc42d4bb0a0, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc4234bec80, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00)
	<autogenerated>:1 +0x75
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:52 +0x37d
net/http.HandlerFunc.ServeHTTP(0xc424618cd0, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:165 +0x42a
net/http.HandlerFunc.ServeHTTP(0xc42bcbc8c0, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a
net/http.HandlerFunc.ServeHTTP(0xc424618d70, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:78 +0x2b1
net/http.HandlerFunc.ServeHTTP(0xc424619090, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1(0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb
net/http.HandlerFunc.ServeHTTP(0xc4234bed60, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1(0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x17b
net/http.HandlerFunc.ServeHTTP(0xc42afdac00, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc4234bf1c0, 0xacd1e40, 0xc42ccb6990, 0xc42ba6cc00, 0xc429dc8f00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d
created by github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab

logging error output: "k8s\x00\n\f\n\x02v1\x12\x06Status\x12d\n\x06\n\x00\x12\x00\x1a\x00\x12\aFailure\x1a9Timeout: request did not complete within allowed duration\"\aTimeout*\n\n\x00\x12\x00\x1a\x00(\x002\x000\xf8\x03\x1a\x00\"\x00"
 [[hyperkube/v1.10.0+b81c8f8 (linux/amd64) kubernetes/b81c8f8] 172.18.13.29:48460]
E0622 17:41:36.747223       1 status.go:64] apiserver received an error that is not an metav1.Status: etcdserver: request timed out
I0622 17:41:36.747454       1 trace.go:76] Trace[1686606681]: "Create /api/v1/namespaces" (started: 2018-06-22 17:41:25.478377273 +0000 UTC m=+76.175059756) (total time: 11.269046609s):
Trace[1686606681]: [48.412µs] [48.412µs] About to convert to expected version
Trace[1686606681]: [90.243µs] [41.831µs] Conversion done
Trace[1686606681]: [325.525044ms] [325.434801ms] About to store object in database
Trace[1686606681]: [11.269046609s] [10.943521565s] END
I0622 17:41:36.747573       1 wrap.go:42] POST /api/v1/namespaces: (11.269491026s) 500
goroutine 3119 [running]:
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc4249349a0, 0x1f4)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc4249349a0, 0x1f4)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc42ba6bae0, 0x1f4)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:188 +0xac
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics.(*ResponseWriterDelegator).WriteHeader(0xc42bc7db90, 0x1f4)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:281 +0x45
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.SerializeObject(0x5aef814, 0x23, 0x7fde84c598a8, 0xc438874b80, 0xacc5e40, 0xc42ccb6538, 0xc4303dee00, 0x1f4, 0xac9df80, 0xc4259ad710)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:95 +0x8d
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.WriteObjectNegotiated(0x7fde80a1e320, 0xc42b0118c0, 0xacce140, 0xc42926b5c0, 0x0, 0x0, 0x5a48e11, 0x2, 0xacc5e40, 0xc42ccb6538, ...)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:123 +0x32b
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.ErrorNegotiated(0x7fde80a1e320, 0xc42b0118c0, 0xac896c0, 0xc4343c06a0, 0xacce140, 0xc42926b5c0, 0x0, 0x0, 0x5a48e11, 0x2, ...)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:142 +0x163
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers.(*RequestScope).err(0xc4258d2dc0, 0xac896c0, 0xc4343c06a0, 0xacc5e40, 0xc42ccb6538, 0xc4303dee00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:67 +0x10c
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers.createHandler.func1(0xacc5e40, 0xc42ccb6538, 0xc4303dee00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/create.go:120 +0x1409
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints.restfulCreateResource.func1(0xc42bc7d260, 0xc429cfcf60)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1028 +0xd5
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc42bc7d260, 0xc429cfcf60)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:199 +0x208
github.com/openshift/origin/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc427078750, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/github.com/emicklei/go-restful/container.go:277 +0xb18
github.com/openshift/origin/vendor/github.com/emicklei/go-restful.(*Container).Dispatch(0xc427078750, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/github.com/emicklei/go-restful/container.go:199 +0x57
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5a68967, 0xe, 0xc427078750, 0xc4250a72d0, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:152 +0x4e0
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc4298228c0, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00)
	<autogenerated>:1 +0x75
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc4307716c0, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc42d4bb0a0, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5a6d554, 0xf, 0xc427078360, 0xc42d4bb0a0, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc4234bec80, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00)
	<autogenerated>:1 +0x75
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7fde80381330, 0xc42ccb6528, 0xc4303dee00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:52 +0x37d
net/http.HandlerFunc.ServeHTTP(0xc424618cd0, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7fde80381330, 0xc42ccb6528, 0xc4303dee00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:165 +0x42a
net/http.HandlerFunc.ServeHTTP(0xc42bcbc8c0, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7fde80381330, 0xc42ccb6528, 0xc4303dee00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a
net/http.HandlerFunc.ServeHTTP(0xc424618d70, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7fde80381330, 0xc42ccb6528, 0xc4303dee00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:78 +0x2b1
net/http.HandlerFunc.ServeHTTP(0xc424619090, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1(0x7fde80381330, 0xc42ccb6528, 0xc4303dee00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb
net/http.HandlerFunc.ServeHTTP(0xc4234bed60, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1(0x7fde80381330, 0xc42ccb6528, 0xc4303dee00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x17b
net/http.HandlerFunc.ServeHTTP(0xc42afdac00, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc4234bf1c0, 0xacd1e40, 0xc42ccb6528, 0xc4303dee00, 0xc42be74720)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d
created by github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab

logging error output: "k8s\x00\n\f\n\x02v1\x12\x06Status\x125\n\x06\n\x00\x12\x00\x1a\x00\x12\aFailure\x1a\x1detcdserver: request timed out\"\x000\xf4\x03\x1a\x00\"\x00"
 [[openshift/v1.10.0+b81c8f8 (linux/amd64) kubernetes/b81c8f8] 127.0.0.1:35442]
E0622 17:41:36.779668       1 status.go:64] apiserver received an error that is not an metav1.Status: etcdserver: request timed out
I0622 17:41:36.779888       1 trace.go:76] Trace[1694635747]: "Create /apis/oauth.openshift.io/v1/oauthclients" (started: 2018-06-22 17:41:25.991556119 +0000 UTC m=+76.688238567) (total time: 10.788309441s):
Trace[1694635747]: [38.449µs] [38.449µs] About to convert to expected version
Trace[1694635747]: [474.007531ms] [473.969082ms] Conversion done
Trace[1694635747]: [474.376527ms] [368.996µs] About to store object in database
Trace[1694635747]: [10.788309441s] [10.313932914s] END
I0622 17:41:36.780034       1 wrap.go:42] POST /apis/oauth.openshift.io/v1/oauthclients: (10.788962126s) 500
goroutine 3220 [running]:
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc424ab4540, 0x1f4)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc424ab4540, 0x1f4)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc42a7de9a0, 0x1f4)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:188 +0xac
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics.(*ResponseWriterDelegator).WriteHeader(0xc431aeaba0, 0x1f4)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:281 +0x45
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.SerializeObject(0x5aef814, 0x23, 0x7fde84c598a8, 0xc4301b8680, 0xacc5e40, 0xc42ccb6b68, 0xc430d7a200, 0x1f4, 0xac9df80, 0xc42674fb00)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:95 +0x8d
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.WriteObjectNegotiated(0x7fde80a1e320, 0xc42f2f7ad0, 0xacce140, 0xc420e43440, 0x5a7d70e, 0x12, 0x5a48e11, 0x2, 0xacc5e40, 0xc42ccb6b68, ...)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:123 +0x32b
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.ErrorNegotiated(0x7fde80a1e320, 0xc42f2f7ad0, 0xac896c0, 0xc42b45d8a0, 0xacce140, 0xc420e43440, 0x5a7d70e, 0x12, 0x5a48e11, 0x2, ...)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:142 +0x163
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers.(*RequestScope).err(0xc4236f6140, 0xac896c0, 0xc42b45d8a0, 0xacc5e40, 0xc42ccb6b68, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:67 +0x10c
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers.createHandler.func1(0xacc5e40, 0xc42ccb6b68, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/create.go:120 +0x1409
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints.restfulCreateResource.func1(0xc431aeab10, 0xc42efa4360)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1028 +0xd5
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc431aeab10, 0xc42efa4360)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:199 +0x208
github.com/openshift/origin/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc421788090, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/github.com/emicklei/go-restful/container.go:277 +0xb18
github.com/openshift/origin/vendor/github.com/emicklei/go-restful.(*Container).Dispatch(0xc421788090, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/github.com/emicklei/go-restful/container.go:199 +0x57
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5abf1b5, 0x1c, 0xc421788090, 0xc4240935e0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:152 +0x4e0
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc420556660, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	<autogenerated>:1 +0x75
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc4233a3ac0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc423009650, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5acdf28, 0x1e, 0xc421f8c900, 0xc423009650, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc420dd84c0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	<autogenerated>:1 +0x75
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc424da7200, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc424cfcd20, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5abf685, 0x1c, 0xc421a5e900, 0xc424cfcd20, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc42380f040, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	<autogenerated>:1 +0x75
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc4207770c0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc424576a80, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5abf845, 0x1c, 0xc421f674d0, 0xc424576a80, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc424f90940, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	<autogenerated>:1 +0x75
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc42432f6c0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc4243a68c0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5ad5a5d, 0x1f, 0xc420816630, 0xc4243a68c0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc422932fe0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	<autogenerated>:1 +0x75
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc4216ca540, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc420e3cf50, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5ad5d07, 0x1f, 0xc425370ab0, 0xc420e3cf50, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc422295960, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	<autogenerated>:1 +0x75
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc425670040, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc422a9d0a0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5ab8e67, 0x1b, 0xc424fed5f0, 0xc422a9d0a0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc422a59d80, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	<autogenerated>:1 +0x75
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc42219de40, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc425e9d1f0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5a83171, 0x13, 0xc423ee83f0, 0xc425e9d1f0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc4244b99a0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	<autogenerated>:1 +0x75
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc42bcbc640, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc4250a72d0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5a68967, 0xe, 0xc427078750, 0xc4250a72d0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc4298228c0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	<autogenerated>:1 +0x75
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc4307716c0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc42d4bb0a0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5a6d554, 0xf, 0xc427078360, 0xc42d4bb0a0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc4234bec80, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	<autogenerated>:1 +0x75
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:52 +0x37d
net/http.HandlerFunc.ServeHTTP(0xc424618cd0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:165 +0x42a
net/http.HandlerFunc.ServeHTTP(0xc42bcbc8c0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a
net/http.HandlerFunc.ServeHTTP(0xc424618d70, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:78 +0x2b1
net/http.HandlerFunc.ServeHTTP(0xc424619090, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1(0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb
net/http.HandlerFunc.ServeHTTP(0xc4234bed60, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1(0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x17b
net/http.HandlerFunc.ServeHTTP(0xc42afdac00, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc4234bf1c0, 0xacd1e40, 0xc42ccb6b58, 0xc430d7a200, 0xc429dc98c0)
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d
created by github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
	/builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab

logging error output: "k8s\x00\n\f\n\x02v1\x12\x06Status\x125\n\x06\n\x00\x12\x00\x1a\x00\x12\aFailure\x1a\x1detcdserver: request timed out\"\x000\xf4\x03\x1a\x00\"\x00"
 [[openshift/v1.10.0+b81c8f8 (linux/amd64) kubernetes/b81c8f8] 172.18.8.18:52012]
E0622 17:41:36.780449       1 openshift_apiserver.go:725] error creating namespace "openshift-infra": etcdserver: request timed out
F0622 17:41:36.812114       1 hooks.go:188] PostStartHook "authorization.openshift.io-ensureopenshift-infra" failed: etcdserver: request timed out
--------------------------


And it seems that openshift-infra namespace is not being created:

E0622 17:41:36.780449       1 openshift_apiserver.go:725] error creating namespace "openshift-infra": etcdserver: request timed out

Comment 10 Avesh Agarwal 2018-06-22 18:39:39 UTC

In etcd logs, I am seeing following error:

embed: serving client requests on 172.18.8.18:2379
WARNING: 2018/06/22 17:43:20 Failed to dial 172.18.8.18:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.

Comment 11 zhou ying 2018-06-25 00:53:34 UTC

When I recover the master-config.yaml, the api and controller still not be running, so I don't think the root issue is related with the master-config.yaml.

Comment 12 zhou ying 2018-06-25 08:55:20 UTC

Do I need to preserve my env ? If not , I'll stop it ?

Comment 13 Seth Jennings 2018-06-25 15:58:12 UTC

zhou, keep it up for now.  i'll look at it today.  just got back from PTO.

Comment 16 Ryan Phillips 2018-06-25 19:40:09 UTC

According to the upstream etcd issue [1] we need to add ClientAuth to the allowed certificate usage field based on the fix for kubeadm [2] and comments on the upstream issue.

I will create a PR for the fix shortly.

[1] https://github.com/coreos/etcd/issues/8603#issuecomment-379224315
[2] https://github.com/kubernetes/kubernetes/pull/65020/files#diff-9887cabd66128ee027c291e406c04ddaR380

Comment 17 Ryan Phillips 2018-06-25 20:16:18 UTC

PR: https://github.com/openshift/openshift-ansible/pull/8968

Comment 18 Seth Jennings 2018-06-25 22:47:14 UTC

Ryan, any idea why are we just now seeing this issue?  Seems like this would effect all 3.10 installs at least.

Comment 19 Ryan Phillips 2018-06-26 03:30:12 UTC

@seth It does seem like it would be more widespread. I will look more into it tomorrow on why we might be seeing it now.

Comment 20 Ryan Phillips 2018-06-26 15:18:37 UTC

* OSE 3.11+ uses the quay Etcd 3.3 tag, the clientauth patch fixes OSE 3.11+
* OSE 3.10 is using Etcd 3.2.15, which needs a bump to at least 3.2.19 to pickup (https://github.com/coreos/etcd/blob/master/CHANGELOG-3.2.md#security-authentication-1). 3.2.22 is the latest so a bump to 3.2.22 would be beneficial for 3.10.

Comment 21 Ryan Phillips 2018-06-26 15:35:26 UTC

This issue is intermittent due to the timing of the TLS reload server side in etcd (could be on initial handshake or afterwards), and how the certificates are created (SNI and SAN fields).

More information: https://github.com/coreos/etcd/pull/9570#issue-181598004

Comment 35 zhou ying 2018-07-09 01:40:08 UTC

Ryan Phillips:
    I only reproduce this issue with the EC2.

Comment 37 Ryan Phillips 2018-07-09 18:12:24 UTC

After further debugging, I noticed the default m3.medium instances thrashing with high I/O and cpu pressure. I tweaked the instance type to m3.large via LAUNCHER_VARS=vm_type: m3.large and spun up another instance. This tweak fixed the issue on my test instance.

Another possibility is tweaking the liveness probe periodSeconds or failureThreshold. Tweaking these values is just a band-aid, because the instance is oversaturated.

Zhou: Can you try bumping the instance type to m3.large?

Comment 38 zhou ying 2018-07-10 01:45:41 UTC

Ryan Phillips:

    Thanks. I've tried with the m3.large ,can't reproduce it . I think we could verify this. 

openshift version
openshift v3.10.15

Comment 39 Ryan Phillips 2018-07-10 14:31:42 UTC

Looks like the changes for the test framework are in openshift-misc. Moving over to the test team.

Comment 40 zhou ying 2018-07-12 05:08:28 UTC

Confirmed with latest ocp and m3.large instance, the issue has fixed:
[root@ip-172-18-12-37 ~]#  openshift version
openshift v3.10.16

Comment 41 Thom Carlin 2018-07-13 12:50:13 UTC

Will this be backported?  I'm seeing it in OpenShift 3.9