Description of problem: add 'masterConfig.ImagePolicyConfig.ExternalRegistryHostname' for the master-config.yaml, the restart the api and controller service, the api pod will always recreate, the controllers pod will 'CrashLoopBackOff' Version-Release number of selected component (if applicable): v3.10.2 How reproducible: 30% Steps to Reproduce: 1. Update the master-config.yaml with : imagePolicyConfig: externalRegistryHostname: host of the registry's route 2. Try to restart the api and controller service 3. Even recover the master-config.yaml, the api and controller still not be running. Actual results: 2. The api pod will always recreate and the controller pod will 'CrashLoopBackOff' Expected results: 2. The api and controller pod restart succeed Additional info: docker version: docker-1.13.1-63.git94f4240.el7.x86_64 The api pod didn't have any error info, only could get some error info from the docker log: [root@ip-172-18-8-18 cron.hourly]# docker ps -a |grep master-api f2262dec097e 22afa152a447 "/bin/bash -c '#!/..." About a minute ago Exited (137) 25 seconds ago k8s_api_master-api-ip-172-18-8-18.ec2.internal_kube-system_f8ddfa563d27a7f9ca95644fe2f8cdd0_11 6be5580e806b registry.reg-aws.openshift.com:443/openshift3/ose-pod:v3.10.2 [root@ip-172-18-8-18 master]# journalctl -f -u docker |grep f2262dec097e44 Jun 21 05:02:17 ip-172-18-8-18.ec2.internal dockerd-current[4118]: time="2018-06-21T05:02:17.490491984-04:00" level=warning msg="Unknown healthcheck type 'NONE' (expected 'CMD') in container f2262dec097e445eadc788bb54600533744aeaf3e73a5b4583c549d059a7c240" Jun 21 05:03:47 ip-172-18-8-18.ec2.internal dockerd-current[4118]: time="2018-06-21T05:03:47.976889423-04:00" level=info msg="Container f2262dec097e445eadc788bb54600533744aeaf3e73a5b4583c549d059a7c240 failed to exit within 30 seconds of signal 15 - using the force" Jun 21 05:03:48 ip-172-18-8-18.ec2.internal dockerd-current[4118]: time="2018-06-21T05:03:48.209765256-04:00" level=error msg="containerd: deleting container" error="exit status 1: \"container f2262dec097e445eadc788bb54600533744aeaf3e73a5b4583c549d059a7c240 does not exist\\none or more of the container deletions failed\\n\"" Jun 21 05:03:48 ip-172-18-8-18.ec2.internal dockerd-current[4118]: time="2018-06-21T05:03:48.223903073-04:00" level=error msg="Error closing logger: close /var/lib/docker/containers/f2262dec097e445eadc788bb54600533744aeaf3e73a5b4583c549d059a7c240/f2262dec097e445eadc788bb54600533744aeaf3e73a5b4583c549d059a7c240-json.log: file already closed" Jun 21 05:03:48 ip-172-18-8-18.ec2.internal dockerd-current[4118]: time="2018-06-21T05:03:48.306809354-04:00" level=warning msg="f2262dec097e445eadc788bb54600533744aeaf3e73a5b4583c549d059a7c240 cleanup: failed to unmount secrets: invalid argument" Logs from the controller pod: E0621 09:10:42.742822 1 reflector.go:205] github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:125: Failed to list <nil>: Get https://ip-172-18-8-18.ec2.internal:8443/apis/automationbroker.io/v1alpha1/bundlebindings?limit=500&resourceVersion=0: dial tcp 172.18.8.18:8443: getsockopt: connection refused I0621 09:10:42.746636 1 reconciler.go:112] Skipping reconciling attached volumes still attached since it is disabled via the command line. I0621 09:10:42.846752 1 reconciler.go:112] Skipping reconciling attached volumes still attached since it is disabled via the command line. I0621 09:10:42.931206 1 reflector.go:240] Listing and watching <nil> from github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:125 I0621 09:10:42.933086 1 reflector.go:240] Listing and watching *v1beta1.VolumeAttachment from github.com/openshift/origin/vendor/k8s.io/client-go/informers/factory.go:87 E0621 09:10:42.933526 1 reflector.go:205] github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:125: Failed to list <nil>: Get https://ip-172-18-8-18.ec2.internal:8443/apis/servicecatalog.k8s.io/v1beta1/clusterserviceclasses?limit=500&resourceVersion=0: dial tcp 172.18.8.18:8443: getsockopt: connection refused E0621 09:10:42.933591 1 leaderelection.go:224] error retrieving resource lock kube-system/kube-controller-manager: Get https://ip-172-18-8-18.ec2.internal:8443/api/v1/namespaces/kube-system/configmaps/kube-controller-manager: dial tcp 172.18.8.18:8443: getsockopt: connection refused E0621 09:10:42.933622 1 event.go:260] Could not construct reference to: '&v1.ConfigMap{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Data:map[string]string(nil), BinaryData:map[string][]uint8(nil)}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'LeaderElection' 'ip-172-18-8-18.ec2.internal_8c58a1a1-7532-11e8-95da-0e77881bc7be stopped leading' I0621 09:10:42.933675 1 leaderelection.go:203] failed to renew lease kube-system/kube-controller-manager: timed out waiting for the condition F0621 09:10:42.933712 1 controllermanager.go:216] leaderelection lost
Strange, it does not seems this is kubelet deleting the container because it fails health checks (the healthz probe succeed after ~20s after start). The error in Docker logs suggests that the container did not died on its own (segfault or panic) and also there are no logs in api container that proving it. The Docker log seems to suggests that "something" killed that container. Moving to Pod team to investigate (but I haven't seen a kill due to liveness probe executed from kubelet).
I am looking into it.
Not related to this, but just curious but wondering why docker is setup with devicemapper when the default is overlay2?
(In reply to Avesh Agarwal from comment #4) > Not related to this, but just curious but wondering why docker is setup with > devicemapper when the default is overlay2? We are testing different martrix, uppper the priority, as this should be an release blocker
Zhou, Could you share what exact value you gave to ExternalRegistryHostname? I am having hard time reproducing it on my cluster.
(In reply to Avesh Agarwal from comment #6) > Zhou, > > Could you share what exact value you gave to ExternalRegistryHostname? I am > having hard time reproducing it on my cluster. Or if you could share your exact master-config.yaml that would be helpful too.
I looked into logs from one instance of exited api server pod, and I noticed following stack strace: docker logs 865bfa8feee1: -------------------------- Trace[1707801746]: [1.959416ms] [1.959416ms] Recorded the audit event Trace[1707801746]: [2.128801ms] [169.385µs] About to apply patch Trace[1707801746]: [735.820788ms] [733.691987ms] About to check admission control Trace[1707801746]: [10.251151633s] [9.515330845s] END I0622 17:41:35.243070 1 wrap.go:42] PATCH /api/v1/nodes/ip-172-18-8-18.ec2.internal/status?timeout=10s: (10.251825797s) 504 goroutine 2903 [running]: github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc4248112d0, 0x1f8) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc4248112d0, 0x1f8) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc43139d980, 0x1f8) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:188 +0xac github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics.(*ResponseWriterDelegator).WriteHeader(0xc430e8fa10, 0x1f8) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:281 +0x45 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.SerializeObject(0x5aef814, 0x23, 0x7fde84c598a8, 0xc4308d8a80, 0xacc5e40, 0xc433def4e0, 0xc430c60800, 0x1f8, 0xac9df80, 0xc437871e60) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:95 +0x8d github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.WriteObjectNegotiated(0x7fde80a1e320, 0xc431aaff20, 0xacce140, 0xc42926b5c0, 0x0, 0x0, 0x5a48e11, 0x2, 0xacc5e40, 0xc433def4e0, ...) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:123 +0x32b github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.ErrorNegotiated(0x7fde80a1e320, 0xc431aaff20, 0xac7e5c0, 0xc437871dd0, 0xacce140, 0xc42926b5c0, 0x0, 0x0, 0x5a48e11, 0x2, ...) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:142 +0x163 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers.(*RequestScope).err(0xc423bdf400, 0xac7e5c0, 0xc437871dd0, 0xacc5e40, 0xc433def4e0, 0xc430c60800) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:67 +0x10c github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers.PatchResource.func1(0xacc5e40, 0xc433def4e0, 0xc430c60800) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/patch.go:130 +0x1205 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints.restfulPatchResource.func1(0xc430e8f950, 0xc43285a840) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1052 +0x12c github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc430e8f950, 0xc43285a840) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:199 +0x208 github.com/openshift/origin/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc427078750, 0x7fde80381330, 0xc433def4d0, 0xc430c60800) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/github.com/emicklei/go-restful/container.go:277 +0xb18 github.com/openshift/origin/vendor/github.com/emicklei/go-restful.(*Container).Dispatch(0xc427078750, 0x7fde80381330, 0xc433def4d0, 0xc430c60800) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/github.com/emicklei/go-restful/container.go:199 +0x57 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5a68967, 0xe, 0xc427078750, 0xc4250a72d0, 0x7fde80381330, 0xc433def4d0, 0xc430c60800) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:152 +0x4e0 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc4298228c0, 0x7fde80381330, 0xc433def4d0, 0xc430c60800) <autogenerated>:1 +0x75 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc4307716c0, 0x7fde80381330, 0xc433def4d0, 0xc430c60800) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc42d4bb0a0, 0x7fde80381330, 0xc433def4d0, 0xc430c60800) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5a6d554, 0xf, 0xc427078360, 0xc42d4bb0a0, 0x7fde80381330, 0xc433def4d0, 0xc430c60800) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc4234bec80, 0x7fde80381330, 0xc433def4d0, 0xc430c60800) <autogenerated>:1 +0x75 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7fde80381330, 0xc433def4d0, 0xc430c60800) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:52 +0x37d net/http.HandlerFunc.ServeHTTP(0xc424618cd0, 0x7fde80381330, 0xc433def4d0, 0xc430c60800) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7fde80381330, 0xc433def4d0, 0xc430c60800) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:165 +0x42a net/http.HandlerFunc.ServeHTTP(0xc42bcbc8c0, 0x7fde80381330, 0xc433def4d0, 0xc430c60800) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7fde80381330, 0xc433def4d0, 0xc430c60800) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a net/http.HandlerFunc.ServeHTTP(0xc424618d70, 0x7fde80381330, 0xc433def4d0, 0xc430c60800) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7fde80381330, 0xc433def4d0, 0xc430c60800) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:78 +0x2b1 net/http.HandlerFunc.ServeHTTP(0xc424619090, 0x7fde80381330, 0xc433def4d0, 0xc430c60800) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1(0x7fde80381330, 0xc433def4d0, 0xc430c60800) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb net/http.HandlerFunc.ServeHTTP(0xc4234bed60, 0x7fde80381330, 0xc433def4d0, 0xc430c60800) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1(0x7fde80381330, 0xc433def4d0, 0xc430c60800) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x17b net/http.HandlerFunc.ServeHTTP(0xc42afdac00, 0x7fde80381330, 0xc433def4d0, 0xc430c60800) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc4234bf1c0, 0xacd1e40, 0xc433def4d0, 0xc430c60800, 0xc42a8862a0) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d created by github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab logging error output: "k8s\x00\n\f\n\x02v1\x12\x06Status\x12d\n\x06\n\x00\x12\x00\x1a\x00\x12\aFailure\x1a9Timeout: request did not complete within allowed duration\"\aTimeout*\n\n\x00\x12\x00\x1a\x00(\x002\x000\xf8\x03\x1a\x00\"\x00" [[hyperkube/v1.10.0+b81c8f8 (linux/amd64) kubernetes/b81c8f8] 172.18.8.18:52014] I0622 17:41:35.269237 1 handler.go:159] kube-aggregator: GET "/api/v1/nodes/ip-172-18-13-29.ec2.internal" satisfied by nonGoRestful I0622 17:41:35.269270 1 pathrecorder.go:247] kube-aggregator: "/api/v1/nodes/ip-172-18-13-29.ec2.internal" satisfied by prefix /api/ I0622 17:41:35.269289 1 handler.go:149] kube-apiserver: GET "/api/v1/nodes/ip-172-18-13-29.ec2.internal" satisfied by gorestful with webservice /api/v1 I0622 17:41:35.844442 1 rbac.go:118] RBAC DENY: user "system:anonymous" groups ["system:unauthenticated"] cannot "get" nonResourceURL "/healthz/ready" cluster-wide I0622 17:41:35.844555 1 authorization.go:60] Forbidden: "/healthz/ready", Reason: "User \"system:anonymous\" cannot \"get\" on \"/healthz/ready\": [clusterrole.rbac.authorization.k8s.io \"system:discovery\" not found, clusterrole.rbac.authorization.k8s.io \"system:oauth-token-deleter\" not found, clusterrole.rbac.authorization.k8s.io \"system:discovery\" not found, clusterrole.rbac.authorization.k8s.io \"system:webhook\" not found, clusterrole.rbac.authorization.k8s.io \"system:basic-user\" not found, clusterrole.rbac.authorization.k8s.io \"cluster-status\" not found, clusterrole.rbac.authorization.k8s.io \"system:scope-impersonation\" not found, clusterrole.rbac.authorization.k8s.io \"self-access-reviewer\" not found]" I0622 17:41:35.844654 1 wrap.go:42] GET /healthz/ready: (412.582µs) 403 [[kube-probe/1.10+] 172.18.8.18:52184] I0622 17:41:35.909034 1 trace.go:76] Trace[1501879407]: "Patch /api/v1/nodes/ip-172-18-13-29.ec2.internal/status" (started: 2018-06-22 17:41:25.865717345 +0000 UTC m=+76.562399803) (total time: 10.043278754s): Trace[1501879407]: [75.721µs] [75.721µs] Recorded the audit event Trace[1501879407]: [301.515µs] [225.794µs] About to apply patch Trace[1501879407]: [1.398615ms] [1.0971ms] About to check admission control Trace[1501879407]: [10.043278754s] [10.041880139s] END I0622 17:41:35.909116 1 wrap.go:42] PATCH /api/v1/nodes/ip-172-18-13-29.ec2.internal/status?timeout=10s: (10.043763946s) 504 goroutine 3201 [running]: github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc42761c8c0, 0x1f8) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc42761c8c0, 0x1f8) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc432288ca0, 0x1f8) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:188 +0xac github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics.(*ResponseWriterDelegator).WriteHeader(0xc42b994de0, 0x1f8) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:281 +0x45 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.SerializeObject(0x5aef814, 0x23, 0x7fde84c598a8, 0xc4323e9f00, 0xacc5e40, 0xc42ccb69a0, 0xc42ba6cc00, 0x1f8, 0xac9df80, 0xc422d5ed80) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:95 +0x8d github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.WriteObjectNegotiated(0x7fde80a1e320, 0xc431a7da10, 0xacce140, 0xc42926b5c0, 0x0, 0x0, 0x5a48e11, 0x2, 0xacc5e40, 0xc42ccb69a0, ...) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:123 +0x32b github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.ErrorNegotiated(0x7fde80a1e320, 0xc431a7da10, 0xac7e5c0, 0xc422d5eab0, 0xacce140, 0xc42926b5c0, 0x0, 0x0, 0x5a48e11, 0x2, ...) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:142 +0x163 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers.(*RequestScope).err(0xc423ccd540, 0xac7e5c0, 0xc422d5eab0, 0xacc5e40, 0xc42ccb69a0, 0xc42ba6cc00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:67 +0x10c github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers.PatchResource.func1(0xacc5e40, 0xc42ccb69a0, 0xc42ba6cc00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/patch.go:130 +0x1205 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints.restfulPatchResource.func1(0xc42b994d50, 0xc42cf05020) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1052 +0x12c github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc42b994d50, 0xc42cf05020) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:199 +0x208 github.com/openshift/origin/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc427078750, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/github.com/emicklei/go-restful/container.go:277 +0xb18 github.com/openshift/origin/vendor/github.com/emicklei/go-restful.(*Container).Dispatch(0xc427078750, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/github.com/emicklei/go-restful/container.go:199 +0x57 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5a68967, 0xe, 0xc427078750, 0xc4250a72d0, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:152 +0x4e0 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc4298228c0, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00) <autogenerated>:1 +0x75 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc4307716c0, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc42d4bb0a0, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5a6d554, 0xf, 0xc427078360, 0xc42d4bb0a0, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc4234bec80, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00) <autogenerated>:1 +0x75 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:52 +0x37d net/http.HandlerFunc.ServeHTTP(0xc424618cd0, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:165 +0x42a net/http.HandlerFunc.ServeHTTP(0xc42bcbc8c0, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a net/http.HandlerFunc.ServeHTTP(0xc424618d70, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:78 +0x2b1 net/http.HandlerFunc.ServeHTTP(0xc424619090, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1(0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb net/http.HandlerFunc.ServeHTTP(0xc4234bed60, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1(0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x17b net/http.HandlerFunc.ServeHTTP(0xc42afdac00, 0x7fde80381330, 0xc42ccb6990, 0xc42ba6cc00) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc4234bf1c0, 0xacd1e40, 0xc42ccb6990, 0xc42ba6cc00, 0xc429dc8f00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d created by github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab logging error output: "k8s\x00\n\f\n\x02v1\x12\x06Status\x12d\n\x06\n\x00\x12\x00\x1a\x00\x12\aFailure\x1a9Timeout: request did not complete within allowed duration\"\aTimeout*\n\n\x00\x12\x00\x1a\x00(\x002\x000\xf8\x03\x1a\x00\"\x00" [[hyperkube/v1.10.0+b81c8f8 (linux/amd64) kubernetes/b81c8f8] 172.18.13.29:48460] E0622 17:41:36.747223 1 status.go:64] apiserver received an error that is not an metav1.Status: etcdserver: request timed out I0622 17:41:36.747454 1 trace.go:76] Trace[1686606681]: "Create /api/v1/namespaces" (started: 2018-06-22 17:41:25.478377273 +0000 UTC m=+76.175059756) (total time: 11.269046609s): Trace[1686606681]: [48.412µs] [48.412µs] About to convert to expected version Trace[1686606681]: [90.243µs] [41.831µs] Conversion done Trace[1686606681]: [325.525044ms] [325.434801ms] About to store object in database Trace[1686606681]: [11.269046609s] [10.943521565s] END I0622 17:41:36.747573 1 wrap.go:42] POST /api/v1/namespaces: (11.269491026s) 500 goroutine 3119 [running]: github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc4249349a0, 0x1f4) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc4249349a0, 0x1f4) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc42ba6bae0, 0x1f4) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:188 +0xac github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics.(*ResponseWriterDelegator).WriteHeader(0xc42bc7db90, 0x1f4) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:281 +0x45 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.SerializeObject(0x5aef814, 0x23, 0x7fde84c598a8, 0xc438874b80, 0xacc5e40, 0xc42ccb6538, 0xc4303dee00, 0x1f4, 0xac9df80, 0xc4259ad710) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:95 +0x8d github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.WriteObjectNegotiated(0x7fde80a1e320, 0xc42b0118c0, 0xacce140, 0xc42926b5c0, 0x0, 0x0, 0x5a48e11, 0x2, 0xacc5e40, 0xc42ccb6538, ...) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:123 +0x32b github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.ErrorNegotiated(0x7fde80a1e320, 0xc42b0118c0, 0xac896c0, 0xc4343c06a0, 0xacce140, 0xc42926b5c0, 0x0, 0x0, 0x5a48e11, 0x2, ...) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:142 +0x163 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers.(*RequestScope).err(0xc4258d2dc0, 0xac896c0, 0xc4343c06a0, 0xacc5e40, 0xc42ccb6538, 0xc4303dee00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:67 +0x10c github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers.createHandler.func1(0xacc5e40, 0xc42ccb6538, 0xc4303dee00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/create.go:120 +0x1409 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints.restfulCreateResource.func1(0xc42bc7d260, 0xc429cfcf60) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1028 +0xd5 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc42bc7d260, 0xc429cfcf60) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:199 +0x208 github.com/openshift/origin/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc427078750, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/github.com/emicklei/go-restful/container.go:277 +0xb18 github.com/openshift/origin/vendor/github.com/emicklei/go-restful.(*Container).Dispatch(0xc427078750, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/github.com/emicklei/go-restful/container.go:199 +0x57 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5a68967, 0xe, 0xc427078750, 0xc4250a72d0, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:152 +0x4e0 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc4298228c0, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00) <autogenerated>:1 +0x75 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc4307716c0, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc42d4bb0a0, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5a6d554, 0xf, 0xc427078360, 0xc42d4bb0a0, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc4234bec80, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00) <autogenerated>:1 +0x75 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7fde80381330, 0xc42ccb6528, 0xc4303dee00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:52 +0x37d net/http.HandlerFunc.ServeHTTP(0xc424618cd0, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7fde80381330, 0xc42ccb6528, 0xc4303dee00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:165 +0x42a net/http.HandlerFunc.ServeHTTP(0xc42bcbc8c0, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7fde80381330, 0xc42ccb6528, 0xc4303dee00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a net/http.HandlerFunc.ServeHTTP(0xc424618d70, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7fde80381330, 0xc42ccb6528, 0xc4303dee00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:78 +0x2b1 net/http.HandlerFunc.ServeHTTP(0xc424619090, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1(0x7fde80381330, 0xc42ccb6528, 0xc4303dee00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb net/http.HandlerFunc.ServeHTTP(0xc4234bed60, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1(0x7fde80381330, 0xc42ccb6528, 0xc4303dee00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x17b net/http.HandlerFunc.ServeHTTP(0xc42afdac00, 0x7fde80381330, 0xc42ccb6528, 0xc4303dee00) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc4234bf1c0, 0xacd1e40, 0xc42ccb6528, 0xc4303dee00, 0xc42be74720) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d created by github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab logging error output: "k8s\x00\n\f\n\x02v1\x12\x06Status\x125\n\x06\n\x00\x12\x00\x1a\x00\x12\aFailure\x1a\x1detcdserver: request timed out\"\x000\xf4\x03\x1a\x00\"\x00" [[openshift/v1.10.0+b81c8f8 (linux/amd64) kubernetes/b81c8f8] 127.0.0.1:35442] E0622 17:41:36.779668 1 status.go:64] apiserver received an error that is not an metav1.Status: etcdserver: request timed out I0622 17:41:36.779888 1 trace.go:76] Trace[1694635747]: "Create /apis/oauth.openshift.io/v1/oauthclients" (started: 2018-06-22 17:41:25.991556119 +0000 UTC m=+76.688238567) (total time: 10.788309441s): Trace[1694635747]: [38.449µs] [38.449µs] About to convert to expected version Trace[1694635747]: [474.007531ms] [473.969082ms] Conversion done Trace[1694635747]: [474.376527ms] [368.996µs] About to store object in database Trace[1694635747]: [10.788309441s] [10.313932914s] END I0622 17:41:36.780034 1 wrap.go:42] POST /apis/oauth.openshift.io/v1/oauthclients: (10.788962126s) 500 goroutine 3220 [running]: github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc424ab4540, 0x1f4) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc424ab4540, 0x1f4) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc42a7de9a0, 0x1f4) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:188 +0xac github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics.(*ResponseWriterDelegator).WriteHeader(0xc431aeaba0, 0x1f4) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:281 +0x45 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.SerializeObject(0x5aef814, 0x23, 0x7fde84c598a8, 0xc4301b8680, 0xacc5e40, 0xc42ccb6b68, 0xc430d7a200, 0x1f4, 0xac9df80, 0xc42674fb00) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:95 +0x8d github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.WriteObjectNegotiated(0x7fde80a1e320, 0xc42f2f7ad0, 0xacce140, 0xc420e43440, 0x5a7d70e, 0x12, 0x5a48e11, 0x2, 0xacc5e40, 0xc42ccb6b68, ...) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:123 +0x32b github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.ErrorNegotiated(0x7fde80a1e320, 0xc42f2f7ad0, 0xac896c0, 0xc42b45d8a0, 0xacce140, 0xc420e43440, 0x5a7d70e, 0x12, 0x5a48e11, 0x2, ...) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:142 +0x163 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers.(*RequestScope).err(0xc4236f6140, 0xac896c0, 0xc42b45d8a0, 0xacc5e40, 0xc42ccb6b68, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:67 +0x10c github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers.createHandler.func1(0xacc5e40, 0xc42ccb6b68, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/create.go:120 +0x1409 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints.restfulCreateResource.func1(0xc431aeab10, 0xc42efa4360) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1028 +0xd5 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc431aeab10, 0xc42efa4360) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:199 +0x208 github.com/openshift/origin/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc421788090, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/github.com/emicklei/go-restful/container.go:277 +0xb18 github.com/openshift/origin/vendor/github.com/emicklei/go-restful.(*Container).Dispatch(0xc421788090, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/github.com/emicklei/go-restful/container.go:199 +0x57 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5abf1b5, 0x1c, 0xc421788090, 0xc4240935e0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:152 +0x4e0 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc420556660, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) <autogenerated>:1 +0x75 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc4233a3ac0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc423009650, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5acdf28, 0x1e, 0xc421f8c900, 0xc423009650, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc420dd84c0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) <autogenerated>:1 +0x75 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc424da7200, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc424cfcd20, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5abf685, 0x1c, 0xc421a5e900, 0xc424cfcd20, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc42380f040, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) <autogenerated>:1 +0x75 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc4207770c0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc424576a80, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5abf845, 0x1c, 0xc421f674d0, 0xc424576a80, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc424f90940, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) <autogenerated>:1 +0x75 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc42432f6c0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc4243a68c0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5ad5a5d, 0x1f, 0xc420816630, 0xc4243a68c0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc422932fe0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) <autogenerated>:1 +0x75 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc4216ca540, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc420e3cf50, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5ad5d07, 0x1f, 0xc425370ab0, 0xc420e3cf50, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc422295960, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) <autogenerated>:1 +0x75 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc425670040, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc422a9d0a0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5ab8e67, 0x1b, 0xc424fed5f0, 0xc422a9d0a0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc422a59d80, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) <autogenerated>:1 +0x75 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc42219de40, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc425e9d1f0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5a83171, 0x13, 0xc423ee83f0, 0xc425e9d1f0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc4244b99a0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) <autogenerated>:1 +0x75 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc42bcbc640, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc4250a72d0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5a68967, 0xe, 0xc427078750, 0xc4250a72d0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc4298228c0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) <autogenerated>:1 +0x75 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc4307716c0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc42d4bb0a0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5a6d554, 0xf, 0xc427078360, 0xc42d4bb0a0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc4234bec80, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) <autogenerated>:1 +0x75 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:52 +0x37d net/http.HandlerFunc.ServeHTTP(0xc424618cd0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:165 +0x42a net/http.HandlerFunc.ServeHTTP(0xc42bcbc8c0, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a net/http.HandlerFunc.ServeHTTP(0xc424618d70, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:78 +0x2b1 net/http.HandlerFunc.ServeHTTP(0xc424619090, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1(0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb net/http.HandlerFunc.ServeHTTP(0xc4234bed60, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1(0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x17b net/http.HandlerFunc.ServeHTTP(0xc42afdac00, 0x7fde80381330, 0xc42ccb6b58, 0xc430d7a200) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc4234bf1c0, 0xacd1e40, 0xc42ccb6b58, 0xc430d7a200, 0xc429dc98c0) /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d created by github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP /builddir/build/BUILD/atomic-openshift-git-0.c418a80/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab logging error output: "k8s\x00\n\f\n\x02v1\x12\x06Status\x125\n\x06\n\x00\x12\x00\x1a\x00\x12\aFailure\x1a\x1detcdserver: request timed out\"\x000\xf4\x03\x1a\x00\"\x00" [[openshift/v1.10.0+b81c8f8 (linux/amd64) kubernetes/b81c8f8] 172.18.8.18:52012] E0622 17:41:36.780449 1 openshift_apiserver.go:725] error creating namespace "openshift-infra": etcdserver: request timed out F0622 17:41:36.812114 1 hooks.go:188] PostStartHook "authorization.openshift.io-ensureopenshift-infra" failed: etcdserver: request timed out -------------------------- And it seems that openshift-infra namespace is not being created: E0622 17:41:36.780449 1 openshift_apiserver.go:725] error creating namespace "openshift-infra": etcdserver: request timed out
In etcd logs, I am seeing following error: embed: serving client requests on 172.18.8.18:2379 WARNING: 2018/06/22 17:43:20 Failed to dial 172.18.8.18:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.
When I recover the master-config.yaml, the api and controller still not be running, so I don't think the root issue is related with the master-config.yaml.
Do I need to preserve my env ? If not , I'll stop it ?
zhou, keep it up for now. i'll look at it today. just got back from PTO.
According to the upstream etcd issue [1] we need to add ClientAuth to the allowed certificate usage field based on the fix for kubeadm [2] and comments on the upstream issue. I will create a PR for the fix shortly. [1] https://github.com/coreos/etcd/issues/8603#issuecomment-379224315 [2] https://github.com/kubernetes/kubernetes/pull/65020/files#diff-9887cabd66128ee027c291e406c04ddaR380
PR: https://github.com/openshift/openshift-ansible/pull/8968
Ryan, any idea why are we just now seeing this issue? Seems like this would effect all 3.10 installs at least.
@seth It does seem like it would be more widespread. I will look more into it tomorrow on why we might be seeing it now.
* OSE 3.11+ uses the quay Etcd 3.3 tag, the clientauth patch fixes OSE 3.11+ * OSE 3.10 is using Etcd 3.2.15, which needs a bump to at least 3.2.19 to pickup (https://github.com/coreos/etcd/blob/master/CHANGELOG-3.2.md#security-authentication-1). 3.2.22 is the latest so a bump to 3.2.22 would be beneficial for 3.10.
This issue is intermittent due to the timing of the TLS reload server side in etcd (could be on initial handshake or afterwards), and how the certificates are created (SNI and SAN fields). More information: https://github.com/coreos/etcd/pull/9570#issue-181598004
Ryan Phillips: I only reproduce this issue with the EC2.
After further debugging, I noticed the default m3.medium instances thrashing with high I/O and cpu pressure. I tweaked the instance type to m3.large via LAUNCHER_VARS=vm_type: m3.large and spun up another instance. This tweak fixed the issue on my test instance. Another possibility is tweaking the liveness probe periodSeconds or failureThreshold. Tweaking these values is just a band-aid, because the instance is oversaturated. Zhou: Can you try bumping the instance type to m3.large?
Ryan Phillips: Thanks. I've tried with the m3.large ,can't reproduce it . I think we could verify this. openshift version openshift v3.10.15
Looks like the changes for the test framework are in openshift-misc. Moving over to the test team.
Confirmed with latest ocp and m3.large instance, the issue has fixed: [root@ip-172-18-12-37 ~]# openshift version openshift v3.10.16
Will this be backported? I'm seeing it in OpenShift 3.9