Bug 1845558
| Summary: | CNV HyperConverged nmstate-handler-worker high cpu | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Container Native Virtualization (CNV) | Reporter: | Jason Huddleston <jhuddles> | ||||
| Component: | Networking | Assignee: | Petr Horáček <phoracek> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Meni Yakove <myakove> | ||||
| Severity: | low | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 2.4.0 | CC: | cnv-qe-bugs, danken, ellorent, ncredi | ||||
| Target Milestone: | --- | Keywords: | Performance | ||||
| Target Release: | 2.5.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2020-09-29 12:59:53 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1906151 | ||||||
| Attachments: |
|
||||||
|
Description
Jason Huddleston
2020-06-09 14:06:00 UTC
Created attachment 1696766 [details]
Console screenshot
Screenshot showing CPU Usage.
in cnv.engineering (cnv-2.3) I see nmstate-handler taking ~0.5 per master, and nmstate-handler-worker taking 0.3-0.5 core per worker. However a closer look suggests a bigger etcdserver error.
An nmstate-handler pod is busy tight-looping with
{"level":"error","ts":1592671549.9629877,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"nodenetworkstate-controller","request":"/cnv-x4lcc-master-2","error":"etcdserver: request timed out","stacktrace":"github.com/nmstate/kubernetes-nmstate/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/nmstate/kubernetes-nmstate/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/nmstate/kubernetes-nmstate/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/nmstate/kubernetes-nmstate/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:218\ngithub.com/nmstate/kubernetes-nmstate/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/nmstate/kubernetes-nmstate/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:192\ngithub.com/nmstate/kubernetes-nmstate/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/nmstate/kubernetes-nmstate/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:171\ngithub.com/nmstate/kubernetes-nmstate/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/nmstate/kubernetes-nmstate/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\ngithub.com/nmstate/kubernetes-nmstate/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/nmstate/kubernetes-nmstate/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\ngithub.com/nmstate/kubernetes-nmstate/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/nmstate/kubernetes-nmstate/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
And ab nmstate-handler-work pod is in a similar condition.
{"level":"error","ts":1592671713.9723308,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"nodenetworkstate-controller","request":"/cnv-x4lcc-worker-0-gndlf","error":"etcdserver: request timed out","stacktrace":"github.com/nmstate/kubernetes-nmstate/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/nmstate/kubernetes-nmstate/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/nmstate/kubernetes-nmstate/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/nmstate/kubernetes-nmstate/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:218\ngithub.com/nmstate/kubernetes-nmstate/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/nmstate/kubernetes-nmstate/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:192\ngithub.com/nmstate/kubernetes-nmstate/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/nmstate/kubernetes-nmstate/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:171\ngithub.com/nmstate/kubernetes-nmstate/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/nmstate/kubernetes-nmstate/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\ngithub.com/nmstate/kubernetes-nmstate/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/nmstate/kubernetes-nmstate/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\ngithub.com/nmstate/kubernetes-nmstate/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/nmstate/kubernetes-nmstate/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
I have start to verify this at github.com/nmstate/kubevirt kubevirtci deploy and I see that every often handler goes to 45% of 6 cores with means it's around 3 cores althugh this cores are vcpus so I don't know if this correlates to the issue here. Tested with nmstate 0.3 and nmstate 0.4 both uses around 40% of CPUs everytime nmstatectl show is called. I have being playing around with nmstatectl varlink implementation at nmstate 0.4 and since nmstate is up and running as daemon using it the consumption drops to 6%. Jason reported that new releases are better we can close it, in case it reappear we can open it again. Yes, you can close this BZ. |