Bug 1220113
| Summary: | vdsm NUMA code not effective, slowing down statistics retrieval | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Virtualization Manager | Reporter: | rhev-integ | ||||||
| Component: | vdsm | Assignee: | Martin Sivák <msivak> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Artyom <alukiano> | ||||||
| Severity: | urgent | Docs Contact: | |||||||
| Priority: | urgent | ||||||||
| Version: | 3.5.0 | CC: | alukiano, bazulay, dfediuck, ecohen, fromani, gklein, istein, lpeer, lsurette, mkalinin, msivak, rhodain, sherold, yeylon | ||||||
| Target Milestone: | --- | Keywords: | Triaged, ZStream | ||||||
| Target Release: | 3.5.3 | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | sla | ||||||||
| Fixed In Version: | vdsm-4.16.18-1.el6ev | Doc Type: | Bug Fix | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | 1182094 | Environment: | |||||||
| Last Closed: | 2015-06-15 13:19:14 UTC | Type: | --- | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | SLA | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | 1182094 | ||||||||
| Bug Blocks: | |||||||||
| Attachments: |
|
||||||||
|
Comment 2
Artyom
2015-05-14 08:36:37 UTC
Yep, you need couple of VMs with enabled NUMA. Checked on vdsm-4.16.16-1.el7ev.x86_64 Have four vms, with exactly the same amount of memory and cpu's, but two of them have 2 numa nodes with numa pinning(test_numa, test_numa_1) Run two of vms without NUMA: [root@alma05 ~]# time for x in $(seq 100); do vdsClient -s 0 getAllVmStats >/dev/null; done real 0m14.522s user 0m11.366s sys 0m2.347s [root@alma05 ~]# vdsClient -s 0 list table da8a3132-d242-43ae-af9e-03a5507a0591 4581 test_1 Up 0630332a-c9f9-4cd4-b9b4-f9a215eead08 4494 test Up Run two vms with NUMA: [root@alma05 ~]# time for x in $(seq 100); do vdsClient -s 0 getAllVmStats >/dev/null; done real 0m18.588s user 0m11.521s sys 0m2.415s [root@alma05 ~]# vdsClient -s 0 list table eec8bbc0-ded5-4cce-89ef-9aac41cddd41 4975 test_numa_1 Up 01273fa1-0900-4bf0-a932-16772d70a463 4892 test_numa Up Created attachment 1026433 [details]
vms configuration in REST
Created attachment 1026434 [details]
vdsm log
Please use the same VMs WITH NUMA before/after the update is applied for the time measurement comparison. You can't compare apples and oranges.. (I hope only two VMs were running during both measurements). Also, did you restart VDSM after upgrading it? Are you sure that the /etc/vdsm/mom.d/04-cputune.policy is equal to https://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=vdsm/mom.d/04-cputune.policy;h=92e331f3b7a959bd5a709ebc4167e03051c0c6df;hb=cf7248fdfe93aa08d686a87435d9c9a826e4ce28 I double checked it, on same two vms(one time run with NUMA, and another without NUMA), also I restarted vdsm and verified mom policy file, it no difference, results still the same: Without NUMA: [root@alma05 ~]# time for x in $(seq 100); do vdsClient -s 0 getAllVmStats >/dev/null; done real 0m14.373s user 0m11.372s sys 0m2.151s With NUMA: [root@alma05 ~]# time for x in $(seq 100); do vdsClient -s 0 getAllVmStats >/dev/null; done real 0m18.176s user 0m11.439s sys 0m2.217s Verified on vdsm-4.16.18-1.el7ev.x86_64 With NUMA: [root@alma05 ~]# time for x in $(seq 100); do vdsClient -s 0 getAllVmStats >/dev/null; done real 0m14.569s user 0m11.440s sys 0m2.303s Without NUMA: [root@alma05 ~]# time for x in $(seq 100); do vdsClient -s 0 getAllVmStats >/dev/null; done real 0m14.518s user 0m11.351s sys 0m2.357s Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-1103.html |