Bug 1011786

2.3.1 Option: Available CPUs

[[ Some background info follows, but you should probably rewrite it if you decide to use any of it. ]]

CPUs are overcommitted if the sum of VCPUs for all guests on the system is greater than the number of native CPUs on the system.  You can overcommit CPUs with one or multiple guests, if the total number of VCPUs is greater than the number of host CPUs.

If guests will not be simultaneously busy, it might be fine to overcommit resources, knowing that not all CPUs will be busy at the same time.  If there will be heavy or unpredicatable load on the guests, overcommitting resources can lead to poor performance.

If hyper-threads are active on the host, we should distinguish between logical CPUs and physical CPUs, since they share hardware resources.  Though a hyper-thread (aka logical) CPU is counted as a real CPU, there is actually little hardware to back them up.  Depending on the application, the best performance might be achieved by limiting the number of VCPUs to the number of real, physical CPUs on the host.

If possible, best performance on NUMA systems can be achieved by limiting guest size to the amount of resources on a single NUMA node.   If small to medium guests are being used, try to size them so they pack evenly into NUMA nodes, and avoid necessarily splitting across NUMA nodes.


2.3.4. Option: CPU Pinning:

[[ It is not clear where a NUMA mask info would be used here. ]]


7.4: libvirt NUMA Tuning

[[ will want to use "# numastat -cm qemu"  -- still need to provide output ]]


7.5: in the Important box:

[[ "When KSM is used on a NUMA host..." should be clarified to be "When KSM is merging across nodes on a NUMA host..."  This change will make it more obviously an eplanation of the impact of the option in the previous paragraph.  ]]

Numastat showing some virtual guests with mediocre memory alignment:

numastat -c qemu-kvm

Per-node process memory usage (in MBs)
PID              Node 0 Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Node 7 Total
---------------  ------ ------ ------ ------ ------ ------ ------ ------ -----
51722 (qemu-kvm)     68     16    357   6936      2      3    147    598  8128
51747 (qemu-kvm)    245     11      5     18   5172   2532      1     92  8076
53736 (qemu-kvm)     62    432   1661    506   4851    136     22    445  8116
53773 (qemu-kvm)   1393      3      1      2     12      0      0   6702  8114
---------------  ------ ------ ------ ------ ------ ------ ------ ------ -----
Total              1769    463   2024   7462  10037   2672    169   7837 32434


and after running numad to align the guests' CPUs and memory resources:

# numastat -c qemu-kvm

Per-node process memory usage (in MBs)
PID              Node 0 Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Node 7 Total
---------------  ------ ------ ------ ------ ------ ------ ------ ------ -----
51747 (qemu-kvm)      0      0      7      0   8072      0      1      0  8080
53736 (qemu-kvm)      0      0      7      0      0      0   8113      0  8120
53773 (qemu-kvm)      0      0      7      0      0      0      1   8110  8118
59065 (qemu-kvm)      0      0   8050      0      0      0      0      0  8051
---------------  ------ ------ ------ ------ ------ ------ ------ ------ -----
Total                 0      0   8072      0   8072      0   8114   8110 32368


The "-c" option just makes the display somewhat more compact.  See numastat manpage.  The "-m" option would just add system-wide memory information displayed on a per-node basis, for example:

# numastat -cm 

Per-node system memory usage (in MBs):
                 Node 0 Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Node 7   Total
                 ------ ------ ------ ------ ------ ------ ------ ------ -------
MemTotal         131061 131072 131072 131072 131072 131072 131072 131072 1048565
MemFree          126550 128178 128244 127849 128275 128209 128273 127545 1023123
MemUsed            4511   2894   2828   3223   2797   2863   2799   3527   25442
Active             1305     30     29    298      1     53      8    544    2269
Inactive              8      5      1     68      8      7      4     60     160
Active(anon)          3      7     19      9      0     21      4     15      79
Inactive(anon)        0      0      0      0      8      0      0      8      16
Active(file)       1302     23     10    288      1     32      5    529    2191
Inactive(file)        8      5      1     67      0      7      4     52     144
Unevictable           0      0      0      0      0      0      0      0       0
Mlocked               0      0      0      0      0      0      0      0       0
Dirty                 0      0      0      0      0      0      0      0       0
Writeback             0      0      0      0      0      0      0      0       0
FilePages          1311     28     11    356     10     39      9    588    2352
Mapped                0      0      4      0      8      1      3     16      32
AnonPages             3      7     19      9      0     21      3     15      78
Shmem                 0      0      0      0      8      0      0      8      17
KernelStack           5      1      1      1      1      1      1      1      10
PageTables            0      0      1      1      0      1      1      1       5
NFS_Unstable          0      0      0      0      0      0      0      0       0
Bounce                0      0      0      0      0      0      0      0       0
WritebackTmp          0      0      0      0      0      0      0      0       0
Slab                 82     78     23     76     22     34     19    127     460
SReclaimable         47     56      3     48      5      9      3     96     267
SUnreclaim           34     23     20     28     17     24     16     31     193
AnonHugePages         0      0      0      0      0      0      0      6       6
HugePages_Total       0      0      0      0      0      0      0      0       0
HugePages_Free        0      0      0      0      0      0      0      0       0
HugePages_Surp        0      0      0      0      0      0      0      0       0