Bug 1375268

Summary: 'virsh setvcpus' fails on systems with no memory in some NUMA nodes
Product: Red Hat Enterprise Linux 7 Reporter: Andrea Bolognani <abologna>
Component: libvirtAssignee: Peter Krempa <pkrempa>
Status: CLOSED ERRATA QA Contact: Dan Zheng <dzheng>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.3CC: abologna, bugproxy, dyuan, dzheng, gsun, hannsj_uhl, ipinto, jdenemar, lhuang, mgoldboi, rbalakri, xuzhang, yalzhang
Target Milestone: rcKeywords: Patch
Target Release: 7.4   
Hardware: ppc64le   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-2.5.0-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-01 17:14:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1224341, 1299988    
Attachments:
Description Flags
guest configuration
none
abakvm dmesg and var-log-messages none

Description Andrea Bolognani 2016-09-12 14:56:58 UTC 
Created attachment 1200236 [details]
guest configuration

I have a fairly basic guest[1] configured with

  <vcpu placement='static' current='8'>16</vcpu>
  ...
  <cpu>
    <topology sockets='2' cores='1' threads='8'/>
  </cpu>

and attempts to change the number of vCPUs at runtime
fails with

  # virsh setvcpus abologna-rhel73 16 --live
  error: Invalid value '0,8' for 'cpuset.mems': Invalid argument

The value libvirt is trying to use for cpuset.mems is
indeed invalid:

  # cat /sys/fs/cgroup/cpuset/cpuset.mems
  0

which, in turn, can be explained with the fact that
NUMA node 8 contains no memory:

  # numactl --hardware
  available: 2 nodes (0,8)
  node 0 cpus: 40 48 56 64 72 80 88 96
  node 0 size: 32768 MB
  node 0 free: 8015 MB
  node 8 cpus: 0 8 16 24 32
  node 8 size: 0 MB
  node 8 free: 0 MB
  node distances:
  node   0   8
    0:  10  40
    8:  40  10

libvirt should take this sort of configuration into account,
and only ever attempt to set cpuset.mems using valid values.


[1] Full guest configuration attached
Comment 3 Peter Krempa 2016-09-14 08:43:55 UTC 
Fixed upstream:

commit 77cb01bc0fec4d0da02e1d4df75d28870b0ef926
Author: Peter Krempa <pkrempa>
Date:   Tue Sep 13 15:55:06 2016 +0200

    numa: Rename virNumaGetHostNodeset and make it return only nodes with memory
    
    Name it virNumaGetHostMemoryNodeset and return only NUMA nodes which
    have memory installed. This is necessary as the kernel is not very happy
    to set the memory cgroup setting for nodes which do not have any memory.
    
    This would break vcpu hotplug with following message on such
    configruation:
    
      Invalid value '0,8' for 'cpuset.mems': Invalid argument
Comment 5 Jingjing Shao 2017-03-06 07:38:05 UTC 
Hi Andrea,

I try to reproduce this bug on x86_64 and ppc64le system, but I can not reproduce it on x86_64 system. The details steps are as below.

I wander that this issue just can be reproduced on ppc arch system, Is it right? If so, it need to change the "Hardware" parameter from "all" to the specified one . If not, can you provide more info to help me ? thank you in advance.



(1)For x86_64, I can not reproduce this issue. 
   
   On the host, I pull out the memory from the numa node and start the system. The numa node without memory is just ignored and can not be found in the system.
   
   I also try to reproduce this issue with vm and get the result as below.
    
    # virsh dumpxml rhel7.3 | grep cpu -A10
    ......
    <cpu mode='custom' match='exact'>
    <model fallback='allow'>Nehalem</model>
    <feature policy='require' name='vmx'/>
    <numa>
    <cell id='0' cpus='0-3' memory='512000' unit='KiB'/>
    <cell id='1' cpus='4-7' memory='0' unit='KiB'/>
    </numa>
    </cpu>
    ......

login in the guest.

    numactl -H
    available: 1 nodes (0)
    node 0 cpus: 0 1 2 3 4 5 6 7
    node 0 size: 499 MB
    node 0 free: 111 MB
    node distances:
    node 0
    0: 10

the numa node 1  also has been ignored.


(2) For the ppc64le, I can reproduce this issue. 
   
 # numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 8 16 24 32
node 0 size: 32768 MB
node 0 free: 18351 MB
node 1 cpus: 40 48 56 64 72
node 1 size: 0 MB
node 1 free: 0 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 



 # virsh dumpxml rhel73 | grep cpu -A5
  ...
  <vcpu placement='static' current='8'>16</vcpu>
  ...
   <cpu>
    <topology sockets='2' cores='1' threads='8'/>
  </cpu>
  ...


 # virsh setvcpus rhel73 16 --live
 error: Invalid value '0-1' for 'cpuset.mems': Invalid argument
Comment 6 Peter Krempa 2017-03-06 08:06:11 UTC 
(In reply to Jingjing Shao from comment #5)
> Hi Andrea,
> 
> I try to reproduce this bug on x86_64 and ppc64le system, but I can not
> reproduce it on x86_64 system. The details steps are as below.
> 
> I wander that this issue just can be reproduced on ppc arch system, Is it
> right? If so, it need to change the "Hardware" parameter from "all" to the
> specified one . If not, can you provide more info to help me ? thank you in
> advance.

The problem above should be possible to reproduce on any architecture but as you've noted below on x86_64 it's not easy to achieve a NUMA node without memory since it's usually ignored.

On PPC64 there were machines which reported one node without memory incorrectly and thus the bug manifested itself there. I think it's sufficient to reproduce it on PPC64 only.
Comment 7 Jingjing Shao 2017-03-07 07:46:58 UTC 
(In reply to Peter Krempa from comment #6)
> The problem above should be possible to reproduce on any architecture but as
> you've noted below on x86_64 it's not easy to achieve a NUMA node without
> memory since it's usually ignored.
> 
> On PPC64 there were machines which reported one node without memory
> incorrectly and thus the bug manifested itself there. I think it's
> sufficient to reproduce it on PPC64 only.


Thanks Peter, I will update the "Hardware" parameter from "all" to "ppc" and change the QA Contact
Comment 8 Andrea Bolognani 2017-03-07 14:13:06 UTC 
(In reply to Jingjing Shao from comment #7)
> Thanks Peter, I will update the "Hardware" parameter from "all" to "ppc" and
> change the QA Contact

I think Hardware should be ppc64 or ppc64le, not ppc.
Comment 9 Andrea Bolognani 2017-03-07 17:31:16 UTC 
*** Bug 1428226 has been marked as a duplicate of this bug. ***
Comment 10 IBM Bug Proxy 2017-03-08 09:00:27 UTC 
Created attachment 1261141 [details]
abakvm dmesg and var-log-messages
Comment 11 Dan Zheng 2017-03-09 07:09:51 UTC 
Test packages:

libvirt-3.1.0-1.el7.ppc64le
qemu-kvm-rhev-2.8.0-5.el7.ppc64le
kernel-3.10.0-578.el7.ppc64le

Steps:
0. Check host numa node
# numactl --hard
available: 2 nodes (0-1)
node 0 cpus: 0 8 16 24 32
node 0 size: 32768 MB
node 0 free: 22567 MB
node 1 cpus: 40 48 56 64 72
node 1 size: 0 MB
node 1 free: 0 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 

1. Start a guest with below XML:
 <vcpu placement='static' current='8'>10</vcpu>
 <cpu>
   <topology sockets='2' cores='1' threads='8'/>
 </cpu>

2. Hotplug vcpu from 8 to 16
# virsh setvcpus dd 16 --live
3. Dumpxml
# virsh dumpxml dd
  <vcpu placement='static' current='8'>16</vcpu>
  <vcpus>
    <vcpu id='0' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='1' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='2' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='3' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='4' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='5' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='6' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='7' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='8' enabled='yes' hotpluggable='yes' order='2'/>
    <vcpu id='9' enabled='yes' hotpluggable='yes' order='2'/>
    <vcpu id='10' enabled='yes' hotpluggable='yes' order='2'/>
    <vcpu id='11' enabled='yes' hotpluggable='yes' order='2'/>
    <vcpu id='12' enabled='yes' hotpluggable='yes' order='2'/>
    <vcpu id='13' enabled='yes' hotpluggable='yes' order='2'/>
    <vcpu id='14' enabled='yes' hotpluggable='yes' order='2'/>
    <vcpu id='15' enabled='yes' hotpluggable='yes' order='2'/>
  </vcpus>
4. Hotunplug vcpu from 16 to 8
#  virsh setvcpus dd 8 --live
  <vcpus>
    <vcpu id='0' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='1' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='2' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='3' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='4' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='5' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='6' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='7' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='8' enabled='no' hotpluggable='yes'/>
    <vcpu id='9' enabled='no' hotpluggable='yes'/>
    <vcpu id='10' enabled='no' hotpluggable='yes'/>
    <vcpu id='11' enabled='no' hotpluggable='yes'/>
    <vcpu id='12' enabled='no' hotpluggable='yes'/>
    <vcpu id='13' enabled='no' hotpluggable='yes'/>
    <vcpu id='14' enabled='no' hotpluggable='yes'/>
    <vcpu id='15' enabled='no' hotpluggable='yes'/>
  </vcpus>

So make it pass.
Comment 12 Andrea Bolognani 2017-03-09 13:27:40 UTC 
(In reply to Dan Zheng from comment #11)
> 1. Start a guest with below XML:
>  <vcpu placement='static' current='8'>10</vcpu>
>  <cpu>
>    <topology sockets='2' cores='1' threads='8'/>
>  </cpu>

Was the value of <vcpu> for the guest really 10?

You should have gotten error

  unsupported configuration: CPU topology doesn't match
  maximum vcpu count

unless you set it to 16.
Comment 13 Dan Zheng 2017-03-10 02:40:02 UTC 
Hi Andrea,
Thanks for your pointing this out. It is my copy error. I used '16' not '10' for this test. '10' was used in another test.

*** Correct the guest XML in comment 11.
1. Start a guest with below XML:
 <vcpu placement='static' current='8'>16</vcpu>
 <cpu>
   <topology sockets='2' cores='1' threads='8'/>
 </cpu>




(In reply to Andrea Bolognani from comment #12)
> (In reply to Dan Zheng from comment #11)
> > 1. Start a guest with below XML:
> >  <vcpu placement='static' current='8'>10</vcpu>
> >  <cpu>
> >    <topology sockets='2' cores='1' threads='8'/>
> >  </cpu>
> 
> Was the value of <vcpu> for the guest really 10?
> 
> You should have gotten error
> 
>   unsupported configuration: CPU topology doesn't match
>   maximum vcpu count
> 
> unless you set it to 16.
Comment 14 Peter Krempa 2017-03-30 14:49:50 UTC 
*** Bug 1436544 has been marked as a duplicate of this bug. ***
Comment 15 errata-xmlrpc 2017-08-01 17:14:13 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846
Comment 16 errata-xmlrpc 2017-08-01 23:55:08 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846
Comment 17 errata-xmlrpc 2017-08-02 01:27:35 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846