RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1375268 - 'virsh setvcpus' fails on systems with no memory in some NUMA nodes
Summary: 'virsh setvcpus' fails on systems with no memory in some NUMA nodes
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.3
Hardware: ppc64le
OS: Linux
unspecified
high
Target Milestone: rc
: 7.4
Assignee: Peter Krempa
QA Contact: Dan Zheng
URL:
Whiteboard:
: 1428226 1436544 (view as bug list)
Depends On:
Blocks: 1224341 1299988
TreeView+ depends on / blocked
 
Reported: 2016-09-12 14:56 UTC by Andrea Bolognani
Modified: 2017-08-02 01:27 UTC (History)
13 users (show)

Fixed In Version: libvirt-2.5.0-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-01 17:14:13 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
guest configuration (2.21 KB, text/plain)
2016-09-12 14:56 UTC, Andrea Bolognani 		no flags Details
abakvm dmesg and var-log-messages (82.55 KB, text/plain)
2017-03-08 09:00 UTC, IBM Bug Proxy 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1846 0 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2017-08-01 18:02:50 UTC

Description Andrea Bolognani 2016-09-12 14:56:58 UTC 
Created attachment 1200236 [details]
guest configuration

I have a fairly basic guest[1] configured with

  <vcpu placement='static' current='8'>16</vcpu>
  ...
  <cpu>
    <topology sockets='2' cores='1' threads='8'/>
  </cpu>

and attempts to change the number of vCPUs at runtime
fails with

  # virsh setvcpus abologna-rhel73 16 --live
  error: Invalid value '0,8' for 'cpuset.mems': Invalid argument

The value libvirt is trying to use for cpuset.mems is
indeed invalid:

  # cat /sys/fs/cgroup/cpuset/cpuset.mems
  0

which, in turn, can be explained with the fact that
NUMA node 8 contains no memory:

  # numactl --hardware
  available: 2 nodes (0,8)
  node 0 cpus: 40 48 56 64 72 80 88 96
  node 0 size: 32768 MB
  node 0 free: 8015 MB
  node 8 cpus: 0 8 16 24 32
  node 8 size: 0 MB
  node 8 free: 0 MB
  node distances:
  node   0   8
    0:  10  40
    8:  40  10

libvirt should take this sort of configuration into account,
and only ever attempt to set cpuset.mems using valid values.


[1] Full guest configuration attached
Comment 3 Peter Krempa 2016-09-14 08:43:55 UTC 
Fixed upstream:

commit 77cb01bc0fec4d0da02e1d4df75d28870b0ef926
Author: Peter Krempa <pkrempa>
Date:   Tue Sep 13 15:55:06 2016 +0200

    numa: Rename virNumaGetHostNodeset and make it return only nodes with memory
    
    Name it virNumaGetHostMemoryNodeset and return only NUMA nodes which
    have memory installed. This is necessary as the kernel is not very happy
    to set the memory cgroup setting for nodes which do not have any memory.
    
    This would break vcpu hotplug with following message on such
    configruation:
    
      Invalid value '0,8' for 'cpuset.mems': Invalid argument
Comment 5 Jingjing Shao 2017-03-06 07:38:05 UTC 
Hi Andrea,

I try to reproduce this bug on x86_64 and ppc64le system, but I can not reproduce it on x86_64 system. The details steps are as below.

I wander that this issue just can be reproduced on ppc arch system, Is it right? If so, it need to change the "Hardware" parameter from "all" to the specified one . If not, can you provide more info to help me ? thank you in advance.



(1)For x86_64, I can not reproduce this issue. 
   
   On the host, I pull out the memory from the numa node and start the system. The numa node without memory is just ignored and can not be found in the system.
   
   I also try to reproduce this issue with vm and get the result as below.
    
    # virsh dumpxml rhel7.3 | grep cpu -A10
    ......
    <cpu mode='custom' match='exact'>
    <model fallback='allow'>Nehalem</model>
    <feature policy='require' name='vmx'/>
    <numa>
    <cell id='0' cpus='0-3' memory='512000' unit='KiB'/>
    <cell id='1' cpus='4-7' memory='0' unit='KiB'/>
    </numa>
    </cpu>
    ......

login in the guest.

    numactl -H
    available: 1 nodes (0)
    node 0 cpus: 0 1 2 3 4 5 6 7
    node 0 size: 499 MB
    node 0 free: 111 MB
    node distances:
    node 0
    0: 10

the numa node 1  also has been ignored.


(2) For the ppc64le, I can reproduce this issue. 
   
 # numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 8 16 24 32
node 0 size: 32768 MB
node 0 free: 18351 MB
node 1 cpus: 40 48 56 64 72
node 1 size: 0 MB
node 1 free: 0 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 



 # virsh dumpxml rhel73 | grep cpu -A5
  ...
  <vcpu placement='static' current='8'>16</vcpu>
  ...
   <cpu>
    <topology sockets='2' cores='1' threads='8'/>
  </cpu>
  ...


 # virsh setvcpus rhel73 16 --live
 error: Invalid value '0-1' for 'cpuset.mems': Invalid argument
Comment 6 Peter Krempa 2017-03-06 08:06:11 UTC 
(In reply to Jingjing Shao from comment #5)
> Hi Andrea,
> 
> I try to reproduce this bug on x86_64 and ppc64le system, but I can not
> reproduce it on x86_64 system. The details steps are as below.
> 
> I wander that this issue just can be reproduced on ppc arch system, Is it
> right? If so, it need to change the "Hardware" parameter from "all" to the
> specified one . If not, can you provide more info to help me ? thank you in
> advance.

The problem above should be possible to reproduce on any architecture but as you've noted below on x86_64 it's not easy to achieve a NUMA node without memory since it's usually ignored.

On PPC64 there were machines which reported one node without memory incorrectly and thus the bug manifested itself there. I think it's sufficient to reproduce it on PPC64 only.
Comment 7 Jingjing Shao 2017-03-07 07:46:58 UTC 
(In reply to Peter Krempa from comment #6)
> The problem above should be possible to reproduce on any architecture but as
> you've noted below on x86_64 it's not easy to achieve a NUMA node without
> memory since it's usually ignored.
> 
> On PPC64 there were machines which reported one node without memory
> incorrectly and thus the bug manifested itself there. I think it's
> sufficient to reproduce it on PPC64 only.


Thanks Peter, I will update the "Hardware" parameter from "all" to "ppc" and change the QA Contact
Comment 8 Andrea Bolognani 2017-03-07 14:13:06 UTC 
(In reply to Jingjing Shao from comment #7)
> Thanks Peter, I will update the "Hardware" parameter from "all" to "ppc" and
> change the QA Contact

I think Hardware should be ppc64 or ppc64le, not ppc.
Comment 9 Andrea Bolognani 2017-03-07 17:31:16 UTC 
*** Bug 1428226 has been marked as a duplicate of this bug. ***
Comment 10 IBM Bug Proxy 2017-03-08 09:00:27 UTC 
Created attachment 1261141 [details]
abakvm dmesg and var-log-messages
Comment 11 Dan Zheng 2017-03-09 07:09:51 UTC 
Test packages:

libvirt-3.1.0-1.el7.ppc64le
qemu-kvm-rhev-2.8.0-5.el7.ppc64le
kernel-3.10.0-578.el7.ppc64le

Steps:
0. Check host numa node
# numactl --hard
available: 2 nodes (0-1)
node 0 cpus: 0 8 16 24 32
node 0 size: 32768 MB
node 0 free: 22567 MB
node 1 cpus: 40 48 56 64 72
node 1 size: 0 MB
node 1 free: 0 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 

1. Start a guest with below XML:
 <vcpu placement='static' current='8'>10</vcpu>
 <cpu>
   <topology sockets='2' cores='1' threads='8'/>
 </cpu>

2. Hotplug vcpu from 8 to 16
# virsh setvcpus dd 16 --live
3. Dumpxml
# virsh dumpxml dd
  <vcpu placement='static' current='8'>16</vcpu>
  <vcpus>
    <vcpu id='0' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='1' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='2' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='3' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='4' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='5' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='6' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='7' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='8' enabled='yes' hotpluggable='yes' order='2'/>
    <vcpu id='9' enabled='yes' hotpluggable='yes' order='2'/>
    <vcpu id='10' enabled='yes' hotpluggable='yes' order='2'/>
    <vcpu id='11' enabled='yes' hotpluggable='yes' order='2'/>
    <vcpu id='12' enabled='yes' hotpluggable='yes' order='2'/>
    <vcpu id='13' enabled='yes' hotpluggable='yes' order='2'/>
    <vcpu id='14' enabled='yes' hotpluggable='yes' order='2'/>
    <vcpu id='15' enabled='yes' hotpluggable='yes' order='2'/>
  </vcpus>
4. Hotunplug vcpu from 16 to 8
#  virsh setvcpus dd 8 --live
  <vcpus>
    <vcpu id='0' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='1' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='2' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='3' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='4' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='5' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='6' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='7' enabled='yes' hotpluggable='no' order='1'/>
    <vcpu id='8' enabled='no' hotpluggable='yes'/>
    <vcpu id='9' enabled='no' hotpluggable='yes'/>
    <vcpu id='10' enabled='no' hotpluggable='yes'/>
    <vcpu id='11' enabled='no' hotpluggable='yes'/>
    <vcpu id='12' enabled='no' hotpluggable='yes'/>
    <vcpu id='13' enabled='no' hotpluggable='yes'/>
    <vcpu id='14' enabled='no' hotpluggable='yes'/>
    <vcpu id='15' enabled='no' hotpluggable='yes'/>
  </vcpus>

So make it pass.
Comment 12 Andrea Bolognani 2017-03-09 13:27:40 UTC 
(In reply to Dan Zheng from comment #11)
> 1. Start a guest with below XML:
>  <vcpu placement='static' current='8'>10</vcpu>
>  <cpu>
>    <topology sockets='2' cores='1' threads='8'/>
>  </cpu>

Was the value of <vcpu> for the guest really 10?

You should have gotten error

  unsupported configuration: CPU topology doesn't match
  maximum vcpu count

unless you set it to 16.
Comment 13 Dan Zheng 2017-03-10 02:40:02 UTC 
Hi Andrea,
Thanks for your pointing this out. It is my copy error. I used '16' not '10' for this test. '10' was used in another test.

*** Correct the guest XML in comment 11.
1. Start a guest with below XML:
 <vcpu placement='static' current='8'>16</vcpu>
 <cpu>
   <topology sockets='2' cores='1' threads='8'/>
 </cpu>




(In reply to Andrea Bolognani from comment #12)
> (In reply to Dan Zheng from comment #11)
> > 1. Start a guest with below XML:
> >  <vcpu placement='static' current='8'>10</vcpu>
> >  <cpu>
> >    <topology sockets='2' cores='1' threads='8'/>
> >  </cpu>
> 
> Was the value of <vcpu> for the guest really 10?
> 
> You should have gotten error
> 
>   unsupported configuration: CPU topology doesn't match
>   maximum vcpu count
> 
> unless you set it to 16.
Comment 14 Peter Krempa 2017-03-30 14:49:50 UTC 
*** Bug 1436544 has been marked as a duplicate of this bug. ***
Comment 15 errata-xmlrpc 2017-08-01 17:14:13 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846
Comment 16 errata-xmlrpc 2017-08-01 23:55:08 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846
Comment 17 errata-xmlrpc 2017-08-02 01:27:35 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846
Note You need to log in before you can comment on or make changes to this bug.