Bug 1436544

Created attachment 1266846 [details]
vdsm log

Are you sure this has been working before? I don't remember if this has been ever tested. In which version did it work?

Also, there is a libvirt error, can you please provide libvirt and qemu logs?

If it is of any relevance, I can see that there may be some difference in NUMA policies on the two hosts.

Source:
  
  # numactl --show
  policy: default
  preferred node: current
  physcpubind: 0 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 152 
  cpubind: 0 1 16 17 
  nodebind: 0 1 16 17 
  membind: 0 1 16 17 

Destination:

  # numactl --show
  policy: default
  preferred node: current
  physcpubind: 0 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 152 
  cpubind: 0 16 
  nodebind: 0 16 
  membind: 0 16 

Comparing with the reported error

  libvirtError: Invalid value '0-1,16-17' for 'cpuset.mems': Invalid argument

there can be some relation. Possibly changed NUMA policies could be also one of possible explanations why the bug hasn't been present before.

Created attachment 1267539 [details]
Source host libvirtd log

Created attachment 1267540 [details]
Destination host libvirtd log

The bug seems to be in libvirt or below, reassigning to it.

The two hosts are in NUMA configuration described in comment #4.  The destination host has 4 NUMA nodes (0, 1, 16, 17), but 2 of them (1, 17) have inaccessible memory and the CPUs assigned to them have been offlined.  The CPU state is as follows:

  # lscpu
  Architecture:          ppc64le
  Byte Order:            Little Endian
  CPU(s):                160
  On-line CPU(s) list:   0,8,16,24,32,40,48,56,64,72
  Off-line CPU(s) list:  1-7,9-15,17-23,25-31,33-39,41-47,49-55,57-63,65-71,73-159
  Thread(s) per core:    1
  Core(s) per socket:    5
  Socket(s):             2
  NUMA node(s):          4
  Model:                 2.0 (pvr 004b 0200)
  Model name:            POWER8E (raw), altivec supported
  L1d cache:             64K
  L1i cache:             32K
  L2 cache:              512K
  L3 cache:              8192K
  NUMA node0 CPU(s):     0,8,16,24,32
  NUMA node1 CPU(s):     
  NUMA node16 CPU(s):    40,48,56,64,72
  NUMA node17 CPU(s):    

Here are the details how to reproduce the bug with virsh on that setup:

- On the source host, create dummy.xml domain XML file with the following content:

  <domain type='kvm'>
    <name>dummy</name>
    <uuid>4f836db9-e87f-425b-9278-5693be04a978</uuid>
    <vcpu placement='static' current='1'>16</vcpu>
    <memory unit='KiB'>1048576</memory>
    <os>
      <type arch='ppc64le' machine='pseries-rhel7.3.0'>hvm</type>
    </os>
  </domain>

- Start the VM:

  virsh create dummy.xml

- Hot plug 1 CPU to it:

  virsh setvcpus dummy 2

- Try to migrate the VM:

  virsh migrate dummy qemu+tcp://root@DESTINATION-HOST/system --verbose --live --auto-converge --compressed --p2p --abort-on-error

The command fails with the following error:

  error: Invalid value '0-1,16-17' for 'cpuset.mems': Invalid argument

See virsh-*.log attachments for libvirtd logs from the two hosts.

libvirt-2.0.0-10.el7_3.5.ppc64le
qemu-kvm-rhev-2.6.0-28.el7_3.8.ppc64le
kernel 3.10.0-514.16.1.el7.ppc64le

Since the nodes have no memory in it it's an instance of bug https://bugzilla.redhat.com/show_bug.cgi?id=1375268 . vCPU hotplug triggered this as vCPUs are hotplugged right after the qemu process is started.

*** This bug has been marked as a duplicate of bug 1375268 ***