Bug 1168944

Summary: "libvirtError: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000002.scope/cpuset.mems': Device or resource busy"
Product: Red Hat Enterprise Linux 7 Reporter: Kashyap Chamarthy <kchamart>
Component: libvirtAssignee: Libvirt Maintainers <libvirt-maint>
Status: CLOSED DUPLICATE QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.1CC: agedosier, berrange, clalancette, extras-qa, itamar, jforbes, laine, libvirt-maint, mkletzan, ndipanov, rbalakri, veillard, virt-maint
Target Milestone: rcKeywords: Upstream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1168672 Environment:
Last Closed: 2014-11-28 14:13:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1168672    
Bug Blocks: 1168866    

Description Kashyap Chamarthy 2014-11-28 13:46:27 UTC 
[This bug is fixed in libvirt upstream git, and will be available in 1.2.11. libvirt commit that fixes it: c6e90248676126c209b3b6017ad27cf6c6a0ab8f ]

+++ This bug was initially created as a clone of Bug #1168672 +++

Description of problem
----------------------

This occurs when you boot a Nova instance with NUMA topology.

[. . .]
libvirtError: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000002.scope/cpuset.mems': Device or resource busy
[. . .'

Version
-------

Versions of libvirt, QEMU, systemd in the OpenStack setup (running in a
DevStack VM):

  $ uname -r; rpm -q libvirt-daemon-kvm qemu-system-x86 systemd
  3.18.0-0.rc6.git0.1.fc22.x86_64
  libvirt-daemon-kvm-1.2.10-3.fc22.x86_64
  qemu-system-x86-2.2.0-0.1.rc1.fc22.x86_64
  systemd-216-11.fc21.x86_64


How reproducible: Atleast twice.


Steps to reproduce
------------------

This occurred in when booting a Nova instance in a DevStack (OpenStack
developer setup) environment. The DevStack machine is a KVM guest
hypervisor, and the Nova guest is a nested guest running on this.

It's a fairly involved test environment, details here:

http://docs-draft.openstack.org/18/131818/1/check/gate-nova-docs/2ddc418/doc/build/html/devref/testing/libvirt-numa.html#testing-basis-non-numa-usage
docs-draft.openstack.org/18/131818/1/check/gate-nova-docs/2ddc418/doc/build/html/devref/testing/libvirt-numa.html#testing-basis-non-numa-usage


Actual results
--------------

[. . .]
2014-11-26 20:17:28.722 ERROR nova.compute.manager [-] [instance: bcb53b78-452c-4695-b39d-754389cd3dd5] Instance failed to spawn
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5] Traceback (most recent call last):
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/home/kashyapc/src/cloud/nova/nova/compute
/manager.py", line 2247, in _build_resources
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     yield resources
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/home/kashyapc/src/cloud/nova/nova/compute
/manager.py", line 2117, in _build_and_run_instance
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     instance_type=instance_type)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/home/kashyapc/src/cloud/nova/nova/virt/li
bvirt/driver.py", line 2640, in spawn
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     block_device_info, disk_info=disk_info)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/home/kashyapc/src/cloud/nova/nova/virt/libvirt/driver.py", line 4500, in _create_domain_and_network
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     power_on=power_on)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/home/kashyapc/src/cloud/nova/nova/virt/libvirt/driver.py", line 4433, in _create_domain
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     LOG.error(err)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/usr/lib/python2.7/site-packages/oslo/utils/excutils.py", line 82, in __exit__
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     six.reraise(self.type_, self.value, self.tb)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/home/kashyapc/src/cloud/nova/nova/virt/libvirt/driver.py", line 4423, in _create_domain
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     domain.createWithFlags(launch_flags)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     rv = execute(f, *args, **kwargs)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     six.reraise(c, e, tb)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     rv = meth(*args, **kwargs)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1033, in createWithFlags
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5] libvirtError: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000002.scope/cpuset.mems': Device or resource busy
[. . .]


Expected results
----------------

Nova instance (libvirt nested guest) should boot successfully.


Additional info
---------------

[1] Inventory of available NUMA nodes on the physical host:

$ numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 4 8 12 16 20 24 28 32 36 40 44
node 0 size: 257954 MB
node 0 free: 248486 MB
node 1 cpus: 1 5 9 13 17 21 25 29 33 37 41 45
node 1 size: 258045 MB
node 1 free: 256470 MB
node 2 cpus: 2 6 10 14 18 22 26 30 34 38 42 46
node 2 size: 258045 MB
node 2 free: 256507 MB
node 3 cpus: 3 7 11 15 19 23 27 31 35 39 43 47
node 3 size: 258040 MB
node 3 free: 256457 MB
node distances:
node   0   1   2   3 
  0:  10  20  20  20 
  1:  20  10  20  20 
  2:  20  20  10  20 
  3:  20  20  20  10 

[2] From sytemd `journactl`:

$ sudo journalctl -u libvirtd -l -p err
[. . .]
Nov 26 09:19:49 devstack libvirtd[32697]: driver in virRegisterStorageDriver must not be NULL
Nov 26 09:19:49 devstack libvirtd[32697]: Failed module registration vboxStorageRegister
Nov 26 10:33:51 devstack libvirtd[32697]: End of file while reading data: Input/output error
Nov 26 11:12:36 devstack libvirtd[32697]: End of file while reading data: Input/output error
Nov 26 20:17:28 devstack libvirtd[32697]: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000002.scope/cpuset.mems': Device or resource busy
Nov 26 20:17:28 devstack libvirtd[32697]: error from service: TerminateMachine: No machine 'qemu-instance-00000002' known
[. . .]


[3] From an existing Nova instance (that was booted _without_ NUMA)

    $ find /sys/fs/cgroup/cpuset/machine.slice/machine-qemu*/ -name cpuset.mems 
    /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000001.scope/vcpu0/cpuset.mems
    /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000001.scope/cpuset.mems
    /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000001.scope/emulator/cpuset.mems
    
    $ cd /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\\x2dinstance\\x2d00000001.scope/
    $ cat cpuset.mems vcpu0/cpuset.mems emulator/cpuset.mems 
    0-2
    0-2
    0-2


[4] libvirt developer Martin Kletzander confirmed on IRC (#virt, OFTC) that
this is a bug.

--- Additional comment from Kashyap Chamarthy on 2014-11-27 10:16:02 EST ---

Contextual snippet related to cgroups from the attached libvirtd debug log:

[. . .]
2014-11-27 14:18:20.835+0000: 25475: debug : virCgroupSetValueStr:718 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000003.scope/cpuset.mems' to '0'
2014-11-27 14:18:20.836+0000: 25475: debug : virFileClose:99 : Closed fd 25
2014-11-27 14:18:20.836+0000: 25475: error : virCgroupSetValueStr:728 : Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000003.scope/cpuset.mems': Device or resource busy
2014-11-27 14:18:20.836+0000: 25475: debug : virFileClose:99 : Closed fd 24
[. . .]
2014-11-27 14:18:21.041+0000: 25475: debug : virDBusMessageIterEncode:640 : Popping iter=0x7fd43ef944d0
2014-11-27 14:18:21.042+0000: 25475: error : virDBusCall:1537 : error from service: TerminateMachine: No machine 'qemu-instance-00000003' known
2014-11-27 14:18:21.042+0000: 25475: debug : qemuRemoveCgroup:1222 : Failed to terminate cgroup for instance-00000003
2014-11-27 14:18:21.042+0000: 25475: debug : virObjectUnref:259 : OBJECT_UNREF: obj=0x7fd4300e6890
2014-11-27 14:18:21.042+0000: 25475: debug : virCgroupRemove:3331 : Removing cgroup /machine.slice/machine-qemu\x2dinstance\x2d00000003.scope
2014-11-27 14:18:21.042+0000: 25475: debug : virCgroupRemove:3352 : Removing cgroup /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2dinstance\x2d00000003.scope/ and all child cgroups
2014-11-27 14:18:21.042+0000: 25475: debug : virCgroupRemove:3352 : Removing cgroup /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2dinstance\x2d00000003.scope/ and all child cgroups
2014-11-27 14:18:21.042+0000: 25475: debug : virCgroupRemove:3352 : Removing cgroup /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000003.scope/ and all child cgroups
2014-11-27 14:18:21.042+0000: 25475: debug : virCgroupRemoveRecursively:3302 : Removing cgroup /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000003.scope//emulator
[. . .]

--- Additional comment from Kashyap Chamarthy on 2014-11-27 10:28:09 EST ---

DevStack machine's capabilities where the Nova Compute service is running and a libvirt instance (a nested guest) failed to started)

---------------------
DevStack>$ virsh capabilities
<capabilities>

  <host>
    <uuid>7dae2ee3-8950-9a41-9e24-a463a8563bbd</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>Nehalem</model>
      <vendor>Intel</vendor>
      <topology sockets='1' cores='8' threads='1'/>
      <feature name='rdtscp'/>
      <feature name='hypervisor'/>
      <feature name='x2apic'/>
      <feature name='vmx'/>
      <feature name='ss'/>
      <feature name='vme'/>
      <pages unit='KiB' size='4'/>
      <pages unit='KiB' size='2048'/>
    </cpu>
    <power_management>
      <suspend_mem/>
      <suspend_disk/>
      <suspend_hybrid/>
    </power_management>
    <migration_features>
      <live/>
      <uri_transports>
        <uri_transport>tcp</uri_transport>
        <uri_transport>rdma</uri_transport>
      </uri_transports>
    </migration_features>
    <topology>
      <cells num='3'>
        <cell id='0'>
          <memory unit='KiB'>3949740</memory>
          <pages unit='KiB' size='4'>987435</pages>
          <pages unit='KiB' size='2048'>0</pages>
          <distances>
            <sibling id='0' value='10'/>
            <sibling id='1' value='20'/>
            <sibling id='2' value='20'/>
          </distances>
          <cpus num='4'>
            <cpu id='0' socket_id='0' core_id='0' siblings='0'/>
            <cpu id='1' socket_id='1' core_id='0' siblings='1'/>
            <cpu id='2' socket_id='2' core_id='0' siblings='2'/>
            <cpu id='3' socket_id='3' core_id='0' siblings='3'/>
          </cpus>
        </cell>
        <cell id='1'>
          <memory unit='KiB'>2016864</memory>
          <pages unit='KiB' size='4'>504216</pages>
          <pages unit='KiB' size='2048'>0</pages>
          <distances>
            <sibling id='0' value='20'/>
            <sibling id='1' value='10'/>
            <sibling id='2' value='20'/>
          </distances>
          <cpus num='2'>
            <cpu id='4' socket_id='4' core_id='0' siblings='4'/>
            <cpu id='5' socket_id='5' core_id='0' siblings='5'/>
          </cpus>
        </cell>
        <cell id='2'>
          <memory unit='KiB'>2014304</memory>
          <pages unit='KiB' size='4'>503576</pages>
          <pages unit='KiB' size='2048'>0</pages>
          <distances>
            <sibling id='0' value='20'/>
            <sibling id='1' value='20'/>
            <sibling id='2' value='10'/>
          </distances>
          <cpus num='2'>
            <cpu id='6' socket_id='6' core_id='0' siblings='6'/>
            <cpu id='7' socket_id='7' core_id='0' siblings='7'/>
          </cpus>
        </cell>
      </cells>
    </topology>
    <secmodel>
      <model>selinux</model>
      <doi>0</doi>
      <baselabel type='kvm'>system_u:system_r:svirt_t:s0</baselabel>
      <baselabel type='qemu'>system_u:system_r:svirt_tcg_t:s0</baselabel>
    </secmodel>
    <secmodel>
      <model>dac</model>
      <doi>0</doi>
      <baselabel type='kvm'>+107:+107</baselabel>
      <baselabel type='qemu'>+107:+107</baselabel>
    </secmodel>
  </host>

  <guest>
    <os_type>hvm</os_type>
    <arch name='i686'>
      <wordsize>32</wordsize>
      <emulator>/usr/bin/qemu-system-i386</emulator>
      <machine canonical='pc-i440fx-2.2' maxCpus='255'>pc</machine>
      <machine maxCpus='255'>pc-0.12</machine>
      <machine maxCpus='255'>pc-1.3</machine>
      <machine maxCpus='255'>pc-q35-1.6</machine>
      <machine maxCpus='255'>pc-q35-1.5</machine>
      <machine maxCpus='255'>pc-i440fx-1.6</machine>
      <machine canonical='pc-q35-2.2' maxCpus='255'>q35</machine>
      <machine maxCpus='1'>xenpv</machine>
      <machine maxCpus='255'>pc-i440fx-1.7</machine>
      <machine maxCpus='255'>pc-q35-2.1</machine>
      <machine maxCpus='255'>pc-0.11</machine>
      <machine maxCpus='255'>pc-0.10</machine>
      <machine maxCpus='255'>pc-1.2</machine>
      <machine maxCpus='1'>isapc</machine>
      <machine maxCpus='255'>pc-q35-1.4</machine>
      <machine maxCpus='128'>xenfv</machine>
      <machine maxCpus='255'>pc-0.15</machine>
      <machine maxCpus='255'>pc-i440fx-1.5</machine>
      <machine maxCpus='255'>pc-0.14</machine>
      <machine maxCpus='255'>pc-q35-2.0</machine>
      <machine maxCpus='255'>pc-i440fx-1.4</machine>
      <machine maxCpus='255'>pc-1.1</machine>
      <machine maxCpus='255'>pc-q35-1.7</machine>
      <machine maxCpus='255'>pc-i440fx-2.1</machine>
      <machine maxCpus='255'>pc-1.0</machine>
      <machine maxCpus='255'>pc-i440fx-2.0</machine>
      <machine maxCpus='255'>pc-0.13</machine>
      <domain type='qemu'>
      </domain>
      <domain type='kvm'>
        <emulator>/usr/bin/qemu-kvm</emulator>
        <machine canonical='pc-i440fx-2.2' maxCpus='255'>pc</machine>
        <machine maxCpus='255'>pc-1.3</machine>
        <machine maxCpus='255'>pc-0.12</machine>
        <machine maxCpus='255'>pc-q35-1.6</machine>
        <machine maxCpus='255'>pc-q35-1.5</machine>
        <machine maxCpus='255'>pc-i440fx-1.6</machine>
        <machine canonical='pc-q35-2.2' maxCpus='255'>q35</machine>
        <machine maxCpus='255'>pc-i440fx-1.7</machine>
        <machine maxCpus='1'>xenpv</machine>
        <machine maxCpus='255'>pc-q35-2.1</machine>
        <machine maxCpus='255'>pc-0.11</machine>
        <machine maxCpus='255'>pc-0.10</machine>
        <machine maxCpus='255'>pc-1.2</machine>
        <machine maxCpus='1'>isapc</machine>
        <machine maxCpus='255'>pc-q35-1.4</machine>
        <machine maxCpus='128'>xenfv</machine>
        <machine maxCpus='255'>pc-0.15</machine>
        <machine maxCpus='255'>pc-i440fx-1.5</machine>
        <machine maxCpus='255'>pc-i440fx-1.4</machine>
        <machine maxCpus='255'>pc-q35-2.0</machine>
        <machine maxCpus='255'>pc-0.14</machine>
        <machine maxCpus='255'>pc-1.1</machine>
        <machine maxCpus='255'>pc-q35-1.7</machine>
        <machine maxCpus='255'>pc-i440fx-2.1</machine>
        <machine maxCpus='255'>pc-1.0</machine>
        <machine maxCpus='255'>pc-i440fx-2.0</machine>
        <machine maxCpus='255'>pc-0.13</machine>
      </domain>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <disksnapshot default='on' toggle='no'/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
      <pae/>
      <nonpae/>
    </features>
  </guest>

  <guest>
    <os_type>hvm</os_type>
    <arch name='x86_64'>
      <wordsize>64</wordsize>
      <emulator>/usr/bin/qemu-system-x86_64</emulator>
      <machine canonical='pc-i440fx-2.2' maxCpus='255'>pc</machine>
      <machine maxCpus='255'>pc-1.3</machine>
      <machine maxCpus='255'>pc-0.12</machine>
      <machine maxCpus='255'>pc-q35-1.6</machine>
      <machine maxCpus='255'>pc-q35-1.5</machine>
      <machine maxCpus='255'>pc-i440fx-1.6</machine>
      <machine canonical='pc-q35-2.2' maxCpus='255'>q35</machine>
      <machine maxCpus='255'>pc-i440fx-1.7</machine>
      <machine maxCpus='1'>xenpv</machine>
      <machine maxCpus='255'>pc-q35-2.1</machine>
      <machine maxCpus='255'>pc-0.11</machine>
      <machine maxCpus='255'>pc-0.10</machine>
      <machine maxCpus='255'>pc-1.2</machine>
      <machine maxCpus='1'>isapc</machine>
      <machine maxCpus='255'>pc-q35-1.4</machine>
      <machine maxCpus='128'>xenfv</machine>
      <machine maxCpus='255'>pc-0.15</machine>
      <machine maxCpus='255'>pc-i440fx-1.5</machine>
      <machine maxCpus='255'>pc-i440fx-1.4</machine>
      <machine maxCpus='255'>pc-q35-2.0</machine>
      <machine maxCpus='255'>pc-0.14</machine>
      <machine maxCpus='255'>pc-1.1</machine>
      <machine maxCpus='255'>pc-q35-1.7</machine>
      <machine maxCpus='255'>pc-i440fx-2.1</machine>
      <machine maxCpus='255'>pc-1.0</machine>
      <machine maxCpus='255'>pc-i440fx-2.0</machine>
      <machine maxCpus='255'>pc-0.13</machine>
      <domain type='qemu'>
      </domain>
      <domain type='kvm'>
        <emulator>/usr/bin/qemu-kvm</emulator>
        <machine canonical='pc-i440fx-2.2' maxCpus='255'>pc</machine>
        <machine maxCpus='255'>pc-1.3</machine>
        <machine maxCpus='255'>pc-0.12</machine>
        <machine maxCpus='255'>pc-q35-1.6</machine>
        <machine maxCpus='255'>pc-q35-1.5</machine>
        <machine maxCpus='255'>pc-i440fx-1.6</machine>
        <machine canonical='pc-q35-2.2' maxCpus='255'>q35</machine>
        <machine maxCpus='255'>pc-i440fx-1.7</machine>
        <machine maxCpus='1'>xenpv</machine>
        <machine maxCpus='255'>pc-q35-2.1</machine>
        <machine maxCpus='255'>pc-0.11</machine>
        <machine maxCpus='255'>pc-0.10</machine>
        <machine maxCpus='255'>pc-1.2</machine>
        <machine maxCpus='1'>isapc</machine>
        <machine maxCpus='255'>pc-q35-1.4</machine>
        <machine maxCpus='128'>xenfv</machine>
        <machine maxCpus='255'>pc-0.15</machine>
        <machine maxCpus='255'>pc-i440fx-1.5</machine>
        <machine maxCpus='255'>pc-i440fx-1.4</machine>
        <machine maxCpus='255'>pc-q35-2.0</machine>
        <machine maxCpus='255'>pc-0.14</machine>
        <machine maxCpus='255'>pc-1.1</machine>
        <machine maxCpus='255'>pc-q35-1.7</machine>
        <machine maxCpus='255'>pc-i440fx-2.1</machine>
        <machine maxCpus='255'>pc-1.0</machine>
        <machine maxCpus='255'>pc-i440fx-2.0</machine>
        <machine maxCpus='255'>pc-0.13</machine>
      </domain>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <disksnapshot default='on' toggle='no'/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
    </features>
  </guest>

</capabilities>
---------------------

--- Additional comment from Kashyap Chamarthy on 2014-11-27 10:29:07 EST ---



--- Additional comment from Kashyap Chamarthy on 2014-11-27 10:33:39 EST ---

Contextual snippet from the attachment:

[. . .]
  <memory>1048576</memory>
  <numatune>
    <memory mode="strict" nodeset="0"/>
    <memnode cellid="0" mode="strict" nodeset="0"/>
  </numatune>
  <vcpu>4</vcpu>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="2015.1"/>
      <nova:name>cirrvm3</nova:name>
      <nova:creationTime>2014-11-27 14:18:18</nova:creationTime>
      <nova:flavor name="m1.numa">
        <nova:memory>1024</nova:memory>
        <nova:disk>1</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>4</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="e2ab0e48d003456da53e892366651175">admin</nova:user>
        <nova:project uuid="a9a2cd5511214089a290ccfcac47502c">admin</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="178c675a-d5fb-459f-a850-f7ffa6e2c9d2"/>
    </nova:instance>
  </metadata>
[. . .]
  <cputune>
    <emulatorpin cpuset="0-3"/>
    <vcpupin vcpu="0" cpuset="0-3"/>
    <vcpupin vcpu="1" cpuset="0-3"/>
    <vcpupin vcpu="2" cpuset="0-3"/>
    <vcpupin vcpu="3" cpuset="0-3"/>
  </cputune>
[. . .]

--- Additional comment from Nikola Dipanov on 2014-11-27 12:10:46 EST ---

It might also be relevant that in this case, the domain XML will as well have a <numa> element specified.

--- Additional comment from Kashyap Chamarthy on 2014-11-28 07:58:58 EST ---

Contextual snippet of Nova guest XML:
[. . .]
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0-3'/>
    <vcpupin vcpu='1' cpuset='0-3'/>
    <vcpupin vcpu='2' cpuset='0-3'/>
    <vcpupin vcpu='3' cpuset='0-3'/>
    <emulatorpin cpuset='0-3'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
  </numatune>
[. . .]
  <cpu>
    <topology sockets='4' cores='1' threads='1'/>
    <numa>
      <cell id='0' cpus='0-3' memory='1048576'/>
    </numa>
  </cpu>
[. . .]


I was here in Nova's git when I tested this time:

  nova]$ git describe
  2014.2-995-g5d2ea10


Previously, I was at:

  nova]$ git describe
  2014.2-973-g922ca3c

--- Additional comment from Martin Kletzander on 2014-11-28 08:31:01 EST ---

Fixed upstream with v1.2.10-75-gc6e9024:

commit c6e90248676126c209b3b6017ad27cf6c6a0ab8f
Author: Wang Rui <moon.wangrui>
Date:   Mon Nov 10 21:53:19 2014 +0800

    qemu: fix domain startup failing with 'strict' mode in numatune
Comment 2 Kashyap Chamarthy 2014-11-28 14:13:24 UTC 

*** This bug has been marked as a duplicate of bug 1168866 ***