Bug 1168866 – "libvirtError: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000002.scope/cpuset.mems': Device or resource busy"

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1168866 - "libvirtError: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000002.scope/cpuset.mems': Device or resource busy"

Summary:	"libvirtError: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machin...

Status:	CLOSED ERRATA

Aliases:	None

Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	libvirt (Show other bugs)
Sub Component:
Version:	7.1
Hardware:	Unspecified Unspecified

Priority	unspecified Severity medium
Target Milestone:	rc
Target Release:	---
Assigned To:	Martin Kletzander
QA Contact:	Virtualization Bugs
Docs Contact:

URL:
Whiteboard:
Keywords:	Upstream

Duplicates:	1168944 (view as bug list)
Depends On:	1168672 1168944
Blocks:
	Show dependency tree / graph

Reported:	2014-11-28 04:41 EST by Jaroslav Suchanek
Modified:	2015-03-05 02:47 EST (History)
CC List:	21 users (show)

See Also:
Fixed In Version:	libvirt-1.2.8-10.el7
Doc Type:	Bug Fix
Doc Text:
Story Points:	---
Clone Of:	1168672
Environment:
Last Closed:	2015-03-05 02:47:47 EST
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHSA-2015:0323	normal	SHIPPED_LIVE	Low: libvirt security, bug fix, and enhancement update	2015-03-05 07:10:54 EST

Groups: None (edit)

Description Jaroslav Suchanek 2014-11-28 04:41:19 EST

+++ This bug was initially created as a clone of Bug #1168672 +++

Description of problem
----------------------

This occurs when you boot a Nova instance with NUMA topology.

[. . .]
libvirtError: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000002.scope/cpuset.mems': Device or resource busy
[. . .'

Version
-------

Versions of libvirt, QEMU, systemd in the OpenStack setup (running in a
DevStack VM):

  $ uname -r; rpm -q libvirt-daemon-kvm qemu-system-x86 systemd
  3.18.0-0.rc6.git0.1.fc22.x86_64
  libvirt-daemon-kvm-1.2.10-3.fc22.x86_64
  qemu-system-x86-2.2.0-0.1.rc1.fc22.x86_64
  systemd-216-11.fc21.x86_64


How reproducible: Atleast twice.


Steps to reproduce
------------------

This occurred in when booting a Nova instance in a DevStack (OpenStack
developer setup) environment. The DevStack machine is a KVM guest
hypervisor, and the Nova guest is a nested guest running on this.

It's a fairly involved test environment, details here:

http://docs-draft.openstack.org/18/131818/1/check/gate-nova-docs/2ddc418/doc/build/html/devref/testing/libvirt-numa.html#testing-basis-non-numa-usage
docs-draft.openstack.org/18/131818/1/check/gate-nova-docs/2ddc418/doc/build/html/devref/testing/libvirt-numa.html#testing-basis-non-numa-usage


Actual results
--------------

[. . .]
2014-11-26 20:17:28.722 ERROR nova.compute.manager [-] [instance: bcb53b78-452c-4695-b39d-754389cd3dd5] Instance failed to spawn
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5] Traceback (most recent call last):
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/home/kashyapc/src/cloud/nova/nova/compute
/manager.py", line 2247, in _build_resources
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     yield resources
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/home/kashyapc/src/cloud/nova/nova/compute
/manager.py", line 2117, in _build_and_run_instance
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     instance_type=instance_type)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/home/kashyapc/src/cloud/nova/nova/virt/li
bvirt/driver.py", line 2640, in spawn
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     block_device_info, disk_info=disk_info)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/home/kashyapc/src/cloud/nova/nova/virt/libvirt/driver.py", line 4500, in _create_domain_and_network
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     power_on=power_on)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/home/kashyapc/src/cloud/nova/nova/virt/libvirt/driver.py", line 4433, in _create_domain
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     LOG.error(err)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/usr/lib/python2.7/site-packages/oslo/utils/excutils.py", line 82, in __exit__
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     six.reraise(self.type_, self.value, self.tb)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/home/kashyapc/src/cloud/nova/nova/virt/libvirt/driver.py", line 4423, in _create_domain
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     domain.createWithFlags(launch_flags)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     rv = execute(f, *args, **kwargs)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     six.reraise(c, e, tb)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     rv = meth(*args, **kwargs)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1033, in createWithFlags
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5] libvirtError: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000002.scope/cpuset.mems': Device or resource busy
[. . .]


Expected results
----------------

Nova instance (libvirt nested guest) should boot successfully.


Additional info
---------------

[1] Inventory of available NUMA nodes on the physical host:

$ numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 4 8 12 16 20 24 28 32 36 40 44
node 0 size: 257954 MB
node 0 free: 248486 MB
node 1 cpus: 1 5 9 13 17 21 25 29 33 37 41 45
node 1 size: 258045 MB
node 1 free: 256470 MB
node 2 cpus: 2 6 10 14 18 22 26 30 34 38 42 46
node 2 size: 258045 MB
node 2 free: 256507 MB
node 3 cpus: 3 7 11 15 19 23 27 31 35 39 43 47
node 3 size: 258040 MB
node 3 free: 256457 MB
node distances:
node   0   1   2   3 
  0:  10  20  20  20 
  1:  20  10  20  20 
  2:  20  20  10  20 
  3:  20  20  20  10 

[2] From sytemd `journactl`:

$ sudo journalctl -u libvirtd -l -p err
[. . .]
Nov 26 09:19:49 devstack libvirtd[32697]: driver in virRegisterStorageDriver must not be NULL
Nov 26 09:19:49 devstack libvirtd[32697]: Failed module registration vboxStorageRegister
Nov 26 10:33:51 devstack libvirtd[32697]: End of file while reading data: Input/output error
Nov 26 11:12:36 devstack libvirtd[32697]: End of file while reading data: Input/output error
Nov 26 20:17:28 devstack libvirtd[32697]: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000002.scope/cpuset.mems': Device or resource busy
Nov 26 20:17:28 devstack libvirtd[32697]: error from service: TerminateMachine: No machine 'qemu-instance-00000002' known
[. . .]


[3] From an existing Nova instance (that was booted _without_ NUMA)

    $ find /sys/fs/cgroup/cpuset/machine.slice/machine-qemu*/ -name cpuset.mems 
    /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000001.scope/vcpu0/cpuset.mems
    /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000001.scope/cpuset.mems
    /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000001.scope/emulator/cpuset.mems
    
    $ cd /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\\x2dinstance\\x2d00000001.scope/
    $ cat cpuset.mems vcpu0/cpuset.mems emulator/cpuset.mems 
    0-2
    0-2
    0-2


[4] libvirt developer Martin Kletzander confirmed on IRC (#virt, OFTC) that
this is a bug.

--- Additional comment from Kashyap Chamarthy on 2014-11-27 10:16:02 EST ---

Contextual snippet related to cgroups from the attached libvirtd debug log:

[. . .]
2014-11-27 14:18:20.835+0000: 25475: debug : virCgroupSetValueStr:718 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000003.scope/cpuset.mems' to '0'
2014-11-27 14:18:20.836+0000: 25475: debug : virFileClose:99 : Closed fd 25
2014-11-27 14:18:20.836+0000: 25475: error : virCgroupSetValueStr:728 : Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000003.scope/cpuset.mems': Device or resource busy
2014-11-27 14:18:20.836+0000: 25475: debug : virFileClose:99 : Closed fd 24
[. . .]
2014-11-27 14:18:21.041+0000: 25475: debug : virDBusMessageIterEncode:640 : Popping iter=0x7fd43ef944d0
2014-11-27 14:18:21.042+0000: 25475: error : virDBusCall:1537 : error from service: TerminateMachine: No machine 'qemu-instance-00000003' known
2014-11-27 14:18:21.042+0000: 25475: debug : qemuRemoveCgroup:1222 : Failed to terminate cgroup for instance-00000003
2014-11-27 14:18:21.042+0000: 25475: debug : virObjectUnref:259 : OBJECT_UNREF: obj=0x7fd4300e6890
2014-11-27 14:18:21.042+0000: 25475: debug : virCgroupRemove:3331 : Removing cgroup /machine.slice/machine-qemu\x2dinstance\x2d00000003.scope
2014-11-27 14:18:21.042+0000: 25475: debug : virCgroupRemove:3352 : Removing cgroup /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2dinstance\x2d00000003.scope/ and all child cgroups
2014-11-27 14:18:21.042+0000: 25475: debug : virCgroupRemove:3352 : Removing cgroup /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2dinstance\x2d00000003.scope/ and all child cgroups
2014-11-27 14:18:21.042+0000: 25475: debug : virCgroupRemove:3352 : Removing cgroup /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000003.scope/ and all child cgroups
2014-11-27 14:18:21.042+0000: 25475: debug : virCgroupRemoveRecursively:3302 : Removing cgroup /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000003.scope//emulator
[. . .]

--- Additional comment from Kashyap Chamarthy on 2014-11-27 10:28:09 EST ---

DevStack machine's capabilities where the Nova Compute service is running and a libvirt instance (a nested guest) failed to started)

---------------------
DevStack>$ virsh capabilities
<capabilities>

  <host>
    <uuid>7dae2ee3-8950-9a41-9e24-a463a8563bbd</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>Nehalem</model>
      <vendor>Intel</vendor>
      <topology sockets='1' cores='8' threads='1'/>
      <feature name='rdtscp'/>
      <feature name='hypervisor'/>
      <feature name='x2apic'/>
      <feature name='vmx'/>
      <feature name='ss'/>
      <feature name='vme'/>
      <pages unit='KiB' size='4'/>
      <pages unit='KiB' size='2048'/>
    </cpu>
    <power_management>
      <suspend_mem/>
      <suspend_disk/>
      <suspend_hybrid/>
    </power_management>
    <migration_features>
      <live/>
      <uri_transports>
        <uri_transport>tcp</uri_transport>
        <uri_transport>rdma</uri_transport>
      </uri_transports>
    </migration_features>
    <topology>
      <cells num='3'>
        <cell id='0'>
          <memory unit='KiB'>3949740</memory>
          <pages unit='KiB' size='4'>987435</pages>
          <pages unit='KiB' size='2048'>0</pages>
          <distances>
            <sibling id='0' value='10'/>
            <sibling id='1' value='20'/>
            <sibling id='2' value='20'/>
          </distances>
          <cpus num='4'>
            <cpu id='0' socket_id='0' core_id='0' siblings='0'/>
            <cpu id='1' socket_id='1' core_id='0' siblings='1'/>
            <cpu id='2' socket_id='2' core_id='0' siblings='2'/>
            <cpu id='3' socket_id='3' core_id='0' siblings='3'/>
          </cpus>
        </cell>
        <cell id='1'>
          <memory unit='KiB'>2016864</memory>
          <pages unit='KiB' size='4'>504216</pages>
          <pages unit='KiB' size='2048'>0</pages>
          <distances>
            <sibling id='0' value='20'/>
            <sibling id='1' value='10'/>
            <sibling id='2' value='20'/>
          </distances>
          <cpus num='2'>
            <cpu id='4' socket_id='4' core_id='0' siblings='4'/>
            <cpu id='5' socket_id='5' core_id='0' siblings='5'/>
          </cpus>
        </cell>
        <cell id='2'>
          <memory unit='KiB'>2014304</memory>
          <pages unit='KiB' size='4'>503576</pages>
          <pages unit='KiB' size='2048'>0</pages>
          <distances>
            <sibling id='0' value='20'/>
            <sibling id='1' value='20'/>
            <sibling id='2' value='10'/>
          </distances>
          <cpus num='2'>
            <cpu id='6' socket_id='6' core_id='0' siblings='6'/>
            <cpu id='7' socket_id='7' core_id='0' siblings='7'/>
          </cpus>
        </cell>
      </cells>
    </topology>
    <secmodel>
      <model>selinux</model>
      <doi>0</doi>
      <baselabel type='kvm'>system_u:system_r:svirt_t:s0</baselabel>
      <baselabel type='qemu'>system_u:system_r:svirt_tcg_t:s0</baselabel>
    </secmodel>
    <secmodel>
      <model>dac</model>
      <doi>0</doi>
      <baselabel type='kvm'>+107:+107</baselabel>
      <baselabel type='qemu'>+107:+107</baselabel>
    </secmodel>
  </host>

  <guest>
    <os_type>hvm</os_type>
    <arch name='i686'>
      <wordsize>32</wordsize>
      <emulator>/usr/bin/qemu-system-i386</emulator>
      <machine canonical='pc-i440fx-2.2' maxCpus='255'>pc</machine>
      <machine maxCpus='255'>pc-0.12</machine>
      <machine maxCpus='255'>pc-1.3</machine>
      <machine maxCpus='255'>pc-q35-1.6</machine>
      <machine maxCpus='255'>pc-q35-1.5</machine>
      <machine maxCpus='255'>pc-i440fx-1.6</machine>
      <machine canonical='pc-q35-2.2' maxCpus='255'>q35</machine>
      <machine maxCpus='1'>xenpv</machine>
      <machine maxCpus='255'>pc-i440fx-1.7</machine>
      <machine maxCpus='255'>pc-q35-2.1</machine>
      <machine maxCpus='255'>pc-0.11</machine>
      <machine maxCpus='255'>pc-0.10</machine>
      <machine maxCpus='255'>pc-1.2</machine>
      <machine maxCpus='1'>isapc</machine>
      <machine maxCpus='255'>pc-q35-1.4</machine>
      <machine maxCpus='128'>xenfv</machine>
      <machine maxCpus='255'>pc-0.15</machine>
      <machine maxCpus='255'>pc-i440fx-1.5</machine>
      <machine maxCpus='255'>pc-0.14</machine>
      <machine maxCpus='255'>pc-q35-2.0</machine>
      <machine maxCpus='255'>pc-i440fx-1.4</machine>
      <machine maxCpus='255'>pc-1.1</machine>
      <machine maxCpus='255'>pc-q35-1.7</machine>
      <machine maxCpus='255'>pc-i440fx-2.1</machine>
      <machine maxCpus='255'>pc-1.0</machine>
      <machine maxCpus='255'>pc-i440fx-2.0</machine>
      <machine maxCpus='255'>pc-0.13</machine>
      <domain type='qemu'>
      </domain>
      <domain type='kvm'>
        <emulator>/usr/bin/qemu-kvm</emulator>
        <machine canonical='pc-i440fx-2.2' maxCpus='255'>pc</machine>
        <machine maxCpus='255'>pc-1.3</machine>
        <machine maxCpus='255'>pc-0.12</machine>
        <machine maxCpus='255'>pc-q35-1.6</machine>
        <machine maxCpus='255'>pc-q35-1.5</machine>
        <machine maxCpus='255'>pc-i440fx-1.6</machine>
        <machine canonical='pc-q35-2.2' maxCpus='255'>q35</machine>
        <machine maxCpus='255'>pc-i440fx-1.7</machine>
        <machine maxCpus='1'>xenpv</machine>
        <machine maxCpus='255'>pc-q35-2.1</machine>
        <machine maxCpus='255'>pc-0.11</machine>
        <machine maxCpus='255'>pc-0.10</machine>
        <machine maxCpus='255'>pc-1.2</machine>
        <machine maxCpus='1'>isapc</machine>
        <machine maxCpus='255'>pc-q35-1.4</machine>
        <machine maxCpus='128'>xenfv</machine>
        <machine maxCpus='255'>pc-0.15</machine>
        <machine maxCpus='255'>pc-i440fx-1.5</machine>
        <machine maxCpus='255'>pc-i440fx-1.4</machine>
        <machine maxCpus='255'>pc-q35-2.0</machine>
        <machine maxCpus='255'>pc-0.14</machine>
        <machine maxCpus='255'>pc-1.1</machine>
        <machine maxCpus='255'>pc-q35-1.7</machine>
        <machine maxCpus='255'>pc-i440fx-2.1</machine>
        <machine maxCpus='255'>pc-1.0</machine>
        <machine maxCpus='255'>pc-i440fx-2.0</machine>
        <machine maxCpus='255'>pc-0.13</machine>
      </domain>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <disksnapshot default='on' toggle='no'/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
      <pae/>
      <nonpae/>
    </features>
  </guest>

  <guest>
    <os_type>hvm</os_type>
    <arch name='x86_64'>
      <wordsize>64</wordsize>
      <emulator>/usr/bin/qemu-system-x86_64</emulator>
      <machine canonical='pc-i440fx-2.2' maxCpus='255'>pc</machine>
      <machine maxCpus='255'>pc-1.3</machine>
      <machine maxCpus='255'>pc-0.12</machine>
      <machine maxCpus='255'>pc-q35-1.6</machine>
      <machine maxCpus='255'>pc-q35-1.5</machine>
      <machine maxCpus='255'>pc-i440fx-1.6</machine>
      <machine canonical='pc-q35-2.2' maxCpus='255'>q35</machine>
      <machine maxCpus='255'>pc-i440fx-1.7</machine>
      <machine maxCpus='1'>xenpv</machine>
      <machine maxCpus='255'>pc-q35-2.1</machine>
      <machine maxCpus='255'>pc-0.11</machine>
      <machine maxCpus='255'>pc-0.10</machine>
      <machine maxCpus='255'>pc-1.2</machine>
      <machine maxCpus='1'>isapc</machine>
      <machine maxCpus='255'>pc-q35-1.4</machine>
      <machine maxCpus='128'>xenfv</machine>
      <machine maxCpus='255'>pc-0.15</machine>
      <machine maxCpus='255'>pc-i440fx-1.5</machine>
      <machine maxCpus='255'>pc-i440fx-1.4</machine>
      <machine maxCpus='255'>pc-q35-2.0</machine>
      <machine maxCpus='255'>pc-0.14</machine>
      <machine maxCpus='255'>pc-1.1</machine>
      <machine maxCpus='255'>pc-q35-1.7</machine>
      <machine maxCpus='255'>pc-i440fx-2.1</machine>
      <machine maxCpus='255'>pc-1.0</machine>
      <machine maxCpus='255'>pc-i440fx-2.0</machine>
      <machine maxCpus='255'>pc-0.13</machine>
      <domain type='qemu'>
      </domain>
      <domain type='kvm'>
        <emulator>/usr/bin/qemu-kvm</emulator>
        <machine canonical='pc-i440fx-2.2' maxCpus='255'>pc</machine>
        <machine maxCpus='255'>pc-1.3</machine>
        <machine maxCpus='255'>pc-0.12</machine>
        <machine maxCpus='255'>pc-q35-1.6</machine>
        <machine maxCpus='255'>pc-q35-1.5</machine>
        <machine maxCpus='255'>pc-i440fx-1.6</machine>
        <machine canonical='pc-q35-2.2' maxCpus='255'>q35</machine>
        <machine maxCpus='255'>pc-i440fx-1.7</machine>
        <machine maxCpus='1'>xenpv</machine>
        <machine maxCpus='255'>pc-q35-2.1</machine>
        <machine maxCpus='255'>pc-0.11</machine>
        <machine maxCpus='255'>pc-0.10</machine>
        <machine maxCpus='255'>pc-1.2</machine>
        <machine maxCpus='1'>isapc</machine>
        <machine maxCpus='255'>pc-q35-1.4</machine>
        <machine maxCpus='128'>xenfv</machine>
        <machine maxCpus='255'>pc-0.15</machine>
        <machine maxCpus='255'>pc-i440fx-1.5</machine>
        <machine maxCpus='255'>pc-i440fx-1.4</machine>
        <machine maxCpus='255'>pc-q35-2.0</machine>
        <machine maxCpus='255'>pc-0.14</machine>
        <machine maxCpus='255'>pc-1.1</machine>
        <machine maxCpus='255'>pc-q35-1.7</machine>
        <machine maxCpus='255'>pc-i440fx-2.1</machine>
        <machine maxCpus='255'>pc-1.0</machine>
        <machine maxCpus='255'>pc-i440fx-2.0</machine>
        <machine maxCpus='255'>pc-0.13</machine>
      </domain>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <disksnapshot default='on' toggle='no'/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
    </features>
  </guest>

</capabilities>
---------------------

--- Additional comment from Kashyap Chamarthy on 2014-11-27 10:29:07 EST ---



--- Additional comment from Kashyap Chamarthy on 2014-11-27 10:33:39 EST ---

Contextual snippet from the attachment:

[. . .]
  <memory>1048576</memory>
  <numatune>
    <memory mode="strict" nodeset="0"/>
    <memnode cellid="0" mode="strict" nodeset="0"/>
  </numatune>
  <vcpu>4</vcpu>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="2015.1"/>
      <nova:name>cirrvm3</nova:name>
      <nova:creationTime>2014-11-27 14:18:18</nova:creationTime>
      <nova:flavor name="m1.numa">
        <nova:memory>1024</nova:memory>
        <nova:disk>1</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>4</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="e2ab0e48d003456da53e892366651175">admin</nova:user>
        <nova:project uuid="a9a2cd5511214089a290ccfcac47502c">admin</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="178c675a-d5fb-459f-a850-f7ffa6e2c9d2"/>
    </nova:instance>
  </metadata>
[. . .]
  <cputune>
    <emulatorpin cpuset="0-3"/>
    <vcpupin vcpu="0" cpuset="0-3"/>
    <vcpupin vcpu="1" cpuset="0-3"/>
    <vcpupin vcpu="2" cpuset="0-3"/>
    <vcpupin vcpu="3" cpuset="0-3"/>
  </cputune>
[. . .]

--- Additional comment from Nikola Dipanov on 2014-11-27 12:10:46 EST ---

It might also be relevant that in this case, the domain XML will as well have a <numa> element specified.

Comment 1 Martin Kletzander 2014-11-28 08:32:42 EST

Fixed upstream with v1.2.10-75-gc6e9024:

commit c6e90248676126c209b3b6017ad27cf6c6a0ab8f
Author: Wang Rui <moon.wangrui@huawei.com>
Date:   Mon Nov 10 21:53:19 2014 +0800

    qemu: fix domain startup failing with 'strict' mode in numatune

Comment 3 Kashyap Chamarthy 2014-11-28 09:13:24 EST

*** Bug 1168944 has been marked as a duplicate of this bug. ***

Comment 5 Jincheng Miao 2014-12-10 03:01:24 EST

*** Bug 1118517 has been marked as a duplicate of this bug. ***

Comment 6 Jincheng Miao 2014-12-11 03:16:35 EST

Hi Kashyap，

I am libvirt qe, and trying to reproduce this bug on RHEL7.
Could you meet this problem on RHEL7? Is nested KVM necessary for this bug?

Comment 7 Kashyap Chamarthy 2014-12-11 04:14:45 EST

(In reply to Jincheng Miao from comment #6)
> Hi Kashyap，

Hi Jincheng,

> 
> I am libvirt qe, and trying to reproduce this bug on RHEL7.
> Could you meet this problem on RHEL7? 

I didn't test it on RHEL7, but it should be reproducible without this fix on RHEL7, too, when you try to boot a libvirt guest with a single NUMA cell requested.

I hit this bug while booting libvirt guest in the context of OpenStack in a developer environment (DevStack) on Fedora-21, you can see how I tested it here (it's a bit involved procedure):

  https://bugzilla.redhat.com/show_bug.cgi?id=1168672#c8

> Is nested KVM necessary for this bug?

Nested KVM is not mandatory (but my test environment involved nested KVM). However, you do need a physical machine with more than a single NUMA cell. You can find that out by running: 

  $ virsh nodeinfo


You can probably try to reproduce this issue by trying to boot a libvirt guest with a single NUMA node. You can see the QEMU CLI from the parent bug, here:

  https://bugzilla.redhat.com/show_bug.cgi?id=1168672#c9

Specifically, notice this from the QEMU CLI:

[. . .]
-object memory-backend-ram,size=1024M,id=ram-node0,host-nodes=0,policy=bind -numa node,nodeid=0,cpus=0-3,memdev=ram-node0
[. . .]

Comment 8 Jincheng Miao 2014-12-11 05:06:53 EST

Hi Martin and Kashyap,

I tested it on a machine with 2 nodes, and configured 1 guest node.

...
  <vcpu placement='auto'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0-3'/>
    <vcpupin vcpu='1' cpuset='0-3'/>
    <vcpupin vcpu='2' cpuset='0-3'/>
    <vcpupin vcpu='3' cpuset='0-3'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
  </numatune>
...
  <cpu mode='host-passthrough'>
    <numa>
      <cell id='0' cpus='0-3' memory='1048576'/>
    </numa>
  </cpu>
...


But I got 
"
error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2db.scope/vcpu0/cpuset.cpus': Permission denied
"

It is not 'cpuset.mems: Device or resource busy'. 

Does it same with this bug?

Comment 9 Martin Kletzander 2014-12-11 05:46:42 EST

No, permission denied is something completely different.  I don't even know how this is possible :)

Comment 10 Jincheng Miao 2014-12-15 03:32:33 EST

(In reply to Martin Kletzander from comment #9)
> No, permission denied is something completely different.  I don't even know
> how this is possible :)

OK, I already filed a bug 1174125 for "cpuset.cpus: Permission denied" problem.

Comment 11 Jincheng Miao 2014-12-17 04:33:49 EST

This bug is cloned from fedora, so it couldn't be reproduced on RHEL.

But I have one method to reproduce it, using package libvirt-1.2.8-10.el7.

Rebuild libvirt-1.2.8-10.el7 with 411cea6 and without c6e9024


commit 411cea638f6ec8503b7142a31e58b1cd85dbeaba
Author: Zhou yimin <zhouyimin@huawei.com>
Date:   Thu Oct 16 22:18:48 2014 +0800

    qemu: move setting emulatorpin ahead of monitor showing up

commit c6e90248676126c209b3b6017ad27cf6c6a0ab8f
Author: Wang Rui <moon.wangrui@huawei.com>
Date:   Mon Nov 10 21:53:19 2014 +0800

    qemu: fix domain startup failing with 'strict' mode in numatune


After that, I could reproduce this problem:
# virsh edit b
...
  <vcpu placement='auto'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='1-3'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
  </numatune>
...
  <cpu mode='host-passthrough'>
    <numa>
      <cell id='0' cpus='0-3' memory='1048576'/>
    </numa>
  </cpu>
...

# virsh start b
error: Failed to start domain b
error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2db.scope/cpuset.mems': Device or resource busy


As commit c6e9024 said:
"
It's broken by Commit 411cea638f6ec8503b7142a31e58b1cd85dbeaba
which moved qemuSetupCgroupForEmulator() before setting cpuset.mems
in qemuSetupCgroupPostInit.

Directory '$cgroup_path/emulator/' is created in qemuSetupCgroupForEmulator.
But '$cgroup_path/emulator/cpuset.mems' it not set and has a default value
(all nodes, such as 0-1). Then we setup '$cgroup_path/cpuset.mems' to the
nodemask (in this case it's '0') in qemuSetupCgroupPostInit. It must fail.
"

Comment 12 Jincheng Miao 2014-12-17 04:42:44 EST

The problem "cpuset.cpus: Permission denied" is due to the inconsistent between numad and vcpupin user specified. I will post log in bug 1174125.

Comment 13 Jincheng Miao 2014-12-17 05:06:08 EST

So this commit from upstream should be add:

commit 411cea638f6ec8503b7142a31e58b1cd85dbeaba
Author: Zhou yimin <zhouyimin@huawei.com>
Date:   Thu Oct 16 22:18:48 2014 +0800

    qemu: move setting emulatorpin ahead of monitor showing up

Comment 14 Jiri Denemark 2014-12-17 05:09:36 EST

Yeah, it will be added for bug 1170484.

Comment 15 Jincheng Miao 2014-12-17 22:18:22 EST

Since this upstream commit 411cea6 is backported in latest libvirt-1.2.8-11.el7.x86_64, I will use it to test this bug.

# rpm -q libvirt
libvirt-1.2.8-11.el7.x86_64

1. using the same guest configuration:
# virsh edit b
...
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='1-3'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
  </numatune>
...
  <cpu mode='host-passthrough'>
    <numa>
      <cell id='0' cpus='0-3' memory='1048576'/>
    </numa>
  </cpu>
...

2. start guest
# virsh start b
Domain b started

Using vcpu placement='auto' also triggers "cpuset.cpus: Permission denied", but bug 1174125 will cover it.

So I will change the status to VERIFIED.

Comment 17 errata-xmlrpc 2015-03-05 02:47:47 EST

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0323.html

Note You need to log in before you can comment on or make changes to this bug.

agedosier
berrange
clalancette
dkelson
dyuan
extras-qa
honzhang
itamar
jdenemar
jforbes
jmiao
kchamart
laine
libvirt-maint
mkletzan
mzhan
ndipanov
rbalakri
shyu
veillard
virt-maint