1168866 – "libvirtError: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000002.scope/cpuset.mems': Device or resource busy"

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1168866 - "libvirtError: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000002.scope/cpuset.mems': Device or resource busy"

Summary: "libvirtError: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machin...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	libvirt
Sub Component:
Version:	7.1
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	medium
Target Milestone:	rc
Target Release:	---
Assignee:	Martin Kletzander
QA Contact:	Virtualization Bugs
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	1168944 (view as bug list)
Depends On:	1168672 1168944
Blocks:
TreeView+	depends on / blocked

Reported:	2014-11-28 09:41 UTC by Jaroslav Suchanek
Modified:	2015-03-05 07:47 UTC (History)
CC List:	21 users (show)
Fixed In Version:	libvirt-1.2.8-10.el7
Doc Type:	Bug Fix
Doc Text:
Clone Of:	1168672
Environment:
Last Closed:	2015-03-05 07:47:47 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHSA-2015:0323	0	normal	SHIPPED_LIVE	Low: libvirt security, bug fix, and enhancement update	2015-03-05 12:10:54 UTC

Description Jaroslav Suchanek 2014-11-28 09:41:19 UTC

+++ This bug was initially created as a clone of Bug #1168672 +++

Description of problem
----------------------

This occurs when you boot a Nova instance with NUMA topology.

[. . .]
libvirtError: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000002.scope/cpuset.mems': Device or resource busy
[. . .'

Version
-------

Versions of libvirt, QEMU, systemd in the OpenStack setup (running in a
DevStack VM):

  $ uname -r; rpm -q libvirt-daemon-kvm qemu-system-x86 systemd
  3.18.0-0.rc6.git0.1.fc22.x86_64
  libvirt-daemon-kvm-1.2.10-3.fc22.x86_64
  qemu-system-x86-2.2.0-0.1.rc1.fc22.x86_64
  systemd-216-11.fc21.x86_64


How reproducible: Atleast twice.


Steps to reproduce
------------------

This occurred in when booting a Nova instance in a DevStack (OpenStack
developer setup) environment. The DevStack machine is a KVM guest
hypervisor, and the Nova guest is a nested guest running on this.

It's a fairly involved test environment, details here:

http://docs-draft.openstack.org/18/131818/1/check/gate-nova-docs/2ddc418/doc/build/html/devref/testing/libvirt-numa.html#testing-basis-non-numa-usage
docs-draft.openstack.org/18/131818/1/check/gate-nova-docs/2ddc418/doc/build/html/devref/testing/libvirt-numa.html#testing-basis-non-numa-usage


Actual results
--------------

[. . .]
2014-11-26 20:17:28.722 ERROR nova.compute.manager [-] [instance: bcb53b78-452c-4695-b39d-754389cd3dd5] Instance failed to spawn
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5] Traceback (most recent call last):
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/home/kashyapc/src/cloud/nova/nova/compute
/manager.py", line 2247, in _build_resources
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     yield resources
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/home/kashyapc/src/cloud/nova/nova/compute
/manager.py", line 2117, in _build_and_run_instance
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     instance_type=instance_type)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/home/kashyapc/src/cloud/nova/nova/virt/li
bvirt/driver.py", line 2640, in spawn
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     block_device_info, disk_info=disk_info)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/home/kashyapc/src/cloud/nova/nova/virt/libvirt/driver.py", line 4500, in _create_domain_and_network
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     power_on=power_on)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/home/kashyapc/src/cloud/nova/nova/virt/libvirt/driver.py", line 4433, in _create_domain
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     LOG.error(err)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/usr/lib/python2.7/site-packages/oslo/utils/excutils.py", line 82, in __exit__
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     six.reraise(self.type_, self.value, self.tb)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/home/kashyapc/src/cloud/nova/nova/virt/libvirt/driver.py", line 4423, in _create_domain
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     domain.createWithFlags(launch_flags)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     rv = execute(f, *args, **kwargs)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     six.reraise(c, e, tb)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     rv = meth(*args, **kwargs)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1033, in createWithFlags
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2014-11-26 20:17:28.722 TRACE nova.compute.manager [instance: bcb53b78-452c-4695-b39d-754389cd3dd5] libvirtError: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000002.scope/cpuset.mems': Device or resource busy
[. . .]


Expected results
----------------

Nova instance (libvirt nested guest) should boot successfully.


Additional info
---------------

[1] Inventory of available NUMA nodes on the physical host:

$ numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 4 8 12 16 20 24 28 32 36 40 44
node 0 size: 257954 MB
node 0 free: 248486 MB
node 1 cpus: 1 5 9 13 17 21 25 29 33 37 41 45
node 1 size: 258045 MB
node 1 free: 256470 MB
node 2 cpus: 2 6 10 14 18 22 26 30 34 38 42 46
node 2 size: 258045 MB
node 2 free: 256507 MB
node 3 cpus: 3 7 11 15 19 23 27 31 35 39 43 47
node 3 size: 258040 MB
node 3 free: 256457 MB
node distances:
node   0   1   2   3 
  0:  10  20  20  20 
  1:  20  10  20  20 
  2:  20  20  10  20 
  3:  20  20  20  10 

[2] From sytemd `journactl`:

$ sudo journalctl -u libvirtd -l -p err
[. . .]
Nov 26 09:19:49 devstack libvirtd[32697]: driver in virRegisterStorageDriver must not be NULL
Nov 26 09:19:49 devstack libvirtd[32697]: Failed module registration vboxStorageRegister
Nov 26 10:33:51 devstack libvirtd[32697]: End of file while reading data: Input/output error
Nov 26 11:12:36 devstack libvirtd[32697]: End of file while reading data: Input/output error
Nov 26 20:17:28 devstack libvirtd[32697]: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000002.scope/cpuset.mems': Device or resource busy
Nov 26 20:17:28 devstack libvirtd[32697]: error from service: TerminateMachine: No machine 'qemu-instance-00000002' known
[. . .]


[3] From an existing Nova instance (that was booted _without_ NUMA)

    $ find /sys/fs/cgroup/cpuset/machine.slice/machine-qemu*/ -name cpuset.mems 
    /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000001.scope/vcpu0/cpuset.mems
    /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000001.scope/cpuset.mems
    /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000001.scope/emulator/cpuset.mems
    
    $ cd /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\\x2dinstance\\x2d00000001.scope/
    $ cat cpuset.mems vcpu0/cpuset.mems emulator/cpuset.mems 
    0-2
    0-2
    0-2


[4] libvirt developer Martin Kletzander confirmed on IRC (#virt, OFTC) that
this is a bug.

--- Additional comment from Kashyap Chamarthy on 2014-11-27 10:16:02 EST ---

Contextual snippet related to cgroups from the attached libvirtd debug log:

[. . .]
2014-11-27 14:18:20.835+0000: 25475: debug : virCgroupSetValueStr:718 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000003.scope/cpuset.mems' to '0'
2014-11-27 14:18:20.836+0000: 25475: debug : virFileClose:99 : Closed fd 25
2014-11-27 14:18:20.836+0000: 25475: error : virCgroupSetValueStr:728 : Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000003.scope/cpuset.mems': Device or resource busy
2014-11-27 14:18:20.836+0000: 25475: debug : virFileClose:99 : Closed fd 24
[. . .]
2014-11-27 14:18:21.041+0000: 25475: debug : virDBusMessageIterEncode:640 : Popping iter=0x7fd43ef944d0
2014-11-27 14:18:21.042+0000: 25475: error : virDBusCall:1537 : error from service: TerminateMachine: No machine 'qemu-instance-00000003' known
2014-11-27 14:18:21.042+0000: 25475: debug : qemuRemoveCgroup:1222 : Failed to terminate cgroup for instance-00000003
2014-11-27 14:18:21.042+0000: 25475: debug : virObjectUnref:259 : OBJECT_UNREF: obj=0x7fd4300e6890
2014-11-27 14:18:21.042+0000: 25475: debug : virCgroupRemove:3331 : Removing cgroup /machine.slice/machine-qemu\x2dinstance\x2d00000003.scope
2014-11-27 14:18:21.042+0000: 25475: debug : virCgroupRemove:3352 : Removing cgroup /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2dinstance\x2d00000003.scope/ and all child cgroups
2014-11-27 14:18:21.042+0000: 25475: debug : virCgroupRemove:3352 : Removing cgroup /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2dinstance\x2d00000003.scope/ and all child cgroups
2014-11-27 14:18:21.042+0000: 25475: debug : virCgroupRemove:3352 : Removing cgroup /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000003.scope/ and all child cgroups
2014-11-27 14:18:21.042+0000: 25475: debug : virCgroupRemoveRecursively:3302 : Removing cgroup /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dinstance\x2d00000003.scope//emulator
[. . .]

--- Additional comment from Kashyap Chamarthy on 2014-11-27 10:28:09 EST ---

DevStack machine's capabilities where the Nova Compute service is running and a libvirt instance (a nested guest) failed to started)

---------------------
DevStack>$ virsh capabilities
<capabilities>

  <host>
    <uuid>7dae2ee3-8950-9a41-9e24-a463a8563bbd</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>Nehalem</model>
      <vendor>Intel</vendor>
      <topology sockets='1' cores='8' threads='1'/>
      <feature name='rdtscp'/>
      <feature name='hypervisor'/>
      <feature name='x2apic'/>
      <feature name='vmx'/>
      <feature name='ss'/>
      <feature name='vme'/>
      <pages unit='KiB' size='4'/>
      <pages unit='KiB' size='2048'/>
    </cpu>
    <power_management>
      <suspend_mem/>
      <suspend_disk/>
      <suspend_hybrid/>
    </power_management>
    <migration_features>
      <live/>
      <uri_transports>
        <uri_transport>tcp</uri_transport>
        <uri_transport>rdma</uri_transport>
      </uri_transports>
    </migration_features>
    <topology>
      <cells num='3'>
        <cell id='0'>
          <memory unit='KiB'>3949740</memory>
          <pages unit='KiB' size='4'>987435</pages>
          <pages unit='KiB' size='2048'>0</pages>
          <distances>
            <sibling id='0' value='10'/>
            <sibling id='1' value='20'/>
            <sibling id='2' value='20'/>
          </distances>
          <cpus num='4'>
            <cpu id='0' socket_id='0' core_id='0' siblings='0'/>
            <cpu id='1' socket_id='1' core_id='0' siblings='1'/>
            <cpu id='2' socket_id='2' core_id='0' siblings='2'/>
            <cpu id='3' socket_id='3' core_id='0' siblings='3'/>
          </cpus>
        </cell>
        <cell id='1'>
          <memory unit='KiB'>2016864</memory>
          <pages unit='KiB' size='4'>504216</pages>
          <pages unit='KiB' size='2048'>0</pages>
          <distances>
            <sibling id='0' value='20'/>
            <sibling id='1' value='10'/>
            <sibling id='2' value='20'/>
          </distances>
          <cpus num='2'>
            <cpu id='4' socket_id='4' core_id='0' siblings='4'/>
            <cpu id='5' socket_id='5' core_id='0' siblings='5'/>
          </cpus>
        </cell>
        <cell id='2'>
          <memory unit='KiB'>2014304</memory>
          <pages unit='KiB' size='4'>503576</pages>
          <pages unit='KiB' size='2048'>0</pages>
          <distances>
            <sibling id='0' value='20'/>
            <sibling id='1' value='20'/>
            <sibling id='2' value='10'/>
          </distances>
          <cpus num='2'>
            <cpu id='6' socket_id='6' core_id='0' siblings='6'/>
            <cpu id='7' socket_id='7' core_id='0' siblings='7'/>
          </cpus>
        </cell>
      </cells>
    </topology>
    <secmodel>
      <model>selinux</model>
      <doi>0</doi>
      <baselabel type='kvm'>system_u:system_r:svirt_t:s0</baselabel>
      <baselabel type='qemu'>system_u:system_r:svirt_tcg_t:s0</baselabel>
    </secmodel>
    <secmodel>
      <model>dac</model>
      <doi>0</doi>
      <baselabel type='kvm'>+107:+107</baselabel>
      <baselabel type='qemu'>+107:+107</baselabel>
    </secmodel>
  </host>

  <guest>
    <os_type>hvm</os_type>
    <arch name='i686'>
      <wordsize>32</wordsize>
      <emulator>/usr/bin/qemu-system-i386</emulator>
      <machine canonical='pc-i440fx-2.2' maxCpus='255'>pc</machine>
      <machine maxCpus='255'>pc-0.12</machine>
      <machine maxCpus='255'>pc-1.3</machine>
      <machine maxCpus='255'>pc-q35-1.6</machine>
      <machine maxCpus='255'>pc-q35-1.5</machine>
      <machine maxCpus='255'>pc-i440fx-1.6</machine>
      <machine canonical='pc-q35-2.2' maxCpus='255'>q35</machine>
      <machine maxCpus='1'>xenpv</machine>
      <machine maxCpus='255'>pc-i440fx-1.7</machine>
      <machine maxCpus='255'>pc-q35-2.1</machine>
      <machine maxCpus='255'>pc-0.11</machine>
      <machine maxCpus='255'>pc-0.10</machine>
      <machine maxCpus='255'>pc-1.2</machine>
      <machine maxCpus='1'>isapc</machine>
      <machine maxCpus='255'>pc-q35-1.4</machine>
      <machine maxCpus='128'>xenfv</machine>
      <machine maxCpus='255'>pc-0.15</machine>
      <machine maxCpus='255'>pc-i440fx-1.5</machine>
      <machine maxCpus='255'>pc-0.14</machine>
      <machine maxCpus='255'>pc-q35-2.0</machine>
      <machine maxCpus='255'>pc-i440fx-1.4</machine>
      <machine maxCpus='255'>pc-1.1</machine>
      <machine maxCpus='255'>pc-q35-1.7</machine>
      <machine maxCpus='255'>pc-i440fx-2.1</machine>
      <machine maxCpus='255'>pc-1.0</machine>
      <machine maxCpus='255'>pc-i440fx-2.0</machine>
      <machine maxCpus='255'>pc-0.13</machine>
      <domain type='qemu'>
      </domain>
      <domain type='kvm'>
        <emulator>/usr/bin/qemu-kvm</emulator>
        <machine canonical='pc-i440fx-2.2' maxCpus='255'>pc</machine>
        <machine maxCpus='255'>pc-1.3</machine>
        <machine maxCpus='255'>pc-0.12</machine>
        <machine maxCpus='255'>pc-q35-1.6</machine>
        <machine maxCpus='255'>pc-q35-1.5</machine>
        <machine maxCpus='255'>pc-i440fx-1.6</machine>
        <machine canonical='pc-q35-2.2' maxCpus='255'>q35</machine>
        <machine maxCpus='255'>pc-i440fx-1.7</machine>
        <machine maxCpus='1'>xenpv</machine>
        <machine maxCpus='255'>pc-q35-2.1</machine>
        <machine maxCpus='255'>pc-0.11</machine>
        <machine maxCpus='255'>pc-0.10</machine>
        <machine maxCpus='255'>pc-1.2</machine>
        <machine maxCpus='1'>isapc</machine>
        <machine maxCpus='255'>pc-q35-1.4</machine>
        <machine maxCpus='128'>xenfv</machine>
        <machine maxCpus='255'>pc-0.15</machine>
        <machine maxCpus='255'>pc-i440fx-1.5</machine>
        <machine maxCpus='255'>pc-i440fx-1.4</machine>
        <machine maxCpus='255'>pc-q35-2.0</machine>
        <machine maxCpus='255'>pc-0.14</machine>
        <machine maxCpus='255'>pc-1.1</machine>
        <machine maxCpus='255'>pc-q35-1.7</machine>
        <machine maxCpus='255'>pc-i440fx-2.1</machine>
        <machine maxCpus='255'>pc-1.0</machine>
        <machine maxCpus='255'>pc-i440fx-2.0</machine>
        <machine maxCpus='255'>pc-0.13</machine>
      </domain>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <disksnapshot default='on' toggle='no'/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
      <pae/>
      <nonpae/>
    </features>
  </guest>

  <guest>
    <os_type>hvm</os_type>
    <arch name='x86_64'>
      <wordsize>64</wordsize>
      <emulator>/usr/bin/qemu-system-x86_64</emulator>
      <machine canonical='pc-i440fx-2.2' maxCpus='255'>pc</machine>
      <machine maxCpus='255'>pc-1.3</machine>
      <machine maxCpus='255'>pc-0.12</machine>
      <machine maxCpus='255'>pc-q35-1.6</machine>
      <machine maxCpus='255'>pc-q35-1.5</machine>
      <machine maxCpus='255'>pc-i440fx-1.6</machine>
      <machine canonical='pc-q35-2.2' maxCpus='255'>q35</machine>
      <machine maxCpus='255'>pc-i440fx-1.7</machine>
      <machine maxCpus='1'>xenpv</machine>
      <machine maxCpus='255'>pc-q35-2.1</machine>
      <machine maxCpus='255'>pc-0.11</machine>
      <machine maxCpus='255'>pc-0.10</machine>
      <machine maxCpus='255'>pc-1.2</machine>
      <machine maxCpus='1'>isapc</machine>
      <machine maxCpus='255'>pc-q35-1.4</machine>
      <machine maxCpus='128'>xenfv</machine>
      <machine maxCpus='255'>pc-0.15</machine>
      <machine maxCpus='255'>pc-i440fx-1.5</machine>
      <machine maxCpus='255'>pc-i440fx-1.4</machine>
      <machine maxCpus='255'>pc-q35-2.0</machine>
      <machine maxCpus='255'>pc-0.14</machine>
      <machine maxCpus='255'>pc-1.1</machine>
      <machine maxCpus='255'>pc-q35-1.7</machine>
      <machine maxCpus='255'>pc-i440fx-2.1</machine>
      <machine maxCpus='255'>pc-1.0</machine>
      <machine maxCpus='255'>pc-i440fx-2.0</machine>
      <machine maxCpus='255'>pc-0.13</machine>
      <domain type='qemu'>
      </domain>
      <domain type='kvm'>
        <emulator>/usr/bin/qemu-kvm</emulator>
        <machine canonical='pc-i440fx-2.2' maxCpus='255'>pc</machine>
        <machine maxCpus='255'>pc-1.3</machine>
        <machine maxCpus='255'>pc-0.12</machine>
        <machine maxCpus='255'>pc-q35-1.6</machine>
        <machine maxCpus='255'>pc-q35-1.5</machine>
        <machine maxCpus='255'>pc-i440fx-1.6</machine>
        <machine canonical='pc-q35-2.2' maxCpus='255'>q35</machine>
        <machine maxCpus='255'>pc-i440fx-1.7</machine>
        <machine maxCpus='1'>xenpv</machine>
        <machine maxCpus='255'>pc-q35-2.1</machine>
        <machine maxCpus='255'>pc-0.11</machine>
        <machine maxCpus='255'>pc-0.10</machine>
        <machine maxCpus='255'>pc-1.2</machine>
        <machine maxCpus='1'>isapc</machine>
        <machine maxCpus='255'>pc-q35-1.4</machine>
        <machine maxCpus='128'>xenfv</machine>
        <machine maxCpus='255'>pc-0.15</machine>
        <machine maxCpus='255'>pc-i440fx-1.5</machine>
        <machine maxCpus='255'>pc-i440fx-1.4</machine>
        <machine maxCpus='255'>pc-q35-2.0</machine>
        <machine maxCpus='255'>pc-0.14</machine>
        <machine maxCpus='255'>pc-1.1</machine>
        <machine maxCpus='255'>pc-q35-1.7</machine>
        <machine maxCpus='255'>pc-i440fx-2.1</machine>
        <machine maxCpus='255'>pc-1.0</machine>
        <machine maxCpus='255'>pc-i440fx-2.0</machine>
        <machine maxCpus='255'>pc-0.13</machine>
      </domain>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <disksnapshot default='on' toggle='no'/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
    </features>
  </guest>

</capabilities>
---------------------

--- Additional comment from Kashyap Chamarthy on 2014-11-27 10:29:07 EST ---



--- Additional comment from Kashyap Chamarthy on 2014-11-27 10:33:39 EST ---

Contextual snippet from the attachment:

[. . .]
  <memory>1048576</memory>
  <numatune>
    <memory mode="strict" nodeset="0"/>
    <memnode cellid="0" mode="strict" nodeset="0"/>
  </numatune>
  <vcpu>4</vcpu>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="2015.1"/>
      <nova:name>cirrvm3</nova:name>
      <nova:creationTime>2014-11-27 14:18:18</nova:creationTime>
      <nova:flavor name="m1.numa">
        <nova:memory>1024</nova:memory>
        <nova:disk>1</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>4</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="e2ab0e48d003456da53e892366651175">admin</nova:user>
        <nova:project uuid="a9a2cd5511214089a290ccfcac47502c">admin</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="178c675a-d5fb-459f-a850-f7ffa6e2c9d2"/>
    </nova:instance>
  </metadata>
[. . .]
  <cputune>
    <emulatorpin cpuset="0-3"/>
    <vcpupin vcpu="0" cpuset="0-3"/>
    <vcpupin vcpu="1" cpuset="0-3"/>
    <vcpupin vcpu="2" cpuset="0-3"/>
    <vcpupin vcpu="3" cpuset="0-3"/>
  </cputune>
[. . .]

--- Additional comment from Nikola Dipanov on 2014-11-27 12:10:46 EST ---

It might also be relevant that in this case, the domain XML will as well have a <numa> element specified.

Comment 1 Martin Kletzander 2014-11-28 13:32:42 UTC

Fixed upstream with v1.2.10-75-gc6e9024:

commit c6e90248676126c209b3b6017ad27cf6c6a0ab8f
Author: Wang Rui <moon.wangrui>
Date:   Mon Nov 10 21:53:19 2014 +0800

    qemu: fix domain startup failing with 'strict' mode in numatune

Comment 3 Kashyap Chamarthy 2014-11-28 14:13:24 UTC

*** Bug 1168944 has been marked as a duplicate of this bug. ***

Comment 5 Jincheng Miao 2014-12-10 08:01:24 UTC

*** Bug 1118517 has been marked as a duplicate of this bug. ***

Comment 6 Jincheng Miao 2014-12-11 08:16:35 UTC

Hi Kashyap，

I am libvirt qe, and trying to reproduce this bug on RHEL7.
Could you meet this problem on RHEL7? Is nested KVM necessary for this bug?

Comment 7 Kashyap Chamarthy 2014-12-11 09:14:45 UTC

(In reply to Jincheng Miao from comment #6)
> Hi Kashyap，

Hi Jincheng,

> 
> I am libvirt qe, and trying to reproduce this bug on RHEL7.
> Could you meet this problem on RHEL7? 

I didn't test it on RHEL7, but it should be reproducible without this fix on RHEL7, too, when you try to boot a libvirt guest with a single NUMA cell requested.

I hit this bug while booting libvirt guest in the context of OpenStack in a developer environment (DevStack) on Fedora-21, you can see how I tested it here (it's a bit involved procedure):

  https://bugzilla.redhat.com/show_bug.cgi?id=1168672#c8

> Is nested KVM necessary for this bug?

Nested KVM is not mandatory (but my test environment involved nested KVM). However, you do need a physical machine with more than a single NUMA cell. You can find that out by running: 

  $ virsh nodeinfo


You can probably try to reproduce this issue by trying to boot a libvirt guest with a single NUMA node. You can see the QEMU CLI from the parent bug, here:

  https://bugzilla.redhat.com/show_bug.cgi?id=1168672#c9

Specifically, notice this from the QEMU CLI:

[. . .]
-object memory-backend-ram,size=1024M,id=ram-node0,host-nodes=0,policy=bind -numa node,nodeid=0,cpus=0-3,memdev=ram-node0
[. . .]

Comment 8 Jincheng Miao 2014-12-11 10:06:53 UTC

Hi Martin and Kashyap,

I tested it on a machine with 2 nodes, and configured 1 guest node.

...
  <vcpu placement='auto'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0-3'/>
    <vcpupin vcpu='1' cpuset='0-3'/>
    <vcpupin vcpu='2' cpuset='0-3'/>
    <vcpupin vcpu='3' cpuset='0-3'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
  </numatune>
...
  <cpu mode='host-passthrough'>
    <numa>
      <cell id='0' cpus='0-3' memory='1048576'/>
    </numa>
  </cpu>
...


But I got 
"
error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2db.scope/vcpu0/cpuset.cpus': Permission denied
"

It is not 'cpuset.mems: Device or resource busy'. 

Does it same with this bug?

Comment 9 Martin Kletzander 2014-12-11 10:46:42 UTC

No, permission denied is something completely different.  I don't even know how this is possible :)

Comment 10 Jincheng Miao 2014-12-15 08:32:33 UTC

(In reply to Martin Kletzander from comment #9)
> No, permission denied is something completely different.  I don't even know
> how this is possible :)

OK, I already filed a bug 1174125 for "cpuset.cpus: Permission denied" problem.

Comment 11 Jincheng Miao 2014-12-17 09:33:49 UTC

This bug is cloned from fedora, so it couldn't be reproduced on RHEL.

But I have one method to reproduce it, using package libvirt-1.2.8-10.el7.

Rebuild libvirt-1.2.8-10.el7 with 411cea6 and without c6e9024


commit 411cea638f6ec8503b7142a31e58b1cd85dbeaba
Author: Zhou yimin <zhouyimin>
Date:   Thu Oct 16 22:18:48 2014 +0800

    qemu: move setting emulatorpin ahead of monitor showing up

commit c6e90248676126c209b3b6017ad27cf6c6a0ab8f
Author: Wang Rui <moon.wangrui>
Date:   Mon Nov 10 21:53:19 2014 +0800

    qemu: fix domain startup failing with 'strict' mode in numatune


After that, I could reproduce this problem:
# virsh edit b
...
  <vcpu placement='auto'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='1-3'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
  </numatune>
...
  <cpu mode='host-passthrough'>
    <numa>
      <cell id='0' cpus='0-3' memory='1048576'/>
    </numa>
  </cpu>
...

# virsh start b
error: Failed to start domain b
error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2db.scope/cpuset.mems': Device or resource busy


As commit c6e9024 said:
"
It's broken by Commit 411cea638f6ec8503b7142a31e58b1cd85dbeaba
which moved qemuSetupCgroupForEmulator() before setting cpuset.mems
in qemuSetupCgroupPostInit.

Directory '$cgroup_path/emulator/' is created in qemuSetupCgroupForEmulator.
But '$cgroup_path/emulator/cpuset.mems' it not set and has a default value
(all nodes, such as 0-1). Then we setup '$cgroup_path/cpuset.mems' to the
nodemask (in this case it's '0') in qemuSetupCgroupPostInit. It must fail.
"

Comment 12 Jincheng Miao 2014-12-17 09:42:44 UTC

The problem "cpuset.cpus: Permission denied" is due to the inconsistent between numad and vcpupin user specified. I will post log in bug 1174125.

Comment 13 Jincheng Miao 2014-12-17 10:06:08 UTC

So this commit from upstream should be add:

commit 411cea638f6ec8503b7142a31e58b1cd85dbeaba
Author: Zhou yimin <zhouyimin>
Date:   Thu Oct 16 22:18:48 2014 +0800

    qemu: move setting emulatorpin ahead of monitor showing up

Comment 14 Jiri Denemark 2014-12-17 10:09:36 UTC

Yeah, it will be added for bug 1170484.

Comment 15 Jincheng Miao 2014-12-18 03:18:22 UTC

Since this upstream commit 411cea6 is backported in latest libvirt-1.2.8-11.el7.x86_64, I will use it to test this bug.

# rpm -q libvirt
libvirt-1.2.8-11.el7.x86_64

1. using the same guest configuration:
# virsh edit b
...
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='1-3'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
  </numatune>
...
  <cpu mode='host-passthrough'>
    <numa>
      <cell id='0' cpus='0-3' memory='1048576'/>
    </numa>
  </cpu>
...

2. start guest
# virsh start b
Domain b started

Using vcpu placement='auto' also triggers "cpuset.cpus: Permission denied", but bug 1174125 will cover it.

So I will change the status to VERIFIED.

Comment 17 errata-xmlrpc 2015-03-05 07:47:47 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0323.html

Note You need to log in before you can comment on or make changes to this bug.

agedosier
berrange
clalancette
dkelson
dyuan
extras-qa
honzhang
itamar
jdenemar
jforbes
jmiao
kchamart
laine
libvirt-maint
mkletzan
mzhan
ndipanov
rbalakri
shyu
veillard
virt-maint