Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1190719

Summary: when using dedicated cpus, the emulator thread should be affined as well
Product: Red Hat OpenStack Reporter: RHOS Integration <rhos-integ>
Component: openstack-novaAssignee: Sahid Ferdjaoui <sferdjao>
Status: CLOSED ERRATA QA Contact: Sean Toner <stoner>
Severity: high Docs Contact:
Priority: high    
Version: 6.0 (Juno)CC: berrange, dasmith, eglynn, jshortt, kchamart, ndipanov, nlevinki, pbrady, sbauza, sclewis, sferdjao, sgordon, slong, stoner, vromanso, yeylon
Target Milestone: z2Keywords: ZStream
Target Release: 6.0 (Juno)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-nova-2014.2.2-17.el7ost Doc Type: Bug Fix
Doc Text:
Previously, emulator threads (for vCPUs) floated on the union of the set of all NUMA CPUs, even if the CPUs were dedicated, which meant that an emulator thread could consume CPU time from another guest instance. With this fix, emulator threads now only use the union of dedicated host CPUs, and that CPU's time, on which guest vCPUs are running.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2015-04-07 15:09:29 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 743661, 1038706, 1076956, 1198800    

Description RHOS Integration 2015-02-09 14:20:20 UTC 
Cloned from launchpad bug 1417671.

Description:

I'm running nova trunk, commit 752954a.

I configured a flavor with two vcpus and extra specs "hw:cpu_policy=dedicated" in order to enable vcpu pinning.

I booted up an instance with this flavor, and "virsh dumpxml" shows that the two vcpus were affined suitably to host cpus, but the emulator thread was left to float across the available host cores on that numa node.

  <cputune>
    <shares>2048</shares>
        <vcpupin vcpu='0' policy='other' priority='0' cpuset='4'/>
        <vcpupin vcpu='1' policy='other' priority='0' cpuset='5'/>
    <emulatorpin cpuset='3-11'/>
  </cputune>



Looking at the kvm process shortly after creation, we see quite a few emulator threads running with the emulatorpin affinity:

compute-2:~$ taskset -apc 136143
pid 136143's current affinity list: 3-11
pid 136144's current affinity list: 0,3-24,27-47
pid 136146's current affinity list: 4
pid 136147's current affinity list: 5
pid 136149's current affinity list: 0
pid 136433's current affinity list: 3-11
pid 136434's current affinity list: 3-11
pid 136435's current affinity list: 3-11
pid 136436's current affinity list: 3-11
pid 136437's current affinity list: 3-11
pid 136438's current affinity list: 3-11
pid 136439's current affinity list: 3-11
pid 136440's current affinity list: 3-11
pid 136441's current affinity list: 3-11
pid 136442's current affinity list: 3-11
pid 136443's current affinity list: 3-11
pid 136444's current affinity list: 3-11
pid 136445's current affinity list: 3-11
pid 136446's current affinity list: 3-11
pid 136447's current affinity list: 3-11
pid 136448's current affinity list: 3-11
pid 136449's current affinity list: 3-11
pid 136450's current affinity list: 3-11
pid 136451's current affinity list: 3-11
pid 136452's current affinity list: 3-11
pid 136453's current affinity list: 3-11
pid 136454's current affinity list: 3-11


Since the purpose of "hw:cpu_policy=dedicated" is to provide a dedicated host CPU for each guest CPU, the libvirt emulatorpin cpuset for a given guest should be set to one (or possibly more) of the CPUs specified for that guest.  Otherwise, any work done by the emulator threads could rob CPU time from another guest instance.

Personally I'd like to see the emulator thread affined the same as guest vCPU 0 (we use guest vCPU0 as a maintenance processor while doing the "real work" on the other vCPUs), but an argument could be made that it should be affined to the logical OR of all the guest vCPU cpusets.

Specification URL (additional information):

https://bugs.launchpad.net/nova/+bug/1417671
Comment 3 Daniel Berrangé 2015-02-09 14:25:04 UTC 
There is a mistake in the way emulator CPU set is calculated. It is using a union of all CPUs in host NUMA nodes that the guest is running on. It should only be using the union of host CPUs that the guest vCPUs are running on. A subtle but important distinction.
Comment 4 Stephen Gordon 2015-02-10 15:01:41 UTC 
*** Bug 1157736 has been marked as a duplicate of this bug. ***
Comment 5 Daniel Berrangé 2015-02-10 18:02:15 UTC 
In progress fix starting review upstream

https://review.openstack.org/#/c/154580/
Comment 9 Sean Toner 2015-03-17 15:08:40 UTC 
This looks good.  I also created a 2 vcpu flavor with a policy of dedicated, and looked at the domain xml:

  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <emulatorpin cpuset='0-1'/>
  </cputune>


I checked the cpu affinity, and it looks good now:

[root@rhos-compute-node-06 ~]# taskset -apc 21368
pid 21368's current affinity list: 0-11
pid 21372's current affinity list: 0-11
pid 21373's current affinity list: 0-11
pid 21374's current affinity list: 0-11
pid 21375's current affinity list: 0-11
pid 21376's current affinity list: 0-11
pid 17558's current affinity list: 0-11
pid 17567's current affinity list: 0-11
...  # all are affined to 0-11
Comment 13 errata-xmlrpc 2015-04-07 15:09:29 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0790.html