Bug 1656068 - RFE[Backport OSP13]: Dedicate pCPU for emulator thread placement per host rather than per guest.
Summary: RFE[Backport OSP13]: Dedicate pCPU for emulator thread placement per host rat...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z6
: 13.0 (Queens)
Assignee: Artom Lifshitz
QA Contact: nova-maint
URL:
Whiteboard:
: 1606610 (view as bug list)
Depends On: 1468004
Blocks: 1341176 1591229 1691740
TreeView+ depends on / blocked
 
Reported: 2018-12-04 15:26 UTC by atelang
Modified: 2019-09-09 15:47 UTC (History)
28 users (show)

Fixed In Version: openstack-nova-17.0.9-4.el7ost
Doc Type: Enhancement
Doc Text:
Clone Of: 1468004
Environment:
Last Closed: 2019-04-30 17:13:59 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
OpenStack gerrit 511188 None MERGED update: isolate guests emulthreads on CONF.cpu_shared_set 2020-10-14 05:59:08 UTC
Red Hat Product Errata RHBA-2019:0924 None None None 2019-04-30 17:14:13 UTC

Comment 7 Artom Lifshitz 2019-03-26 14:46:21 UTC
*** Bug 1606610 has been marked as a duplicate of this bug. ***

Comment 9 Artom Lifshitz 2019-04-02 13:36:13 UTC
Test plan:

This is the same RFE as bz 1468004, so the test plan from that (if it exists) can be reused. In addition, automated test are currently proposed here: [1].

The automated tests cover 3 scenarios (and assume a `dedicated` CPU policy and `vcpu_pin_set` configured):

1. With emulator threads policy set to `share` and `cpu_share_set` set, emulator threads should be pinned to `cpu_share_set`.
2. With emulator threads policy set to `share` and `cpu_share_set` unset, emulator threads should float over (aka be pinned to) the instance's pCPUs.
3. With emulator threads policy set to `isolate`, `cpu_shared_set` is ignored, and emulator threads should be pinned to a pCPU distinct from the instance's pCPUs.

If we want to be more thorough, here's a truth table of emulator threads policy, cpu_share_set, and expected behaviour:

                           +--------------------------------+---------------------------------------+                          
                           |       cpu_share_set set        |           cpu_share_set unset         |
+--------------------------+--------------------------------+---------------------------------------+
| share (implicit default) | Pinned to cpu_share_set        | Pinned to all of the instance's pCPUs |
+--------------------------+--------------------------------+---------------------------------------+
| share (explicitly set)   | Pinned to cpu_share_set        | Pinned to all of the instance's pCPUs |
+--------------------------+--------------------------------+---------------------------------------+
|                  isolate | Pinned to a single pCPU        | Pinned to a single pCPU               |
|                          | distinct from instance's pCPUs | distinct from instance's pCPUs        |
+--------------------------+--------------------------------+---------------------------------------+

[1] https://review.rdoproject.org/r/#/c/18070/

Comment 11 Artom Lifshitz 2019-04-02 14:12:06 UTC
Correction to the table above - the default policy isn't an implicit share, but just unset. In those cases, regardless of the value of cpu_share_set, the emulator threads float over (aka are pinned to) all of the instance's pCPUs.

                         +--------------------------------+---------------------------------------+
                         |       cpu_share_set set        |           cpu_share_set unset         |
+------------------------+--------------------------------+---------------------------------------+
|        Unset (default) | Pinned to all of the           | Pinned to all of the instance's pCPUs |
|                        | instance's pCPUs               |                                       |
+------------------------+--------------------------------+---------------------------------------+
| share (explicitly set) | Pinned to cpu_share_set        | Pinned to all of the instance's pCPUs |
+------------------------+--------------------------------+---------------------------------------+
|                isolate | Pinned to a single pCPU        | Pinned to a single pCPU               |
|                        | distinct from instance's pCPUs | distinct from instance's pCPUs        |
+------------------------+--------------------------------+---------------------------------------+

Comment 12 Roger Heslop 2019-04-04 21:16:47 UTC
Per cfields I'd like to take a table like this and include it into documentation. I've reworded some of the fields for clarity, thoughts on accuracy?

                          +--------------------------------------+----------------------------------------+
                          |                                      |                                        |
                          |  cpu_share_set: [CPU list defined]   |  cpu_share_set: [CPU list not defined] |
|---------------------------------------------------------------------------------------------------------+
|                         |                                      |                                        |
|  share: unset (default) |  Floats across vcpus in vcpu_pin_set |  Floats across vcpus in vcpu_pin_set   |
+---------------------------------------------------------------------------------------------------------+
|                         |                                      |                                        |
|  share: set             |  Pinned to cpu_share_set list        |  Floats across vcpus in vcpu_pin_set   |
+----------------------------------------------------------------+----------------------------------------+
|                         |                                                                               |
|  isolate                |  Isolated to one CPU in NovaVcpuPinSet. Requires an extra CPU per instance.   |
+-------------------------+-------------------------------------------------------------------------------+

Comment 13 Artom Lifshitz 2019-04-05 11:57:50 UTC
Not quite.

For the first two cells in the left-most columns, it's not 'share: unset/set', it's the flavor extra spec that's either unset or set to 'share'. 

Also, 'Floats across vcpus in vcpu_pin_set' isn't totally correct. It's really the *instance's* CPUs, which are a subset of vcpu_pin_set.

Comment 14 Roger Heslop 2019-04-05 12:24:43 UTC
Thank Artom, that clarifies vcpu_pin_set for me. I've cleaned up the muddied wording on the left hand column as well. 

                            +--------------------------------------+----------------------------------------+
                            |                                      |                                        |
                            |  cpu_share_set: [CPU list defined]   |  cpu_share_set: [CPU list not defined] |
|---------------------------------------------------------------------------------------------------------+
| hw:emulator_thread_policy |                                      |                                        |
|        [unset]            |  Floats across instances vcpus       |  Floats across instances vcpus         |
+-----------------------------------------------------------------------------------------------------------+
| hw:emulator_thread_policy |                                      |                                        |
|         share             |  Pinned to cpu_share_set list        |  Floats across instances vcpus         |
+------------------------------------------------------------------+----------------------------------------+
| hw:emulator_thread_policy |                                                                               |
|        isolate            |  Isolated to one CPU in NovaVcpuPinSet. Requires an extra CPU per instance.   |
+---------------------------+-------------------------------------------------------------------------------+

Comment 15 smooney 2019-04-05 13:17:22 UTC
the use of floats and pinned is inconsistent

you shoudl say that it si "pinned to the instance vcpus" or it "floats across the cpu_share_set"

in both cases we restict the enulator thread to a set of allowed cpus it just a different set.

so either we should use the term floats in all cases or pinned.

also i would remove the term list for this entirly.

the cpu_share_set and vcpu_pin_set are mathmatical sets expresed in a sting encoded form
 [CPU list defined] makes this less clear

i would use "cpu_share_set: [defined]" and cpu_share_set: [undefined]
if more clarity is needed.


the other thing to be aware of is the semantic above are implemention defined and are only vlaid in the context
of the libvirt virt driver. they may change in the future and have changed in backwards incompatible ways in the past.
as such i think this is good to add to the docs but it needs to be revauated depending on the osp version.


+-----------------------------------------------------------------------------------------------------------+
| hw:emulator_thread_policy |                                      |                                        |
|         share             |  Pinned to cpu_share_set list        |  Floats across instances vcpus         |
+------------------------------------------------------------------+----------------------------------------+

is an example of one incompatible change we did when this feature was intoduced.

the meaning of hw:emulator_thread_policy=share was changed by this feature. so on older version of osp it where
cpu_share_set is not a thing hw:emulator_thread_policy=share always means float across instance vcpus.

Comment 16 Roger Heslop 2019-04-05 14:03:27 UTC
This is good feedback, though some of the wording chosen was deliberate. As this is becoming more of a documentation discussion, I'll move this conversation to email. But feel free to copy anyone on it who might be interested.

Comment 22 errata-xmlrpc 2019-04-30 17:13:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0924


Note You need to log in before you can comment on or make changes to this bug.