Bug 1468004

I don't think it's something we should do or at least we should wait before to start working on isolating pCPU(s) to run all emulator threads of guests.

There are two usecases: DPDK and RT.

- For Realtime they want the emulator threads to run on additional CPUs. That's because of a potential non-RT vCPU holding a kernel lock which could have negative impact on the RT vCPUs.
- For DPDK that requirement is not necessary since it's running in user space. So they just want a solution to isolate the emulator threads on a specific vCPU.

In the ML my point was to configure a mask which can be set to the host configuration or flavor extra-specs.

On nova.conf the mask will be applied to the set of CPUs dedicated for Nova (vcpu_pin_set) and basically isolate them to run emulator threads of guests only. On flavor extra specs the mask will isolate the emulator threads on specific vCPUs.

But the KVM-RT team is working on resolving the problem for Realtime so at the end it will not be necessary to have additional CPUs to isolate the emulator threads.

So basically the only change we should make is for DPDK usage. We have to provide the same kind of mask we already have for realtime (hw:cpu_realtime_mask) but to only isolate the emulator threads.

  hw:cpu_emulator_threads_mask=^0

So the emulator threads are going to run on the vCPU0.

OK Sahid, are you planning to work on this with the upstream folks for Queens?

Yes I already updated the spec and asked Franck to confirm whether that is OK.

  https://review.openstack.org/#/c/486617/1/specs/queens/approved/libvirt-emulator-threads-policy.rst

The patches are waiting for review:

   https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/libvirt-emulator-threads-policy

Stephen, is that OK for you if we rename this BZ to something like ?

  RFE: add mask on the guest vCPUs to place emulator threads of the guest on the host related CPUs

(I'm sure you would find something much better :)

(In reply to Sahid Ferdjaoui from comment #5)
> The patches are waiting for review:
> 
>   
> https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:
> master+topic:bp/libvirt-emulator-threads-policy
> 
> Stephen, is that OK for you if we rename this BZ to something like ?
> 
>   RFE: add mask on the guest vCPUs to place emulator threads of the guest on
> the host related CPUs
> 
> (I'm sure you would find something much better :)

Since [0] is closed as WONTFIX, we do not have other solution than provide a config option to isolate the emulator threads of all guests on a set pCPUs

[0] https://bugzilla.redhat.com/show_bug.cgi?id=1468217

(In reply to Sahid Ferdjaoui from comment #5)
> The patches are waiting for review:
> 
>   
> https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:
> master+topic:bp/libvirt-emulator-threads-policy

Hi Sahid,

I notice these changes were recently abandoned. What is the current plan of record based on your discussions with the various other folks familiar with RT?

Thanks,

Steve

(In reply to Stephen Gordon from comment #8)
> (In reply to Sahid Ferdjaoui from comment #5)
> > The patches are waiting for review:
> > 
> >   
> > https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:
> > master+topic:bp/libvirt-emulator-threads-policy
> 
> Hi Sahid,
> 
> I notice these changes were recently abandoned. What is the current plan of
> record based on your discussions with the various other folks familiar with
> RT?
> 
> Thanks,
> 
> Steve

I pushed BP and code for introducing an option 'overhead_pin_set' That one will be used to isolate a set of host CPUs where we should be able to pin the emulator threads on for libvirt driver.

  https://review.openstack.org/#/c/510897/

I need to update the spec already merged to refer that new option but first I have to discussed with community to ensure that they are agree.

http://lists.openstack.org/pipermail/openstack-dev/2017-October/123432.html

Patches sent upstream:

  https://review.openstack.org/#/c/510897/

Spec re-proposed for Rocky:

  https://review.openstack.org/#/c/511188/

Patches merged upstream:

  https://review.openstack.org/#/q/status:merged+project:openstack/nova+branch:master+topic:bp/overhead-pin-set

Inputs for NFV documentation: https://access.redhat.com/solutions/3384881

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045