Bug 1298825

Summary:	Odd number of vCPUs breaks 'prefer' threads policy
Product:	Red Hat OpenStack	Reporter:	Michal Pryc <mpryc>
Component:	openstack-nova	Assignee:	Michal Pryc <mpryc>
Status:	CLOSED ERRATA	QA Contact:	Prasanth Anbalagan <panbalag>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	8.0 (Liberty)	CC:	berrange, dasmith, eglynn, jschluet, kchamart, lyarwood, mlopes, ndipanov, panbalag, sbauza, sferdjao, sgordon, vromanso, yeylon
Target Milestone:	ga
Target Release:	8.0 (Liberty)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-nova-12.0.1-2.el7ost	Doc Type:	Bug Fix
Doc Text:	Previously, selecting an odd number of vCPUs would cause the assignment of one core and one thread in the guest instance per CPU, which would impact performance. The update addresses this issue by correctly assigning pairs of threads and one independent thread per CPU, when an odd number of vCPUs is assigned.	Story Points:	---
Clone Of:
Clones:	1298829 (view as bug list)		Environment:
Last Closed:	2016-04-07 21:23:12 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1298829

Description Michal Pryc 2016-01-15 08:16:24 UTC

Description of problem:

Using a CPU policy of dedicated ('hw:cpu_policy=dedicated') results in vCPUs being pinned to pCPUs, per the original blueprint:

    http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/virt-driver-cpu-pinning.html

When scheduling instance with this extra spec there appears to be an implicit use of the 'prefer' threads policy, i.e. where possible vCPUs are pinned to thread siblings first. This is "implicit" because the threads policy aspect of this spec has not yet been implemented.

However, this implicit 'prefer' policy breaks when a VM with an odd number of vCPUs is booted. This has been seen on a Hyper-Threading-enabled host where "sibling sets" are two long, but it would presumably happen on any host where the number of siblings (or any number between this value and one) is not an factor of the number of vCPUs (i.e. vCPUs % n != 0, for siblings <= n > 0).

It is reasonable to assume that a three vCPU VM, for example, should try best effort and use siblings for at the first two vCPUs of the VM (assuming you're on a host system with HyperThreading and sibling sets are of length two). This would give us a true best effort implementation.


Version-Release number of selected component (if applicable):


How reproducible:
Always


Steps to Reproduce:

# Testing Configuration

Testing was conducted on a single-node, Fedora 21-based (3.17.8-300.fc21.x86_64) OpenStack instance (built with devstack). The system is a dual-socket, 10 core, HT-enabled system (2 sockets * 10 cores * 2 threads = 40 "pCPUs". 0-9,20-29 = node0, 10-19,30-39 = node1).

1.Create two flavors, one with odd vcpu number and second with even:

    openstack flavor create --ram 4096 --disk 20 --vcpus 3 demo.odd
    nova flavor-key demo.odd set hw:cpu_policy=dedicated

    openstack flavor create --ram 4096 --disk 20 --vcpus 4 demo.even
    nova flavor-key demo.even set hw:cpu_policy=dedicated

2.Dump xml using:

    virsh dumpxml [ID]

Actual results:

    Output looks like:

    <cputune>
        <shares>3072</shares>
        <vcpupin vcpu='0' cpuset='1'/>
        <vcpupin vcpu='1' cpuset='0'/>
        <vcpupin vcpu='2' cpuset='25'/>
        <emulatorpin cpuset='0-1,25'/>
    </cputune>


Expected results:

    Should be similar to:

   <cputune>
        <shares>3072</shares>
        <vcpupin vcpu='0' cpuset='0'/>
        <vcpupin vcpu='1' cpuset='20'/>
        <vcpupin vcpu='2' cpuset='1'/>
        <emulatorpin cpuset='0-1,20'/>
    </cputune>


Additional info:

The output from 'virsh dumpxml [ID]' for the four vCPU VM is given below. Similar results can be seen for varying "even" numbers of vCPUs (2, 4, 10 tested):

    <cputune>
        <shares>4096</shares>
        <vcpupin vcpu='0' cpuset='3'/>
        <vcpupin vcpu='1' cpuset='23'/>
        <vcpupin vcpu='2' cpuset='26'/>
        <vcpupin vcpu='3' cpuset='6'/>
        <emulatorpin cpuset='3,6,23,26'/>
    </cputune>

Comment 4 Prasanth Anbalagan 2016-02-02 15:05:56 UTC

Verified as follows,

************
Version
************

[root@seal17 ~(keystone_admin)]# yum list installed | grep openstack-nova
openstack-nova-api.noarch            1:12.0.1-2.el7ost       @RH7-RHOS-8.0      
openstack-nova-cert.noarch           1:12.0.1-2.el7ost       @RH7-RHOS-8.0      
openstack-nova-common.noarch         1:12.0.1-2.el7ost       @RH7-RHOS-8.0      
openstack-nova-compute.noarch        1:12.0.1-2.el7ost       @RH7-RHOS-8.0      
openstack-nova-conductor.noarch      1:12.0.1-2.el7ost       @RH7-RHOS-8.0      
openstack-nova-console.noarch        1:12.0.1-2.el7ost       @RH7-RHOS-8.0      
openstack-nova-novncproxy.noarch     1:12.0.1-2.el7ost       @RH7-RHOS-8.0      
openstack-nova-scheduler.noarch      1:12.0.1-2.el7ost       @RH7-RHOS-8.0      
[root@seal17 ~(keystone_admin)]# 

***********
Logs
***********

[root@seal17 nova]# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23
node 0 size: 32664 MB
node 0 free: 24241 MB
node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31
node 1 size: 32768 MB
node 1 free: 27703 MB
node distances:
node   0   1 
  0:  10  21 
  1:  21  10 
[root@seal17 nova]# 


[root@seal17 ~(keystone_admin)]# nova flavor-create m1.odd 6 2048 20 3
+----+--------+-----------+------+-----------+------+-------+-------------+-----------+
| ID | Name   | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+----+--------+-----------+------+-----------+------+-------+-------------+-----------+
| 6  | m1.odd | 2048      | 20   | 0         |      | 3     | 1.0         | True      |
+----+--------+-----------+------+-----------+------+-------+-------------+-----------+
[root@seal17 ~(keystone_admin)]# nova flavor-key 6 set hw:cpu_policy=dedicated
[root@seal17 ~(keystone_admin)]# 
[root@seal17 ~(keystone_admin)]# 
[root@seal17 ~(keystone_admin)]# nova flavor-create m1.odd 7 2048 20 4
ERROR (Conflict): Flavor with name m1.odd already exists. (HTTP 409) (Request-ID: req-b26e658b-247a-4757-bcd7-ba8b767172b4)
[root@seal17 ~(keystone_admin)]# 
[root@seal17 ~(keystone_admin)]# 
[root@seal17 ~(keystone_admin)]# nova flavor-create m1.even 7 2048 20 4
+----+---------+-----------+------+-----------+------+-------+-------------+-----------+
| ID | Name    | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+----+---------+-----------+------+-----------+------+-------+-------------+-----------+
| 7  | m1.even | 2048      | 20   | 0         |      | 4     | 1.0         | True      |
+----+---------+-----------+------+-----------+------+-------+-------------+-----------+
[root@seal17 ~(keystone_admin)]# 
[root@seal17 ~(keystone_admin)]# nova flavor-key 7 set hw:cpu_policy=dedicated
[root@seal17 ~(keystone_admin)]# 
[root@seal17 ~(keystone_admin)]# 
[root@seal17 ~(keystone_admin)]# nova list
+--------------------------------------+------+--------+------------+-------------+---------------------+
| ID                                   | Name | Status | Task State | Power State | Networks            |
+--------------------------------------+------+--------+------------+-------------+---------------------+
| b83970c4-2c9b-47c0-b6da-9814ffb6dcb7 | vm1  | ACTIVE | -          | Running     | public=172.24.4.227 |
+--------------------------------------+------+--------+------------+-------------+---------------------+
[root@seal17 ~(keystone_admin)]# nova delete vm1
Request to delete server vm1 has been accepted.
[root@seal17 ~(keystone_admin)]# 
[root@seal17 ~(keystone_admin)]# nova boot --image cirros --flavor m1.odd vm1.odd
+--------------------------------------+-----------------------------------------------+
| Property                             | Value                                         |
+--------------------------------------+-----------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                        |
| OS-EXT-AZ:availability_zone          |                                               |
| OS-EXT-SRV-ATTR:host                 | -                                             |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                             |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000002                             |
| OS-EXT-STS:power_state               | 0                                             |
| OS-EXT-STS:task_state                | scheduling                                    |
| OS-EXT-STS:vm_state                  | building                                      |
| OS-SRV-USG:launched_at               | -                                             |
| OS-SRV-USG:terminated_at             | -                                             |
| accessIPv4                           |                                               |
| accessIPv6                           |                                               |
| adminPass                            | gyTdm5VncqiM                                  |
| config_drive                         |                                               |
| created                              | 2016-02-02T14:57:17Z                          |
| flavor                               | m1.odd (6)                                    |
| hostId                               |                                               |
| id                                   | 5926294d-715c-4310-b488-26c57ffd1762          |
| image                                | cirros (7238ec00-3913-4228-a113-a97162ca479f) |
| key_name                             | -                                             |
| metadata                             | {}                                            |
| name                                 | vm1.odd                                       |
| os-extended-volumes:volumes_attached | []                                            |
| progress                             | 0                                             |
| security_groups                      | default                                       |
| status                               | BUILD                                         |
| tenant_id                            | 8897dfddf6df49529f87ab966a124794              |
| updated                              | 2016-02-02T14:57:18Z                          |
| user_id                              | 7771ec536e494880ba0a993f77f9bac9              |
+--------------------------------------+-----------------------------------------------+
[root@seal17 ~(keystone_admin)]# nova list
+--------------------------------------+---------+--------+------------+-------------+---------------------+
| ID                                   | Name    | Status | Task State | Power State | Networks            |
+--------------------------------------+---------+--------+------------+-------------+---------------------+
| 5926294d-715c-4310-b488-26c57ffd1762 | vm1.odd | ACTIVE | -          | Running     | public=172.24.4.228 |
+--------------------------------------+---------+--------+------------+-------------+---------------------+
[root@seal17 ~(keystone_admin)]# 

[root@seal17 ~(keystone_admin)]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 3     instance-00000002              running

[root@seal17 ~(keystone_admin)]# 
[root@seal17 ~(keystone_admin)]# 
[root@seal17 ~(keystone_admin)]# virsh dumpxml 3
..
...
<vcpu placement='static'>3</vcpu>
  <cputune>
    <shares>3072</shares>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='18'/>
    <vcpupin vcpu='2' cpuset='1'/>
    <emulatorpin cpuset='1-2,18'/>
  </cputune>


[root@seal17 ~(keystone_admin)]# nova boot --image cirros --flavor m1.even vm1.even
+--------------------------------------+-----------------------------------------------+
| Property                             | Value                                         |
+--------------------------------------+-----------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                        |
| OS-EXT-AZ:availability_zone          |                                               |
| OS-EXT-SRV-ATTR:host                 | -                                             |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                             |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000003                             |
| OS-EXT-STS:power_state               | 0                                             |
| OS-EXT-STS:task_state                | scheduling                                    |
| OS-EXT-STS:vm_state                  | building                                      |
| OS-SRV-USG:launched_at               | -                                             |
| OS-SRV-USG:terminated_at             | -                                             |
| accessIPv4                           |                                               |
| accessIPv6                           |                                               |
| adminPass                            | VQrTrtW4u53q                                  |
| config_drive                         |                                               |
| created                              | 2016-02-02T15:00:07Z                          |
| flavor                               | m1.even (7)                                   |
| hostId                               |                                               |
| id                                   | 257a142a-ff00-4636-8c70-ecf626f26712          |
| image                                | cirros (7238ec00-3913-4228-a113-a97162ca479f) |
| key_name                             | -                                             |
| metadata                             | {}                                            |
| name                                 | vm1.even                                      |
| os-extended-volumes:volumes_attached | []                                            |
| progress                             | 0                                             |
| security_groups                      | default                                       |
| status                               | BUILD                                         |
| tenant_id                            | 8897dfddf6df49529f87ab966a124794              |
| updated                              | 2016-02-02T15:00:07Z                          |
| user_id                              | 7771ec536e494880ba0a993f77f9bac9              |
+--------------------------------------+-----------------------------------------------+
[root@seal17 ~(keystone_admin)]# nova list
+--------------------------------------+----------+--------+------------+-------------+---------------------+
| ID                                   | Name     | Status | Task State | Power State | Networks            |
+--------------------------------------+----------+--------+------------+-------------+---------------------+
| 257a142a-ff00-4636-8c70-ecf626f26712 | vm1.even | ACTIVE | -          | Running     | public=172.24.4.229 |
| 5926294d-715c-4310-b488-26c57ffd1762 | vm1.odd  | ACTIVE | -          | Running     | public=172.24.4.228 |
+--------------------------------------+----------+--------+------------+-------------+---------------------+
[root@seal17 ~(keystone_admin)]# 

[root@seal17 ~(keystone_admin)]# 
[root@seal17 ~(keystone_admin)]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 3     instance-00000002              running
 4     instance-00000003              running


[root@seal17 ~(keystone_admin)]# virsh dumpxml 4

    <shares>4096</shares>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='16'/>
    <vcpupin vcpu='2' cpuset='3'/>
    <vcpupin vcpu='3' cpuset='19'/>
    <emulatorpin cpuset='0,3,16,19'/>

Comment 5 errata-xmlrpc 2016-04-07 21:23:12 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0603.html