Description of problem: Using a CPU policy of dedicated ('hw:cpu_policy=dedicated') results in vCPUs being pinned to pCPUs, per the original blueprint: http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/virt-driver-cpu-pinning.html When scheduling instance with this extra spec there appears to be an implicit use of the 'prefer' threads policy, i.e. where possible vCPUs are pinned to thread siblings first. This is "implicit" because the threads policy aspect of this spec has not yet been implemented. However, this implicit 'prefer' policy breaks when a VM with an odd number of vCPUs is booted. This has been seen on a Hyper-Threading-enabled host where "sibling sets" are two long, but it would presumably happen on any host where the number of siblings (or any number between this value and one) is not an factor of the number of vCPUs (i.e. vCPUs % n != 0, for siblings <= n > 0). It is reasonable to assume that a three vCPU VM, for example, should try best effort and use siblings for at the first two vCPUs of the VM (assuming you're on a host system with HyperThreading and sibling sets are of length two). This would give us a true best effort implementation. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: # Testing Configuration Testing was conducted on a single-node, Fedora 21-based (3.17.8-300.fc21.x86_64) OpenStack instance (built with devstack). The system is a dual-socket, 10 core, HT-enabled system (2 sockets * 10 cores * 2 threads = 40 "pCPUs". 0-9,20-29 = node0, 10-19,30-39 = node1). 1.Create two flavors, one with odd vcpu number and second with even: openstack flavor create --ram 4096 --disk 20 --vcpus 3 demo.odd nova flavor-key demo.odd set hw:cpu_policy=dedicated openstack flavor create --ram 4096 --disk 20 --vcpus 4 demo.even nova flavor-key demo.even set hw:cpu_policy=dedicated 2.Dump xml using: virsh dumpxml [ID] Actual results: Output looks like: <cputune> <shares>3072</shares> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='0'/> <vcpupin vcpu='2' cpuset='25'/> <emulatorpin cpuset='0-1,25'/> </cputune> Expected results: Should be similar to: <cputune> <shares>3072</shares> <vcpupin vcpu='0' cpuset='0'/> <vcpupin vcpu='1' cpuset='20'/> <vcpupin vcpu='2' cpuset='1'/> <emulatorpin cpuset='0-1,20'/> </cputune> Additional info: The output from 'virsh dumpxml [ID]' for the four vCPU VM is given below. Similar results can be seen for varying "even" numbers of vCPUs (2, 4, 10 tested): <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='3'/> <vcpupin vcpu='1' cpuset='23'/> <vcpupin vcpu='2' cpuset='26'/> <vcpupin vcpu='3' cpuset='6'/> <emulatorpin cpuset='3,6,23,26'/> </cputune>
Verified as follows, ************ Version ************ [root@seal17 ~(keystone_admin)]# yum list installed | grep openstack-nova openstack-nova-api.noarch 1:12.0.1-2.el7ost @RH7-RHOS-8.0 openstack-nova-cert.noarch 1:12.0.1-2.el7ost @RH7-RHOS-8.0 openstack-nova-common.noarch 1:12.0.1-2.el7ost @RH7-RHOS-8.0 openstack-nova-compute.noarch 1:12.0.1-2.el7ost @RH7-RHOS-8.0 openstack-nova-conductor.noarch 1:12.0.1-2.el7ost @RH7-RHOS-8.0 openstack-nova-console.noarch 1:12.0.1-2.el7ost @RH7-RHOS-8.0 openstack-nova-novncproxy.noarch 1:12.0.1-2.el7ost @RH7-RHOS-8.0 openstack-nova-scheduler.noarch 1:12.0.1-2.el7ost @RH7-RHOS-8.0 [root@seal17 ~(keystone_admin)]# *********** Logs *********** [root@seal17 nova]# numactl -H available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23 node 0 size: 32664 MB node 0 free: 24241 MB node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31 node 1 size: 32768 MB node 1 free: 27703 MB node distances: node 0 1 0: 10 21 1: 21 10 [root@seal17 nova]# [root@seal17 ~(keystone_admin)]# nova flavor-create m1.odd 6 2048 20 3 +----+--------+-----------+------+-----------+------+-------+-------------+-----------+ | ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public | +----+--------+-----------+------+-----------+------+-------+-------------+-----------+ | 6 | m1.odd | 2048 | 20 | 0 | | 3 | 1.0 | True | +----+--------+-----------+------+-----------+------+-------+-------------+-----------+ [root@seal17 ~(keystone_admin)]# nova flavor-key 6 set hw:cpu_policy=dedicated [root@seal17 ~(keystone_admin)]# [root@seal17 ~(keystone_admin)]# [root@seal17 ~(keystone_admin)]# nova flavor-create m1.odd 7 2048 20 4 ERROR (Conflict): Flavor with name m1.odd already exists. (HTTP 409) (Request-ID: req-b26e658b-247a-4757-bcd7-ba8b767172b4) [root@seal17 ~(keystone_admin)]# [root@seal17 ~(keystone_admin)]# [root@seal17 ~(keystone_admin)]# nova flavor-create m1.even 7 2048 20 4 +----+---------+-----------+------+-----------+------+-------+-------------+-----------+ | ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public | +----+---------+-----------+------+-----------+------+-------+-------------+-----------+ | 7 | m1.even | 2048 | 20 | 0 | | 4 | 1.0 | True | +----+---------+-----------+------+-----------+------+-------+-------------+-----------+ [root@seal17 ~(keystone_admin)]# [root@seal17 ~(keystone_admin)]# nova flavor-key 7 set hw:cpu_policy=dedicated [root@seal17 ~(keystone_admin)]# [root@seal17 ~(keystone_admin)]# [root@seal17 ~(keystone_admin)]# nova list +--------------------------------------+------+--------+------------+-------------+---------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------+--------+------------+-------------+---------------------+ | b83970c4-2c9b-47c0-b6da-9814ffb6dcb7 | vm1 | ACTIVE | - | Running | public=172.24.4.227 | +--------------------------------------+------+--------+------------+-------------+---------------------+ [root@seal17 ~(keystone_admin)]# nova delete vm1 Request to delete server vm1 has been accepted. [root@seal17 ~(keystone_admin)]# [root@seal17 ~(keystone_admin)]# nova boot --image cirros --flavor m1.odd vm1.odd +--------------------------------------+-----------------------------------------------+ | Property | Value | +--------------------------------------+-----------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | | | OS-EXT-SRV-ATTR:host | - | | OS-EXT-SRV-ATTR:hypervisor_hostname | - | | OS-EXT-SRV-ATTR:instance_name | instance-00000002 | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | - | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | adminPass | gyTdm5VncqiM | | config_drive | | | created | 2016-02-02T14:57:17Z | | flavor | m1.odd (6) | | hostId | | | id | 5926294d-715c-4310-b488-26c57ffd1762 | | image | cirros (7238ec00-3913-4228-a113-a97162ca479f) | | key_name | - | | metadata | {} | | name | vm1.odd | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | default | | status | BUILD | | tenant_id | 8897dfddf6df49529f87ab966a124794 | | updated | 2016-02-02T14:57:18Z | | user_id | 7771ec536e494880ba0a993f77f9bac9 | +--------------------------------------+-----------------------------------------------+ [root@seal17 ~(keystone_admin)]# nova list +--------------------------------------+---------+--------+------------+-------------+---------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+---------+--------+------------+-------------+---------------------+ | 5926294d-715c-4310-b488-26c57ffd1762 | vm1.odd | ACTIVE | - | Running | public=172.24.4.228 | +--------------------------------------+---------+--------+------------+-------------+---------------------+ [root@seal17 ~(keystone_admin)]# [root@seal17 ~(keystone_admin)]# virsh list --all Id Name State ---------------------------------------------------- 3 instance-00000002 running [root@seal17 ~(keystone_admin)]# [root@seal17 ~(keystone_admin)]# [root@seal17 ~(keystone_admin)]# virsh dumpxml 3 .. ... <vcpu placement='static'>3</vcpu> <cputune> <shares>3072</shares> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='18'/> <vcpupin vcpu='2' cpuset='1'/> <emulatorpin cpuset='1-2,18'/> </cputune> [root@seal17 ~(keystone_admin)]# nova boot --image cirros --flavor m1.even vm1.even +--------------------------------------+-----------------------------------------------+ | Property | Value | +--------------------------------------+-----------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | | | OS-EXT-SRV-ATTR:host | - | | OS-EXT-SRV-ATTR:hypervisor_hostname | - | | OS-EXT-SRV-ATTR:instance_name | instance-00000003 | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | - | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | adminPass | VQrTrtW4u53q | | config_drive | | | created | 2016-02-02T15:00:07Z | | flavor | m1.even (7) | | hostId | | | id | 257a142a-ff00-4636-8c70-ecf626f26712 | | image | cirros (7238ec00-3913-4228-a113-a97162ca479f) | | key_name | - | | metadata | {} | | name | vm1.even | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | default | | status | BUILD | | tenant_id | 8897dfddf6df49529f87ab966a124794 | | updated | 2016-02-02T15:00:07Z | | user_id | 7771ec536e494880ba0a993f77f9bac9 | +--------------------------------------+-----------------------------------------------+ [root@seal17 ~(keystone_admin)]# nova list +--------------------------------------+----------+--------+------------+-------------+---------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+----------+--------+------------+-------------+---------------------+ | 257a142a-ff00-4636-8c70-ecf626f26712 | vm1.even | ACTIVE | - | Running | public=172.24.4.229 | | 5926294d-715c-4310-b488-26c57ffd1762 | vm1.odd | ACTIVE | - | Running | public=172.24.4.228 | +--------------------------------------+----------+--------+------------+-------------+---------------------+ [root@seal17 ~(keystone_admin)]# [root@seal17 ~(keystone_admin)]# [root@seal17 ~(keystone_admin)]# virsh list --all Id Name State ---------------------------------------------------- 3 instance-00000002 running 4 instance-00000003 running [root@seal17 ~(keystone_admin)]# virsh dumpxml 4 <shares>4096</shares> <vcpupin vcpu='0' cpuset='0'/> <vcpupin vcpu='1' cpuset='16'/> <vcpupin vcpu='2' cpuset='3'/> <vcpupin vcpu='3' cpuset='19'/> <emulatorpin cpuset='0,3,16,19'/>
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-0603.html