Bug 1358284
| Summary: | evacuation of instance can overcommit target compute if there are multiple nova-scheduler | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Martin Schuppert <mschuppe> | ||||
| Component: | openstack-nova | Assignee: | Eoghan Glynn <eglynn> | ||||
| Status: | CLOSED WONTFIX | QA Contact: | Prasanth Anbalagan <panbalag> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 6.0 (Juno) | CC: | berrange, dasmith, eglynn, ipetrova, kchamart, mschuppe, mtessun, sbauza, sferdjao, sgordon, skinjo, srevivo, vromanso | ||||
| Target Milestone: | async | Keywords: | ZStream | ||||
| Target Release: | 6.0 (Juno) | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-01-17 19:23:17 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Any updates, guys? Any need for additional information? --Irina The scheduler design is made for being optimistic and possibly checking resources being not accurate for the price of being fast returning the destination. For that trade-off (having the scheduler possibly returning a wrong destination), we have a synchronous code section that makes sure for every instance creation (or move) in the compute service that the related resources are free. If the compute resources aren't enough free (for example, not enough free VCPUs), then what we call the "instance claim" from the compute service would then return an exception which would then reschedule the request to another compute. Here, what I don't understand is that even if the scheduler is making a wrong decision, the compute is still happy with two instances being created. My first guess is that you the CPU allocation ratio is not modified. Given that the default value is 16.0, that means that it's possible to have 16 times the number of physical CPUs for one compute. Here, say that you have 2 pCPUs, it's totally okay to have 32 assigned vCPUs, ie. 16 instances with a 2-vcpu flavor. Could you please check the configuration option value for cpu_allocation_ratio ? Could you also please give me compute logs that would recreate the original situation (ie. when cpus_used > cpus) Thanks, -Sylvain Hi Sylvain, (In reply to Sylvain Bauza from comment #3) > The scheduler design is made for being optimistic and possibly checking > resources being not accurate for the price of being fast returning the > destination. For that trade-off (having the scheduler possibly returning a > wrong destination), we have a synchronous code section that makes sure for > every instance creation (or move) in the compute service that the related > resources are free. > > If the compute resources aren't enough free (for example, not enough free > VCPUs), then what we call the "instance claim" from the compute service > would then return an exception which would then reschedule the request to > another compute. > > Here, what I don't understand is that even if the scheduler is making a > wrong decision, the compute is still happy with two instances being created. > My first guess is that you the CPU allocation ratio is not modified. Given > that the default value is 16.0, that means that it's possible to have 16 > times the number of physical CPUs for one compute. Here, say that you have 2 > pCPUs, it's totally okay to have 32 assigned vCPUs, ie. 16 instances with a > 2-vcpu flavor. > > Could you please check the configuration option value for > cpu_allocation_ratio ? This was also our 1st guess, but from initial environment this was reported: $ grep cpu_allocation_ratio controller-?.ncio.localdomain/etc/nova/nova.conf controller-1.localdomain/etc/nova/nova.conf:#cpu_allocation_ratio=16.0 controller-1.localdomain/etc/nova/nova.conf:cpu_allocation_ratio=1.0 controller-2.localdomain/etc/nova/nova.conf:#cpu_allocation_ratio=16.0 controller-2.localdomain/etc/nova/nova.conf:cpu_allocation_ratio=1.0 controller-3.localdomain/etc/nova/nova.conf:#cpu_allocation_ratio=16.0 controller-3.localdomain/etc/nova/nova.conf:cpu_allocation_ratio=1.0 Ans also from the debug output of the reproducer setup in the BZ description we see that cpu_allocation_ratio == 1: 2016-07-20 12:25:13.350 3881 DEBUG nova.scheduler.filters.core_filter [req-d5a97664-727a-40a2-800a-b8d883b6d71a None] vCPU summary - vcpu total 4 ,host_state.vcpus_total 4,cpu_allocation_ratio 1,cpus used 0 host_passes /usr/lib/python2.7/site-packages/nova/scheduler/filters/core_filter.py:66 > Could you also please give me compute logs that would recreate the original > situation (ie. when cpus_used > cpus) I'll attach compute log of osp6-compute2 from the time frame of the description to the BZ where we had 6 used/ 4 available vcpus after evacuation. [root@osp6-controller ~(keystone_admin)]# nova hypervisor-show 2 | egrep "name|vcpus" | hypervisor_hostname | osp6-compute2 | | vcpus | 4 | | vcpus_used | 6 | Thanks, Martin Created attachment 1186293 [details]
nova-compute.log osp6-compute2
Hello, Once again: anything new? The customer is getting rightfully impatient. Best, Irina I made a bit of investigation and I noticed that the process I mentioned on comment #3 about the compute service "claiming" for resources is only implemented in OSP8 (ie. Liberty) and above. That specific implementation requires RPC changes and a new DB object model that specifically prevent us to backport those changes (there are at least 3 commits I noticied) into OSP6. Regarding the customer issue, the possible recommendations I could tell are : - identify in some point in time a possible upgrade to OSP8 or above - prevent doing concurrent evacuations and pause between those (for example, by writing a script that would introduce that delay) - only run one scheduler process out of 3 (and kill the 2 others). For HA purpose, a classical Active/Passive scenario that would run a second scheduler in case the first one was found dead. The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days |
Description of problem: In an HA setup with multiple nova-scheduler an evacuation can overcommit the target compute (tested with vcpu) in case of concurrent scheduling actions. Version-Release number of selected component (if applicable): * OSP6 * openstack-nova-scheduler-2014.2.3-72.el7ost How reproducible: always Steps to Reproduce: * 2 computes with 4 cpus each * 3 nova scheduler processes * instance with flavor of 2 vcpus * 1 instance running on compute2 * 2 instances running on compute3 [root@osp6-controller ~(keystone_admin)]# nova service-list +----+------------------+------------------+----------+---------+-------+----------------------------+-----------------+ | Id | Binary | Host | Zone | Status | State |Updated_at | Disabled Reason | +----+------------------+------------------+----------+---------+-------+----------------------------+-----------------+ | 1 | nova-consoleauth | osp6-controller | internal | enabled | up |2016-07-20T08:16:43.000000 | - | | 2 | nova-scheduler | osp6-controller | internal | enabled | up |2016-07-20T08:16:43.000000 | - | | 3 | nova-conductor | osp6-controller | internal | enabled | up |2016-07-20T08:16:43.000000 | - | | 4 | nova-cert | osp6-controller | internal | enabled | up |2016-07-20T08:16:43.000000 | - | | 5 | nova-compute | osp6-compute1 | AZ | enabled | down |2016-07-12T08:39:28.000000 | - | | 6 | nova-compute | osp6-compute2 | nova | enabled | up |2016-07-20T08:16:40.000000 | - | | 8 | nova-compute | osp6-compute3 | nova | enabled | up |2016-07-20T08:16:36.000000 | - | | 13 | nova-scheduler | osp6-controller2 | internal | enabled | up |2016-07-20T08:16:43.000000 | - | | 14 | nova-scheduler | osp6-controller3 | internal | enabled | up |2016-07-20T08:16:42.000000 | - | +----+------------------+------------------+----------+---------+-------+----------------------------+-----------------+ [root@osp6-controller ~(keystone_admin)]# nova list --field name,host,flavor,status,power_state,task_state,metadata +--------------------------------------+-----------+---------------+--------+--------+-------------+------------+----------+ | ID | Name | Host | Flavor |Status | Power State | Task State | Metadata | +--------------------------------------+-----------+---------------+--------+--------+-------------+------------+----------+ | 6d091c4c-c11c-4b17-bdee-20bddd47958e | instance1 | osp6-compute3 | 11 |ACTIVE | Running | None | {} | | 9dbd97f1-ee69-49cd-af5c-d5e59b6db0c4 | instance2 | osp6-compute2 | 11 |ACTIVE | Running | None | {} | | ef438088-c917-4efd-bd94-26298f1287c5 | instance3 | osp6-compute3 | 11 |ACTIVE | Running | None | {} | +--------------------------------------+-----------+---------------+--------+--------+-------------+------------+----------+ -> Poweroff compute3 [root@hp-dl380-gen9-10 qemu]# virsh destroy osp6-compute3 Domain osp6-compute3 destroyed * when compute marked as down in `nova service-list` ->evacuate both instances: [root@osp6-controller ~(keystone_admin)]# nova evacuate --on-shared-storage instance1 && nova evacuate --on-shared-storage instance3 [root@osp6-controller ~(keystone_admin)]# nova list --field name,host,flavor,status,power_state,task_state,metadata +--------------------------------------+-----------+---------------+--------+--------+-------------+------------+----------+ | ID | Name | Host | Flavor |Status | Power State | Task State | Metadata | +--------------------------------------+-----------+---------------+--------+--------+-------------+------------+----------+ | 6d091c4c-c11c-4b17-bdee-20bddd47958e | instance1 | osp6-compute2 | 11 |ACTIVE | Running | None | {} | | 9dbd97f1-ee69-49cd-af5c-d5e59b6db0c4 | instance2 | osp6-compute2 | 11 |ACTIVE | Running | None | {} | | ef438088-c917-4efd-bd94-26298f1287c5 | instance3 | osp6-compute2 | 11 |ACTIVE | Running | None | {} | +--------------------------------------+-----------+---------------+--------+--------+-------------+------------+----------+ [root@osp6-controller ~(keystone_admin)]# nova hypervisor-show 2 | egrep "name|vcpus" | hypervisor_hostname | osp6-compute2 | | vcpus | 4 | | vcpus_used | 6 | Actual results: all instance get scheduled on target compute and oversubscribe the max number of usable vcpus. Expected results: scheduling fails due to resource limit reached Additional info: From another test (0 instances on target compute) with additional DEBUG output in corefilter we see that used cpus is 0 on all 3 schedulers, probably because the resource usage is being updated after a successful evacuation. From controller1: ~~~ 2016-07-20 12:25:11.636 28443 DEBUG nova.filters [req-a1e9acfd-defa-4c0c-bb5e-ccb152f75cc5 None] Filter AvailabilityZoneFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:11.637 28443 DEBUG nova.filters [req-a1e9acfd-defa-4c0c-bb5e-ccb152f75cc5 None] Filter RamFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:11.637 28443 DEBUG nova.servicegroup.drivers.db [req-a1e9acfd-defa-4c0c-bb5e-ccb152f75cc5 None] Seems service is down. Last heartbeat was 2016-07-12 08:39:28. Elapsed time is 697543.63725 is_up /usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py:75 2016-07-20 12:25:11.637 28443 WARNING nova.scheduler.filters.compute_filter [req-a1e9acfd-defa-4c0c-bb5e-ccb152f75cc5 None] (osp6-compute1, osp6-compute1) ram:7121 disk:94208 io_ops:0 instances:0 has not been heard from in a while 2016-07-20 12:25:11.637 28443 DEBUG nova.filters [req-a1e9acfd-defa-4c0c-bb5e-ccb152f75cc5 None] Filter ComputeFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:11.637 28443 DEBUG nova.filters [req-a1e9acfd-defa-4c0c-bb5e-ccb152f75cc5 None] Filter ComputeCapabilitiesFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:11.637 28443 DEBUG nova.filters [req-a1e9acfd-defa-4c0c-bb5e-ccb152f75cc5 None] Filter ImagePropertiesFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:11.638 28443 DEBUG nova.scheduler.filters.core_filter [req-a1e9acfd-defa-4c0c-bb5e-ccb152f75cc5 None] vCPU summary - vcpu total 4 ,host_state.vcpus_total 4,cpu_allocation_ratio 1,cpus used 0 host_passes /usr/lib/python2.7/site-packages/nova/scheduler/filters/core_filter.py:66 --> 0 used cpus returned 2016-07-20 12:25:11.638 28443 DEBUG nova.filters [req-a1e9acfd-defa-4c0c-bb5e-ccb152f75cc5 None] Filter CoreFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:11.669 28443 INFO nova.openstack.common.periodic_task [req-a1e9acfd-defa-4c0c-bb5e-ccb152f75cc5 None] Skipping periodic task _periodic_update_dns because its interval is negative 2016-07-20 12:25:11.680 28443 DEBUG nova.filters [req-a1e9acfd-defa-4c0c-bb5e-ccb152f75cc5 None] Filter DifferentHostFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:11.684 28443 DEBUG nova.filters [req-a1e9acfd-defa-4c0c-bb5e-ccb152f75cc5 None] Filter AggregateInstanceExtraSpecsFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:11.685 28443 DEBUG nova.filters [req-a1e9acfd-defa-4c0c-bb5e-ccb152f75cc5 None] Filter ServerGroupAntiAffinityFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:11.686 28443 DEBUG nova.filters [req-a1e9acfd-defa-4c0c-bb5e-ccb152f75cc5 None] Filter ServerGroupAffinityFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:11.686 28443 DEBUG nova.filters [req-a1e9acfd-defa-4c0c-bb5e-ccb152f75cc5 None] Filter PciPassthroughFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:11.689 28443 DEBUG nova.filters [req-a1e9acfd-defa-4c0c-bb5e-ccb152f75cc5 None] Filter NUMATopologyFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:11.689 28443 DEBUG nova.scheduler.filter_scheduler [req-a1e9acfd-defa-4c0c-bb5e-ccb152f75cc5 None] Filtered [(osp6-compute3, osp6-compute3) ram:7121 disk:101376 io_ops:0 instances:0] _schedule /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:298 2016-07-20 12:25:11.689 28443 DEBUG nova.scheduler.filter_scheduler [req-a1e9acfd-defa-4c0c-bb5e-ccb152f75cc5 None] Weighed [WeighedHost [host: (osp6-compute3, osp6-compute3) ram:7121 disk:101376 io_ops:0 instances:0, weight: 1.0]] _schedule /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:303 ~~~ * From controller2: ~~~ 2016-07-20 12:25:12.534 2826 DEBUG nova.filters [req-cf0787c4-7549-4502-a5f6-4dde329e1b19 None] Filter AvailabilityZoneFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:12.535 2826 DEBUG nova.filters [req-cf0787c4-7549-4502-a5f6-4dde329e1b19 None] Filter RamFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:12.535 2826 DEBUG nova.servicegroup.drivers.db [req-cf0787c4-7549-4502-a5f6-4dde329e1b19 None] Seems service is down. Last heartbeat was 2016-07-12 08:39:28. Elapsed time is 697544.53567 is_up /usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py:75 2016-07-20 12:25:12.536 2826 WARNING nova.scheduler.filters.compute_filter [req-cf0787c4-7549-4502-a5f6-4dde329e1b19 None] (osp6-compute1, osp6-compute1) ram:7121 disk:94208 io_ops:0 instances:0 has not been heard from in a while 2016-07-20 12:25:12.536 2826 DEBUG nova.filters [req-cf0787c4-7549-4502-a5f6-4dde329e1b19 None] Filter ComputeFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:12.536 2826 DEBUG nova.filters [req-cf0787c4-7549-4502-a5f6-4dde329e1b19 None] Filter ComputeCapabilitiesFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:12.537 2826 DEBUG nova.filters [req-cf0787c4-7549-4502-a5f6-4dde329e1b19 None] Filter ImagePropertiesFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:12.537 2826 DEBUG nova.scheduler.filters.core_filter [req-cf0787c4-7549-4502-a5f6-4dde329e1b19 None] vCPU summary - vcpu total 4 ,host_state.vcpus_total 4,cpu_allocation_ratio 1,cpus used 0 host_passes /usr/lib/python2.7/site-packages/nova/scheduler/filters/core_filter.py:66 --> 0 used cpus returned 2016-07-20 12:25:12.538 2826 DEBUG nova.filters [req-cf0787c4-7549-4502-a5f6-4dde329e1b19 None] Filter CoreFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:12.595 2826 INFO nova.openstack.common.periodic_task [req-cf0787c4-7549-4502-a5f6-4dde329e1b19 None] Skipping periodic task _periodic_update_dns because its interval is negative 2016-07-20 12:25:12.610 2826 DEBUG nova.filters [req-cf0787c4-7549-4502-a5f6-4dde329e1b19 None] Filter DifferentHostFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:12.620 2826 DEBUG nova.filters [req-cf0787c4-7549-4502-a5f6-4dde329e1b19 None] Filter AggregateInstanceExtraSpecsFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:12.621 2826 DEBUG nova.filters [req-cf0787c4-7549-4502-a5f6-4dde329e1b19 None] Filter ServerGroupAntiAffinityFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:12.621 2826 DEBUG nova.filters [req-cf0787c4-7549-4502-a5f6-4dde329e1b19 None] Filter ServerGroupAffinityFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:12.622 2826 DEBUG nova.filters [req-cf0787c4-7549-4502-a5f6-4dde329e1b19 None] Filter PciPassthroughFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:12.625 2826 DEBUG nova.filters [req-cf0787c4-7549-4502-a5f6-4dde329e1b19 None] Filter NUMATopologyFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:12.625 2826 DEBUG nova.scheduler.filter_scheduler [req-cf0787c4-7549-4502-a5f6-4dde329e1b19 None] Filtered [(osp6-compute3, osp6-compute3) ram:7121 disk:101376 io_ops:0 instances:0] _schedule /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:298 2016-07-20 12:25:12.626 2826 DEBUG nova.scheduler.filter_scheduler [req-cf0787c4-7549-4502-a5f6-4dde329e1b19 None] Weighed [WeighedHost [host: (osp6-compute3, osp6-compute3) ram:7121 disk:101376 io_ops:0 instances:0, weight: 1.0]] _schedule /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:303 ~~~ * From controller3: ~~~ 2016-07-20 12:25:13.348 3881 DEBUG nova.filters [req-d5a97664-727a-40a2-800a-b8d883b6d71a None] Filter AvailabilityZoneFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:13.348 3881 DEBUG nova.filters [req-d5a97664-727a-40a2-800a-b8d883b6d71a None] Filter RamFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:13.348 3881 DEBUG nova.servicegroup.drivers.db [req-d5a97664-727a-40a2-800a-b8d883b6d71a None] Seems service is down. Last heartbeat was 2016-07-12 08:39:28. Elapsed time is 697545.348765 is_up /usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py:75 2016-07-20 12:25:13.349 3881 WARNING nova.scheduler.filters.compute_filter [req-d5a97664-727a-40a2-800a-b8d883b6d71a None] (osp6-compute1, osp6-compute1) ram:7121 disk:94208 io_ops:0 instances:0 has not been heard from in a while 2016-07-20 12:25:13.349 3881 DEBUG nova.filters [req-d5a97664-727a-40a2-800a-b8d883b6d71a None] Filter ComputeFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:13.349 3881 DEBUG nova.filters [req-d5a97664-727a-40a2-800a-b8d883b6d71a None] Filter ComputeCapabilitiesFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:13.349 3881 DEBUG nova.filters [req-d5a97664-727a-40a2-800a-b8d883b6d71a None] Filter ImagePropertiesFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:13.350 3881 DEBUG nova.scheduler.filters.core_filter [req-d5a97664-727a-40a2-800a-b8d883b6d71a None] vCPU summary - vcpu total 4 ,host_state.vcpus_total 4,cpu_allocation_ratio 1,cpus used 0 host_passes /usr/lib/python2.7/site-packages/nova/scheduler/filters/core_filter.py:66 --> 0 used cpus returned 2016-07-20 12:25:13.350 3881 DEBUG nova.filters [req-d5a97664-727a-40a2-800a-b8d883b6d71a None] Filter CoreFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:13.389 3881 INFO nova.openstack.common.periodic_task [req-d5a97664-727a-40a2-800a-b8d883b6d71a None] Skipping periodic task _periodic_update_dns because its interval is negative 2016-07-20 12:25:13.402 3881 DEBUG nova.filters [req-d5a97664-727a-40a2-800a-b8d883b6d71a None] Filter DifferentHostFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:13.408 3881 DEBUG nova.filters [req-d5a97664-727a-40a2-800a-b8d883b6d71a None] Filter AggregateInstanceExtraSpecsFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:13.409 3881 DEBUG nova.filters [req-d5a97664-727a-40a2-800a-b8d883b6d71a None] Filter ServerGroupAntiAffinityFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:13.410 3881 DEBUG nova.filters [req-d5a97664-727a-40a2-800a-b8d883b6d71a None] Filter ServerGroupAffinityFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:13.410 3881 DEBUG nova.filters [req-d5a97664-727a-40a2-800a-b8d883b6d71a None] Filter PciPassthroughFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:13.414 3881 DEBUG nova.filters [req-d5a97664-727a-40a2-800a-b8d883b6d71a None] Filter NUMATopologyFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:88 2016-07-20 12:25:13.415 3881 DEBUG nova.scheduler.filter_scheduler [req-d5a97664-727a-40a2-800a-b8d883b6d71a None] Filtered [(osp6-compute3, osp6-compute3) ram:7121 disk:101376 io_ops:0 instances:0] _schedule /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:298 2016-07-20 12:25:13.415 3881 DEBUG nova.scheduler.filter_scheduler [req-d5a97664-727a-40a2-800a-b8d883b6d71a None] Weighed [WeighedHost [host: (osp6-compute3, osp6-compute3) ram:7121 disk:101376 io_ops:0 instances:0, weight: 1.0]] _schedule /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:303 ~~~ The issue is that the DB gets updated AFTER the successful evacuation. Therefore with parallel evacuation take place on different scheduler in a HA environment can lead to the overcommit.