Description of problem: Live migration invoked from Horizon skips scheduler while using metadata Version-Release number of selected component (if applicable): RHOSP10, director controller x 3, compute x 2 How reproducible: Everytime for Cu. Steps to Reproduce: Let's assume cirros image and 2 compute nodes (osp10-comp01, osp10-comp02). 1. upload cirros image. 2. set os_distro=cirros to the image. [stack@osp10-director work(overcloud_admin):v2]$ openstack image set --os-distro cirros cirros 3. Create 2 Host Aggregates named 'ag_cirros' and 'ag_fedora', and set 'os_distro=cirros', 'os_distro=fedora' metadata respectively. [stack@osp10-director work(overcloud_admin):v2]$ openstack aggregate create ag_cirros --property os_distro=cirros [stack@osp10-director work(overcloud_admin):v2]$ openstack aggregate create ag_fedora --property os_distro=fedora 4. Set osp10-comp02 to ag_cirros Host Aggregate and osp10-comp01 to ag_fedora Host Aggregate. [stack@osp10-director work(overcloud_admin):v2]$ openstack aggregate add host ag_cirros osp10-comp02.osptest.local [stack@osp10-director work(overcloud_admin):v2]$ openstack aggregate add host ag_fedora osp10-comp01.osptest.local 5. Create an instance named 'vm1' from cirros image. The instance is scheduled to osp10-comp02 by AggregateImagePropertiesIsolation. [stack@osp10-director work(overcloud_admin):v2]$ openstack image show cirros +------------------+-----------------------------------------------------------------------------------------------------+ | Field | Value | +------------------+-----------------------------------------------------------------------------------------------------+ | checksum | ee1eca47dc88f4879d8a229cc70a07c6 | | container_format | bare | | created_at | 2017-02-24T05:34:56Z | | disk_format | qcow2 | | file | /v2/images/2d8f987e-59d6-4065-88b1-4480b9320e7b/file | | id | 2d8f987e-59d6-4065-88b1-4480b9320e7b | | min_disk | 0 | | min_ram | 0 | | name | cirros | | owner | c4c4af5e7fff40729f50bf2959dc29dd | | properties | direct_url='file:///var/lib/glance/images/2d8f987e-59d6-4065-88b1-4480b9320e7b', os_distro='cirros' | | protected | False | | schema | /v2/schemas/image | | size | 13287936 | | status | active | | tags | | | updated_at | 2017-02-24T06:12:04Z | | virtual_size | None | | visibility | public | +------------------+-----------------------------------------------------------------------------------------------------+ [stack@osp10-director work(overcloud_admin):v2]$ nova boot --poll --flavor m1.nano --key-name sshkey --nic net-name=admin_net --security-groups sg_admin --block-device id=2d8f987e-59d6-4065-88b1-4480b9320e7b,source=image,dest=volume,size=1,bootindex=0,shutdown=remove vm1 /var/log/nova/nova-scheduler.log shows that AggregateImagePropertiesIsolation filter selected 1 host who is in a Host Aggregate with 'os_distro=cirros' metadata (marked as '### XXX'). 2017-02-24 15:18:06.887 370339 DEBUG nova.filters [req-46c08031-4d05-49c2-b19e-d8e2a5aebbe2 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Starting with 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:70 2017-02-24 15:18:06.888 370339 DEBUG nova.filters [req-46c08031-4d05-49c2-b19e-d8e2a5aebbe2 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Filter RetryFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104 2017-02-24 15:18:06.890 370339 DEBUG nova.filters [req-46c08031-4d05-49c2-b19e-d8e2a5aebbe2 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Filter AvailabilityZoneFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104 2017-02-24 15:18:06.890 370339 DEBUG nova.filters [req-46c08031-4d05-49c2-b19e-d8e2a5aebbe2 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Filter RamFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104 2017-02-24 15:18:06.891 370339 DEBUG nova.filters [req-46c08031-4d05-49c2-b19e-d8e2a5aebbe2 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Filter DiskFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104 2017-02-24 15:18:06.891 370339 DEBUG nova.filters [req-46c08031-4d05-49c2-b19e-d8e2a5aebbe2 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Filter ComputeFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104 2017-02-24 15:18:06.892 370339 DEBUG nova.filters [req-46c08031-4d05-49c2-b19e-d8e2a5aebbe2 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Filter ComputeCapabilitiesFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104 2017-02-24 15:18:06.892 370339 DEBUG nova.filters [req-46c08031-4d05-49c2-b19e-d8e2a5aebbe2 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Filter ImagePropertiesFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104 2017-02-24 15:18:06.893 370339 DEBUG nova.filters [req-46c08031-4d05-49c2-b19e-d8e2a5aebbe2 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Filter ServerGroupAntiAffinityFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104 2017-02-24 15:18:06.893 370339 DEBUG nova.filters [req-46c08031-4d05-49c2-b19e-d8e2a5aebbe2 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Filter ServerGroupAffinityFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104 2017-02-24 15:18:06.894 370339 DEBUG nova.scheduler.filters.aggregate_image_properties_isolation [req-46c08031-4d05-49c2-b19e-d8e2a5aebbe2 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] (osp10-comp01.osptest.local, osp10-comp01.osptest.local) ram: 2047MB disk: 56320MB io_ops: 0 instances: 0 fails image aggregate properties requirements. Property cirros does not match set([u'fedora']). host_passes /usr/lib/python2.7/site-packages/nova/scheduler/filters/aggregate_image_properties_isolation.py:67 2017-02-24 15:18:06.894 370339 DEBUG nova.filters [req-46c08031-4d05-49c2-b19e-d8e2a5aebbe2 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Filter AggregateImagePropertiesIsolation returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104 ### XXX 2017-02-24 15:18:06.894 370339 DEBUG nova.scheduler.filter_scheduler [req-46c08031-4d05-49c2-b19e-d8e2a5aebbe2 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Filtered [(osp10-comp02.osptest.local, osp10-comp02.osptest.local) ram: 2047MB disk: 56320MB io_ops: 0 instances: 0] _schedule /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:118 2017-02-24 15:18:06.895 370339 DEBUG nova.scheduler.filter_scheduler [req-46c08031-4d05-49c2-b19e-d8e2a5aebbe2 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Weighed [WeighedHost [host: (osp10-comp02.osptest.local, osp10-comp02.osptest.local) ram: 2047MB disk: 56320MB io_ops: 0 instances: 0, weight: 0.0]] _schedule /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:123 2017-02-24 15:18:06.895 370339 DEBUG nova.scheduler.filter_scheduler [req-46c08031-4d05-49c2-b19e-d8e2a5aebbe2 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Selected host: WeighedHost [host: (osp10-comp02.osptest.local, osp10-comp02.osptest.local) ram: 2047MB disk: 56320MB io_ops: 0 instances: 0, weight: 0.0] _schedule /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:131 6. Run 'nova live migration' CLI command for the instance to migrate to osp10-comp01 who is in a Host Aggregate with 'os_distro=fedora' metadata. The migration fails as expected because the Host Aggregate osp10-comp01 is in doesn't have 'os_distro=cirros' metadata. [stack@osp10-director work(overcloud_admin):v2]$ date; nova live-migration vm1 osp10-comp01.osptest.local Fri Feb 24 15:22:14 JST 2017 [stack@osp10-director work(overcloud_admin):v2]$ /var/log/nova/nova-scheduler.log shows that AggregateImagePropertiesIsolation filter selected no host who is in a Host Aggregate with 'os_distro=cirros' metadata (marked as '### XXX'). 2017-02-24 15:22:17.509 370339 INFO nova.scheduler.host_manager [req-e1ef367f-fbd8-4119-8a02-2b3a389fc9e7 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Host filter only checking host osp10-comp01.osptest.local and node osp10-comp01.osptest.local 2017-02-24 15:22:17.509 370339 INFO nova.scheduler.host_manager [req-e1ef367f-fbd8-4119-8a02-2b3a389fc9e7 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Host filter ignoring hosts: 2017-02-24 15:22:17.510 370339 DEBUG nova.filters [req-e1ef367f-fbd8-4119-8a02-2b3a389fc9e7 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Starting with 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:70 2017-02-24 15:22:17.510 370339 DEBUG nova.scheduler.filters.retry_filter [req-e1ef367f-fbd8-4119-8a02-2b3a389fc9e7 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Re-scheduling is disabled host_passes /usr/lib/python2.7/site-packages/nova/scheduler/filters/retry_filter.py:34 2017-02-24 15:22:17.510 370339 DEBUG nova.filters [req-e1ef367f-fbd8-4119-8a02-2b3a389fc9e7 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Filter RetryFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104 2017-02-24 15:22:17.511 370339 DEBUG nova.filters [req-e1ef367f-fbd8-4119-8a02-2b3a389fc9e7 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Filter AvailabilityZoneFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104 2017-02-24 15:22:17.511 370339 DEBUG nova.filters [req-e1ef367f-fbd8-4119-8a02-2b3a389fc9e7 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Filter RamFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104 2017-02-24 15:22:17.511 370339 DEBUG nova.filters [req-e1ef367f-fbd8-4119-8a02-2b3a389fc9e7 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Filter DiskFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104 2017-02-24 15:22:17.511 370339 DEBUG nova.filters [req-e1ef367f-fbd8-4119-8a02-2b3a389fc9e7 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Filter ComputeFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104 2017-02-24 15:22:17.512 370339 DEBUG nova.filters [req-e1ef367f-fbd8-4119-8a02-2b3a389fc9e7 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Filter ComputeCapabilitiesFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104 2017-02-24 15:22:17.512 370339 DEBUG nova.filters [req-e1ef367f-fbd8-4119-8a02-2b3a389fc9e7 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Filter ImagePropertiesFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104 2017-02-24 15:22:17.512 370339 DEBUG nova.filters [req-e1ef367f-fbd8-4119-8a02-2b3a389fc9e7 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Filter ServerGroupAntiAffinityFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104 2017-02-24 15:22:17.512 370339 DEBUG nova.filters [req-e1ef367f-fbd8-4119-8a02-2b3a389fc9e7 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Filter ServerGroupAffinityFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104 2017-02-24 15:22:17.513 370339 DEBUG nova.scheduler.filters.aggregate_image_properties_isolation [req-e1ef367f-fbd8-4119-8a02-2b3a389fc9e7 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] (osp10-comp01.osptest.local, osp10-comp01.osptest.local) ram: 2047MB disk: 56320MB io_ops: 0 instances: 0 fails image aggregate properties requirements. Property cirros does not match set([u'fedora']). host_passes /usr/lib/python2.7/site-packages/nova/scheduler/filters/aggregate_image_properties_isolation.py:67 2017-02-24 15:22:17.513 370339 INFO nova.filters [req-e1ef367f-fbd8-4119-8a02-2b3a389fc9e7 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Filter AggregateImagePropertiesIsolation returned 0 hosts ### XXX 2017-02-24 15:22:17.513 370339 DEBUG nova.filters [req-e1ef367f-fbd8-4119-8a02-2b3a389fc9e7 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Filtering removed all hosts for the request with instance ID 'd90101fa-216a-47d2-bb4e-8273b2bdc3ad'. Filter results: [('RetryFilter', [(u'osp10-comp01.osptest.local', u'osp10-comp01.osptest.local')]), ('AvailabilityZoneFilter', [(u'osp10-comp01.osptest.local', u'osp10-comp01.osptest.local')]), ('RamFilter', [(u'osp10-comp01.osptest.local', u'osp10-comp01.osptest.local')]), ('DiskFilter', [(u'osp10-comp01.osptest.local', u'osp10-comp01.osptest.local')]), ('ComputeFilter', [(u'osp10-comp01.osptest.local', u'osp10-comp01.osptest.local')]), ('ComputeCapabilitiesFilter', [(u'osp10-comp01.osptest.local', u'osp10-comp01.osptest.local')]), ('ImagePropertiesFilter', [(u'osp10-comp01.osptest.local', u'osp10-comp01.osptest.local')]), ('ServerGroupAntiAffinityFilter', [(u'osp10-comp01.osptest.local', u'osp10-comp01.osptest.local')]), ('ServerGroupAffinityFilter', [(u'osp10-comp01.osptest.local', u'osp10-comp01.osptest.local')]), ('AggregateImagePropertiesIsolation', None)] get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:129 2017-02-24 15:22:17.514 370339 INFO nova.filters [req-e1ef367f-fbd8-4119-8a02-2b3a389fc9e7 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] Filtering removed all hosts for the request with instance ID 'd90101fa-216a-47d2-bb4e-8273b2bdc3ad'. Filter results: ['RetryFilter: (start: 1, end: 1)', 'AvailabilityZoneFilter: (start: 1, end: 1)', 'RamFilter: (start: 1, end: 1)', 'DiskFilter: (start: 1, end: 1)', 'ComputeFilter: (start: 1, end: 1)', 'ComputeCapabilitiesFilter: (start: 1, end: 1)', 'ImagePropertiesFilter: (start: 1, end: 1)', 'ServerGroupAntiAffinityFilter: (start: 1, end: 1)', 'ServerGroupAffinityFilter: (start: 1, end: 1)', 'AggregateImagePropertiesIsolation: (start: 1, end: 0)'] 2017-02-24 15:22:17.514 370339 DEBUG nova.scheduler.filter_scheduler [req-e1ef367f-fbd8-4119-8a02-2b3a389fc9e7 a4a6c19d265e4f6eaccf417a90dd3edc c4c4af5e7fff40729f50bf2959dc29dd - - -] There are 0 hosts available but 1 instances requested to build. select_destinations /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:71 7. Do live migration from Horizon for the same instance to migrate to osp10-comp01 who is in a Host Aggregate with 'os_distro=fedora' metadata. (specifying "New Host" form to 'osp10-comp01') The migration suceeded to migrate to osp10-comp01. This is not an unexpedted result. It seemed that the instance didn't scheduled by nova-scheduler from logs. Actual results: Migration is getting successful from Horizon. Expected results: Migration should not be completed from Horizon as well. Additional info: As per on-site consultant comment. (a) When CLI invokes live migration, nova-api get 'host_name=osp10-comp01' and 'force=False'. (b) When Horizon invokes live migration, nova-api get 'host_name=osp10-comp01' and 'force=None'. In (a), we go into the following if-clause (from API.live_migration()@compute/api.py), then the scheduler is called and AggregateImagePropertiesIsolation filters the destination host(s). ``` # NOTE(sbauza): Force is a boolean by the new related API version if force is False and host_name: nodes = objects.ComputeNodeList.get_all_by_host(context, host_name) if not nodes: raise exception.ComputeHostNotFound(host=host_name) # NOTE(sbauza): Unset the host to make sure we call the scheduler host_name = None # FIXME(sbauza): Since only Ironic driver uses more than one # compute per service but doesn't support evacuations, # let's provide the first one. target = nodes[0] if request_spec: # TODO(sbauza): Hydrate a fake spec for old instances not yet # having a request spec attached to them (particularly true for # cells v1). For the moment, let's keep the same behaviour for # all the instances but provide the destination only if a spec # is found. destination = objects.Destination( host=target.host, node=target.hypervisor_hostname ) request_spec.requested_destination = destination ``` In (b), the if-clause is skipped because 'force' is not False, then scheduler_hist.host is set to 'osp10-comp01.osptest.local' in ComputeTaskAPI.live_migration_insntance()@conductor/api.py, resulted in passing the scheduler filter. (In (a), host_name is overwrote by None in the if-clause, so scheduler_hist.host is None) With the following change, the live migration failed in (b) case, as expected. [root@osp10-ctrl03 site-packages]# pwd /usr/lib/python2.7/site-packages [root@osp10-ctrl03 site-packages]# diff -urN nova.orig nova diff -urN nova.orig/compute/api.py nova/compute/api.py --- nova.orig/compute/api.py 2016-11-22 05:19:53.000000000 +0900 +++ nova/compute/api.py 2017-02-24 15:28:36.833601281 +0900 @@ -3630,7 +3630,7 @@ request_spec = None # NOTE(sbauza): Force is a boolean by the new related API version - if force is False and host_name: + if not force and host_name: nodes = objects.ComputeNodeList.get_all_by_host(context, host_name) if not nodes: raise exception.ComputeHostNotFound(host=host_name)
When I wrote that conditional, I only verified if 'force' was False because we needed to keep the legacy behaviour in case the API call wasn't providing "force". To be clear, the API resource is depending on the microversion number, but we still need to accept old microversions or even calls without microversions (like v2.0 calls). So, we designed the API resource to only default to False if the microversion was fine, or leave force being None in case of not. If we merge your proposal, then live-migration calls having old microversions would change their behaviour by verifying the scheduler which is a problem for operators. So, instead, I think we should ask Horizon to put force to False if the microversion is newer than 2.30 https://docs.openstack.org/developer/nova/api_microversion_history.html#id27 exactly like novaclient does : https://github.com/openstack/python-novaclient/commit/c6496789914e2d88984a65350df1d70e75a4d759
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.
One more question, would it hurt to send False there even when the microversion is lower than 2.30? In other words, do we need to be checking the microversion?
Hello, I have managed to reproduce this in my test environment. I have used ceph as a backend for nova in my test environment. Here are the detailed steps which I performed in my test lab: [stack@instack ~]$ openstack --os-image-api-version 2 image create --public --disk-format qcow2 --container-format bare --file ~/cirros-0.3.4-x86_64-disk.img cirros [stack@instack ~]$ openstack --os-image-api-version 2 image set cirros --os-distro cirros [stack@instack ~]$ openstack --os-image-api-version 2 image show cirros +------------------+-------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------------+-------------------------------------------------------------------------------------------------------------------+ | checksum | ee1eca47dc88f4879d8a229cc70a07c6 | | container_format | bare | | created_at | 2017-03-27T03:42:07Z | | disk_format | qcow2 | | file | /v2/images/c093ab3d-1d2b-4ffe-b5ad-6ed1f1ab2895/file | | id | c093ab3d-1d2b-4ffe-b5ad-6ed1f1ab2895 | | min_disk | 0 | | min_ram | 0 | | name | cirros | | owner | ca244adae0d443c1acfe2336a2a76d80 | | properties | direct_url='rbd://dea376b4-0f91-11e7-a2ec-5254006517db/images/c093ab3d-1d2b-4ffe-b5ad-6ed1f1ab2895/snap', | | | locations='[{u'url': u'rbd://dea376b4-0f91-11e7-a2ec-5254006517db/images/c093ab3d-1d2b-4ffe-b5ad- | | | 6ed1f1ab2895/snap', u'metadata': {}}]', os_distro='cirros' | [stack@instack ~]$ openstack aggregate create ag_cirros --property os_distro=cirros [stack@instack ~]$ openstack aggregate create ag_fedora --property os_distro=fedora [stack@instack ~]$ openstack aggregate add host ag_cirros overcloud-compute-1.localdomain +-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | availability_zone | None | | created_at | 2017-03-27T03:48:22.000000 | | deleted | False | | deleted_at | None | | hosts | [u'overcloud-compute-1.localdomain'] | | id | 1 | | metadata | {u'os_distro': u'cirros'} | | name | ag_cirros | | updated_at | None | +-------------------+--------------------------------------+ [stack@instack ~]$ openstack aggregate add host ag_fedora overcloud-compute-0.localdomain +-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | availability_zone | None | | created_at | 2017-03-27T03:48:56.000000 | | deleted | False | | deleted_at | None | | hosts | [u'overcloud-compute-0.localdomain'] | | id | 2 | | metadata | {u'os_distro': u'fedora'} | | name | ag_fedora | | updated_at | None | +-------------------+--------------------------------------+ As expected initially the instance was launched compute-1 due to "AggregateImagePropertiesIsolation" filter: [stack@instack ~]$ nova boot --poll --flavor m1.tiny --image cirros --nic net-id=84602360-2569-45a7-9ef0-2e855560f118 --security-groups default inst1 +--------------------------------------+----------------------------------------------------------+ | Property | Value | +--------------------------------------+----------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | overcloud-compute-1.localdomain | | OS-EXT-SRV-ATTR:hostname | inst1 | | OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-compute-1.localdomain | | OS-EXT-SRV-ATTR:instance_name | instance-00000006 | Then I tried live migrating the instance to compute-0 via the command line: [stack@instack ~]$ nova --debug live-migration inst1 overcloud-compute-0.localdomain ...... i-Version: 2.37 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version X-Compute-Request-Id: req-42a5cb09-dcc8-4bc1-9801-93ad8c583a34 Date: Tue, 28 Mar 2017 03:45:29 GMT DEBUG (client:85) POST call to compute for http://192.0.2.15:8774/v2.1/servers/7b576745-281f-439b-8ae2-e43e823ae815/action used request id req-42a5cb09-dcc8-4bc1-9801-93ad8c583a34 As expected the logs show that the migration could not succeed due to no hosts were found +++ 017-03-28 03:45:30.042 1578 DEBUG nova.scheduler.filters.aggregate_image_properties_isolation [req-42a5cb09-dcc8-4bc1-9801-93ad8c583a34 caac64b245bb480289b7951e3ff871c4 ca244adae0d443c1acfe2336a2a76d80 - - -] (overcloud-compute-0.localdomain, overcloud-compute-0.localdomain) ram: 6143MB disk: 99328MB io_ops: 0 instances: 0 fails image aggregate properties requirements. Property cirros does not match set([u'fedora']). host_passes /usr/lib/python2.7/site-packages/nova/scheduler/filters/aggregate_image_properties_isolation.py:67 XXXXX 2017-03-28 03:45:30.042 1578 INFO nova.filters [req-42a5cb09-dcc8-4bc1-9801-93ad8c583a34 caac64b245bb480289b7951e3ff871c4 ca244adae0d443c1acfe2336a2a76d80 - - -] Filter AggregateImagePropertiesIsolation returned 0 hosts XXXXX 2017-03-28 03:45:30.042 1578 DEBUG nova.filters [req-42a5cb09-dcc8-4bc1-9801-93ad8c583a34 caac64b245bb480289b7951e3ff871c4 ca244adae0d443c1acfe2336a2a76d80 - - -] Filtering removed all hosts for the request with instance ID '7b576745-281f-439b-8ae2-e43e823ae815'. Filter results: [('RetryFilter', [(u'overcloud-compute-0.localdomain', u'overcloud-compute-0.localdomain')]), ('AvailabilityZoneFilter', [(u'overcloud-compute-0.localdomain', u'overcloud-compute-0.localdomain')]), ('RamFilter', [(u'overcloud-compute-0.localdomain', u'overcloud-compute-0.localdomain')]), ('DiskFilter', [(u'overcloud-compute-0.localdomain', u'overcloud-compute-0.localdomain')]), ('ComputeFilter', [(u'overcloud-compute-0.localdomain', u'overcloud-compute-0.localdomain')]), ('ComputeCapabilitiesFilter', [(u'overcloud-compute-0.localdomain', u'overcloud-compute-0.localdomain')]), ('ImagePropertiesFilter', [(u'overcloud-compute-0.localdomain', u'overcloud-compute-0.localdomain')]), ('ServerGroupAntiAffinityFilter', [(u'overcloud-compute-0.localdomain', u'overcloud-compute-0.localdomain')]), ('ServerGroupAffinityFilter', [(u'overcloud-compute-0.localdomain', u'overcloud-compute-0.localdomain')]), ('AggregateImagePropertiesIsolation', None)] get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:129 2017-03-28 03:45:30.043 1578 INFO nova.filters [req-42a5cb09-dcc8-4bc1-9801-93ad8c583a34 caac64b245bb480289b7951e3ff871c4 ca244adae0d443c1acfe2336a2a76d80 - - -] Filtering removed all hosts for the request with instance ID '7b576745-281f-439b-8ae2-e43e823ae815'. Filter results: ['RetryFilter: (start: 1, end: 1)', 'AvailabilityZoneFilter: (start: 1, end: 1)', 'RamFilter: (start: 1, end: 1)', 'DiskFilter: (start: 1, end: 1)', 'ComputeFilter: (start: 1, end: 1)', 'ComputeCapabilitiesFilter: (start: 1, end: 1)', 'ImagePropertiesFilter: (start: 1, end: 1)', 'ServerGroupAntiAffinityFilter: (start: 1, end: 1)', 'ServerGroupAffinityFilter: (start: 1, end: 1)', 'AggregateImagePropertiesIsolation: (start: 1, end: 0)'] 2017-03-28 03:45:30.043 1578 DEBUG nova.scheduler.filter_scheduler [req-42a5cb09-dcc8-4bc1-9801-93ad8c583a34 caac64b245bb480289b7951e3ff871c4 ca244adae0d443c1acfe2336a2a76d80 - - -] There are 0 hosts available but 1 instances requested to build. select_destinations /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:71 XXXXX 2017-03-28 03:45:30.043 1578 DEBUG oslo_messaging.rpc.server [req-42a5cb09-dcc8-4bc1-9801-93ad8c583a34 caac64b245bb480289b7951e3ff871c4 ca244adae0d443c1acfe2336a2a76d80 - - -] Expected exception during message handling () _process_incoming /usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py:136 ++++ Important lines are marked with "XXXXX" After a few minutes, I checked and the instance was still running on compute-1 [stack@instack ~]$ nova show inst1 +--------------------------------------+----------------------------------------------------------+ | Property | Value | +--------------------------------------+----------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | overcloud-compute-1.localdomain | | OS-EXT-SRV-ATTR:hostname | inst1 | | OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-compute-1.localdomain | | OS-EXT-SRV-ATTR:instance_name | instance-00000006 | However, when I performed the same migration from the dashboard it succeeded [stack@instack ~]$ nova show inst1 +--------------------------------------+----------------------------------------------------------+ | Property | Value | +--------------------------------------+----------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | overcloud-compute-0.localdomain | | OS-EXT-SRV-ATTR:hostname | inst1 | | OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-compute-0.localdomain | I am also attaching sos report from the controller node from my test environment and screenshots for migration from the dashboard. Thanks and Regards, Punit Kundal Red Hat OpenStack Team India
Created attachment 1266907 [details] selecting overcloud-compute-0 as the new compute node
Created attachment 1266908 [details] live migration success to overcloud-compute-0 from the dashboard
Would it hurt to send False there even when the microversion is lower than 2.30? In other words, do we need to be checking the microversion?
Given 'force' is a new attribute, the Nova API is not looking at it if the asked microversion is lower than 2.30. In that case, given we don't check it if the microversion is <2.30 [1], I think it's fine by me to ask for force=false even if the microversion is older. [1] https://github.com/openstack/nova/blob/94671e279fddc0d8eba29375750dd85410a0b9b9/nova/api/openstack/compute/migrate_server.py#L79-L80 If the microversion is lower than 2.30, then force is set to None in the Nova API so it provides the legacy behaviour (ie. not checking the scheduler). Just make sure you ask that microversion or newer.
Thank you. I asked, because we only support microversions in horizon properly in Pike, and this is a Newton bug, so it's difficult to ask for a specific microversion.
I seem to be hitting this issue with RHOSP 8 but with the CLI.
@dhill if you are hitting this with CLI, it's probably a completely different issue with similar effects. You would probably best report it to Nova.
This behavior is due to changes in the Nova API. Unfortunately, there is no consistent behavior that Horizon could get, since the options available in different microversions of Nova API are mutually exclusive. There no one default behavior that would work across all the versions.