Bug 1435597
| Summary: | AggegateImagePropertiesIsolatefiler not working | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Md Nadeem <mnadeem> |
| Component: | openstack-nova | Assignee: | Sylvain Bauza <sbauza> |
| Status: | CLOSED NOTABUG | QA Contact: | Prasanth Anbalagan <panbalag> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 10.0 (Newton) | CC: | astupnik, berrange, dasmith, eglynn, ipetrova, kchamart, mnadeem, mschuppe, pbarta, sbauza, sferdjao, sgordon, srevivo, vromanso |
| Target Milestone: | --- | Keywords: | Reopened |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-02-09 16:10:17 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
The issue is produce with OSP9 and OSP10 too. (In reply to Md Nadeem from comment #1) > The issue is produce with OSP9 and OSP10 too. Do you have a cute=False value set on the other aggregate? No, on other aggregate i have set devil=True Firstly, unrelated to the problem, you should never set some aggregate metadata information to availability_zone=nova if 'nova' is your default availability zone. Please see https://docs.openstack.org/developer/nova/aggregates.html#availability-zones-azs for further details about why it can be error-prone. Back to the problem, you're saying that some instances using the exact same flavor are being deployed on compute1. Could you please try to reproduce the issue and provide nova-api and nova-scheduler logs accordingly ? Ideally a sos-report would be nice because I'd like to see nova.conf as well. Also, could you please make sure that you restarted nova-scheduler after modifying nova.conf on the node running nova-scheduler ? Okay, I have a customer that upgraded from RHOS-8 to RHOS-9 and the AggegateImagePropertiesIsolatefiler got broken.
The regression is quite easy to reproduce:
----------------------------------------------------------------------------
in RHOS-8 [AggegateImagePropertiesIsolatefiler *does* work]:
----------------------------------------------------------------------------
[stack@undercloud-0 ~]$ for i in $(nova list 2>/dev/null |awk -F= '/Running/ {print substr($2,0,length($2)-2)}'); do ssh heat-admin@$i 'hostname; sudo grep "^scheduler_default_filters" /etc/nova/nova.conf'; done
compute-0
scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter
compute-1
scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter
controller-0
scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,AggregateImagePropertiesIsolation
controller-1
scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,AggregateImagePropertiesIsolation
controller-2
scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,AggregateImagePropertiesIsolation
>> the AggregateImagePropertiesIsolation is enabled in the Controllers (still works, no need to explicitly put the filter in the Computes as well)
[stack@undercloud-0 ~]$ rpm -qa |grep nova
python-nova-12.0.6-17.el7ost.noarch
openstack-nova-conductor-12.0.6-17.el7ost.noarch
openstack-nova-cert-12.0.6-17.el7ost.noarch
openstack-nova-common-12.0.6-17.el7ost.noarch
python-novaclient-3.1.0-2.el7ost.noarch
openstack-nova-compute-12.0.6-17.el7ost.noarch
openstack-nova-scheduler-12.0.6-17.el7ost.noarch
openstack-nova-api-12.0.6-17.el7ost.noarch
[stack@undercloud-0 ~]$ openstack aggregate list --long 2>/dev/null
+----+----------+-------------------+-------------------+
| ID | Name | Availability Zone | Properties |
+----+----------+-------------------+-------------------+
| 7 | aggr-one | None | {u'type': u'one'} |
| 10 | aggr-two | None | {u'type': u'abc'} |
+----+----------+-------------------+-------------------+
[stack@undercloud-0 ~]$ openstack aggregate show aggr-one 2>/dev/null |egrep "hosts|name|properties"
| hosts | [u'compute-0.localdomain'] |
| name | aggr-one |
| properties | {u'type': u'one'} |
[stack@undercloud-0 ~]$ openstack aggregate show aggr-two 2>/dev/null |egrep "hosts|name|properties"
| hosts | [u'compute-1.localdomain'] |
| name | aggr-two |
| properties | {u'type': u'abc'} |
[stack@undercloud-0 ~]$ openstack image show cirros 2>/dev/null |egrep "id|properties"
| id | 34512f8f-10ff-4bb0-a1f2-12577a6e0ce3 |
| properties | type='abc' |
[stack@undercloud-0 ~]$ openstack image show cirros_2 2>/dev/null |egrep "id|properties"
| id | 79e51773-71f9-4762-a59b-c8aaf39b877a |
| properties | type='one' |
[stack@undercloud-0 ~]$ for i in {1..10}; do echo test-vm-$i; openstack server show test-vm-$i 2>/dev/null |egrep -w "OS-EXT-SRV-ATTR:host|^\| name|image"; done
test-vm-1
| OS-EXT-SRV-ATTR:host | compute-1.localdomain |
| image | cirros (34512f8f-10ff-4bb0-a1f2-12577a6e0ce3) |
| name | test-vm-1 |
test-vm-2
| OS-EXT-SRV-ATTR:host | compute-1.localdomain |
| image | cirros (34512f8f-10ff-4bb0-a1f2-12577a6e0ce3) |
| name | test-vm-2 |
test-vm-3
| OS-EXT-SRV-ATTR:host | compute-1.localdomain |
| image | cirros (34512f8f-10ff-4bb0-a1f2-12577a6e0ce3) |
| name | test-vm-3 |
test-vm-4
| OS-EXT-SRV-ATTR:host | compute-1.localdomain |
| image | cirros (34512f8f-10ff-4bb0-a1f2-12577a6e0ce3) |
| name | test-vm-4 |
test-vm-5
| OS-EXT-SRV-ATTR:host | compute-1.localdomain |
| image | cirros (34512f8f-10ff-4bb0-a1f2-12577a6e0ce3) |
| name | test-vm-5 |
test-vm-6
| OS-EXT-SRV-ATTR:host | compute-0.localdomain |
| image | cirros_2 (79e51773-71f9-4762-a59b-c8aaf39b877a) |
| name | test-vm-6 |
test-vm-7
| OS-EXT-SRV-ATTR:host | compute-0.localdomain |
| image | cirros_2 (79e51773-71f9-4762-a59b-c8aaf39b877a) |
| name | test-vm-7 |
test-vm-8
| OS-EXT-SRV-ATTR:host | compute-0.localdomain |
| image | cirros_2 (79e51773-71f9-4762-a59b-c8aaf39b877a) |
| name | test-vm-8 |
test-vm-9
| OS-EXT-SRV-ATTR:host | compute-0.localdomain |
| image | cirros_2 (79e51773-71f9-4762-a59b-c8aaf39b877a) |
| name | test-vm-9 |
test-vm-10
| OS-EXT-SRV-ATTR:host | compute-0.localdomain |
| image | cirros_2 (79e51773-71f9-4762-a59b-c8aaf39b877a) |
| name | test-vm-10 |
[stack@undercloud-0 ~]$
>> the Nova scheduler respects the AggregateImagePropertiesIsolation filter and schedules the instances accordingly
----------------------------------------------------------------------------
in RHOS-10 [AggegateImagePropertiesIsolatefiler does *NOT* work any longer]:
----------------------------------------------------------------------------
[stack@undercloud-0 ~]$ for i in $(nova list 2>/dev/null |awk -F= '/Running/ {print substr($2,0,length($2)-2)}'); do ssh heat-admin@$i 'hostname; sudo grep "cheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,AggregateImagePropertiesIsolation" /etc/nova/nova.conf'; done
compute-0
scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,AggregateImagePropertiesIsolation
compute-1
scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,AggregateImagePropertiesIsolation
controller-0
scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,AggregateImagePropertiesIsolation
controller-1
scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,AggregateImagePropertiesIsolation
controller-2
scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,AggregateImagePropertiesIsolation
[stack@undercloud-0 ~]$
>> the AggregateImagePropertiesIsolation is enabled in *ALL* hosts (just in case; nonetheless, it does *NOT* work); also notice that I'm grepping for the exact same filter (str) I have used for the RHOS-8 environment
[stack@undercloud-0 ~]$ rpm -qa |grep nova
openstack-nova-conductor-14.0.8-2.el7ost.noarch
puppet-nova-9.6.0-1.el7ost.noarch
openstack-nova-api-14.0.8-2.el7ost.noarch
openstack-nova-common-14.0.8-2.el7ost.noarch
openstack-nova-scheduler-14.0.8-2.el7ost.noarch
openstack-nova-cert-14.0.8-2.el7ost.noarch
python-nova-14.0.8-2.el7ost.noarch
openstack-nova-compute-14.0.8-2.el7ost.noarch
python-novaclient-6.0.1-1.el7ost.noarch
[stack@undercloud-0 ~]$ openstack aggregate list --long
+----+----------+-------------------+-------------------+
| ID | Name | Availability Zone | Properties |
+----+----------+-------------------+-------------------+
| 8 | aggr-one | None | {u'type': u'one'} |
| 11 | aggr-two | None | {u'type': u'abc'} |
+----+----------+-------------------+-------------------+
[stack@undercloud-0 ~]$ openstack aggregate show aggr-one |egrep "hosts|name|properties"
| hosts | [u'compute-0.localdomain'] |
| name | aggr-one |
| properties | type='one' |
[stack@undercloud-0 ~]$ openstack aggregate show aggr-two |egrep "hosts|name|properties"
| hosts | [u'compute-1.localdomain'] |
| name | aggr-two |
| properties | type='abc' |
[stack@undercloud-0 ~]$ openstack image show cirros |egrep "id|properties"
| id | 4b8b2af5-4018-40f7-92f0-1a443a342ddf |
| properties | direct_url='swift+config://ref1/glance/4b8b2af5-4018-40f7-92f0-1a443a342ddf', type='abc' |
[stack@undercloud-0 ~]$ openstack image show cirros_2 |egrep "id|properties"
| id | 6f2cc161-b840-435c-a75c-b87cbc735696 |
| properties | direct_url='swift+config://ref1/glance/6f2cc161-b840-435c-a75c-b87cbc735696', type='one' |
[stack@undercloud-0 ~]$ for i in {1..10}; do echo test-vm-$i; openstack server show test-vm-$i 2>/dev/null |egrep -w "OS-EXT-SRV-ATTR:host|^\| name|image"; done
test-vm-1
| OS-EXT-SRV-ATTR:host | compute-1.localdomain |
| image | cirros (4b8b2af5-4018-40f7-92f0-1a443a342ddf) |
| name | test-vm-1 |
test-vm-2
| OS-EXT-SRV-ATTR:host | compute-0.localdomain |
| image | cirros (4b8b2af5-4018-40f7-92f0-1a443a342ddf) |
| name | test-vm-2 |
test-vm-3
| OS-EXT-SRV-ATTR:host | compute-1.localdomain |
| image | cirros (4b8b2af5-4018-40f7-92f0-1a443a342ddf) |
| name | test-vm-3 |
test-vm-4
| OS-EXT-SRV-ATTR:host | compute-0.localdomain |
| image | cirros (4b8b2af5-4018-40f7-92f0-1a443a342ddf) |
| name | test-vm-4 |
test-vm-5
| OS-EXT-SRV-ATTR:host | compute-1.localdomain |
| image | cirros (4b8b2af5-4018-40f7-92f0-1a443a342ddf) |
| name | test-vm-5 |
test-vm-6
| OS-EXT-SRV-ATTR:host | compute-0.localdomain |
| image | cirros_2 (6f2cc161-b840-435c-a75c-b87cbc735696) |
| name | test-vm-6 |
test-vm-7
| OS-EXT-SRV-ATTR:host | compute-1.localdomain |
| image | cirros_2 (6f2cc161-b840-435c-a75c-b87cbc735696) |
| name | test-vm-7 |
test-vm-8
| OS-EXT-SRV-ATTR:host | compute-0.localdomain |
| image | cirros_2 (6f2cc161-b840-435c-a75c-b87cbc735696) |
| name | test-vm-8 |
test-vm-9
| OS-EXT-SRV-ATTR:host | compute-1.localdomain |
| image | cirros_2 (6f2cc161-b840-435c-a75c-b87cbc735696) |
| name | test-vm-9 |
test-vm-10
| OS-EXT-SRV-ATTR:host | compute-0.localdomain |
| image | cirros_2 (6f2cc161-b840-435c-a75c-b87cbc735696) |
| name | test-vm-10 |
[stack@undercloud-0 ~]$
>> the Nova scheduler obviously selects the compute hosts in round-robin! It couldn't care less about my filter.
reproducer steps on OSP9 env:
* Add AggregateImagePropertiesIsolation to scheduler_default_filters
scheduler_default_filters=AggregateImagePropertiesIsolation,RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter
=> restart nova scheduler
* create host aggregates
[stack@undercloud-0 ~]$ nova aggregate-create agr-one
+----+---------+-------------------+-------+----------+
| Id | Name | Availability Zone | Hosts | Metadata |
+----+---------+-------------------+-------+----------+
| 3 | agr-one | - | | |
+----+---------+-------------------+-------+----------+
[stack@undercloud-0 ~]$ nova aggregate-create agr-two
+----+---------+-------------------+-------+----------+
| Id | Name | Availability Zone | Hosts | Metadata |
+----+---------+-------------------+-------+----------+
| 6 | agr-two | - | | |
+----+---------+-------------------+-------+----------+
* set metadata
[stack@undercloud-0 ~]$ nova aggregate-set-metadata agr-one os=linux
Metadata has been successfully updated for aggregate 3.
+----+---------+-------------------+-------+----------+
| Id | Name | Availability Zone | Hosts | Metadata |
+----+---------+-------------------+-------+----------+
| 3 | agr-one | - | | 'os=linux' |
+----+---------+-------------------+-------+----------+
[stack@undercloud-0 ~]$ nova aggregate-set-metadata agr-two os=windows
Metadata has been successfully updated for aggregate 6.
+----+---------+-------------------+-------+----------+
| Id | Name | Availability Zone | Hosts | Metadata |
+----+---------+-------------------+-------+----------+
| 6 | agr-two | - | | 'os=windows' |
+----+---------+-------------------+-------+----------+
* add computes to the agr
[stack@undercloud-0 ~]$ nova aggregate-add-host agr-one compute-0.localdomain
Host compute-0.localdomain has been successfully added for aggregate 3
+----+---------+-------------------+-------------------------+----------+
| Id | Name | Availability Zone | Hosts | Metadata |
+----+---------+-------------------+-------------------------+----------+
| 3 | agr-one | - | 'compute-0.localdomain' | 'os=linux' |
+----+---------+-------------------+-------------------------+----------+
[stack@undercloud-0 ~]$ nova aggregate-add-host agr-two compute-1.localdomain
Host compute-1.localdomain has been successfully added for aggregate 6
+----+---------+-------------------+-------------------------+----------+
| Id | Name | Availability Zone | Hosts | Metadata |
+----+---------+-------------------+-------------------------+----------+
| 6 | agr-two | - | 'compute-1.localdomain' | 'os=windows' |
+----+---------+-------------------+-------------------------+----------+
* set property to the image
[stack@undercloud-0 ~]$ glance image-update cirros --property os=linux
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| Property 'os' | linux |
| checksum | 4bda4108d1a74dd73a6ae6d0ba369916 |
| container_format | bare |
| created_at | 2017-07-29T11:05:42.000000 |
| deleted | False |
| deleted_at | None |
| disk_format | raw |
| id | e0c662b1-0e51-4753-b38d-27f1a43a32b8 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | cirros |
| owner | a126a6b887d6452bb4c8ae99774a07ef |
| protected | False |
| size | 41126400 |
| status | active |
| updated_at | 2017-10-05T09:09:07.000000 |
| virtual_size | None |
+------------------+--------------------------------------+
[stack@undercloud-0 ~]$ glance image-update cirros2 --property os=windows
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| Property 'os' | windows |
| checksum | 4bda4108d1a74dd73a6ae6d0ba369916 |
| container_format | bare |
| created_at | 2017-10-05T09:51:17.000000 |
| deleted | False |
| deleted_at | None |
| disk_format | raw |
| id | 05a631fd-b814-4617-890b-628a9535f538 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | cirros2 |
| owner | a126a6b887d6452bb4c8ae99774a07ef |
| protected | False |
| size | 41126400 |
| status | active |
| updated_at | 2017-10-05T09:52:18.000000 |
| virtual_size | None |
+------------------+--------------------------------------+
Note: not set aggregate_image_properties_isolation_namespace or aggregate_image_properties_isolation_separator in /etc/nova/nova.conf
$ openstack server create --flavor m1.tiny --image e0c662b1-0e51-4753-b38d-27f1a43a32b8 --nic net-id=8008d543-d9e3-464c-99c0-7b25a24c330d test-one-1
+--------------------------------------+-----------------------------------------------+
| Field | Value |
+--------------------------------------+-----------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | None |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| OS-EXT-SRV-ATTR:instance_name | instance-000000ae |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| adminPass | zP97zVfrpA6H |
| config_drive | |
| created | 2017-10-05T15:08:49Z |
| flavor | m1.tiny (1) |
| hostId | |
| id | 8a0933a4-177b-4625-a8c2-f2f53efe084f |
| image | cirros (e0c662b1-0e51-4753-b38d-27f1a43a32b8) |
| key_name | None |
| name | test-one-1 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| project_id | a126a6b887d6452bb4c8ae99774a07ef |
| properties | |
| security_groups | [{u'name': u'default'}] |
| status | BUILD |
| updated | 2017-10-05T15:08:50Z |
| user_id | fb4114459eec4a039caaace6ad1c2583 |
+--------------------------------------+-----------------------------------------------+
Note: had to recreate the aggregates, therefore the Ids in the log differs from the ids above
scheduler log when created above instance:
* agr-one update info:
2017-10-05 15:08:50.219 29063 DEBUG nova.scheduler.host_manager [req-ccae57fb-a30e-4b66-99bb-aa64f45e8e0b fb4114459eec4a039caaace6ad1c2583 a126a6b887d6452bb4c8ae99774a07ef - - -] Update host state with aggregates: [Aggregate(created_at=2017-10-05T14:54:25Z,deleted=False,deleted_at=None,hosts=['compute-0.localdomain'],id=12,metadata={os='linux'},name='agr-one',updated_at=None,uuid=c28f6d7d-4661-4da5-9332-49b287efdc87)] _locked_update /usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py:171
* agr-two update info:
2017-10-05 15:08:50.239 29063 DEBUG nova.scheduler.host_manager [req-ccae57fb-a30e-4b66-99bb-aa64f45e8e0b fb4114459eec4a039caaace6ad1c2583 a126a6b887d6452bb4c8ae99774a07ef - - -] Update host state with aggregates: [Aggregate(created_at=2017-10-05T14:53:37Z,deleted=False,deleted_at=None,hosts=['compute-1.localdomain'],id=9,metadata={os='windows'},name='agr-two',updated_at=None,uuid=51496acf-835f-4c1f-b0c7-d6b3c92f9605)] _locked_update /usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py:171
* filter fails to get the image metadata, or wrong config?
2017-10-05 15:08:50.244 29063 WARNING nova.scheduler.filters.aggregate_image_properties_isolation [req-ccae57fb-a30e-4b66-99bb-aa64f45e8e0b fb4114459eec4a039caaace6ad1c2583 a126a6b887d6452bb4c8ae99774a07ef - - -] Host 'compute-1.localdomain' has a metadata key 'os' that is not present in the image metadata.
2017-10-05 15:08:50.246 29063 WARNING nova.scheduler.filters.aggregate_image_properties_isolation [req-ccae57fb-a30e-4b66-99bb-aa64f45e8e0b fb4114459eec4a039caaace6ad1c2583 a126a6b887d6452bb4c8ae99774a07ef - - -] Host 'compute-0.localdomain' has a metadata key 'os' that is not present in the image metadata.
* because of the above we get all nodes as possible dst hosts:
2017-10-05 15:08:50.247 29063 DEBUG nova.filters [req-ccae57fb-a30e-4b66-99bb-aa64f45e8e0b fb4114459eec4a039caaace6ad1c2583 a126a6b887d6452bb4c8ae99774a07ef - - -] Filter AggregateImagePropertiesIsolation returned 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104
When we dump the image meta props in the filter we see:
2017-10-05 15:17:35.436 1988 DEBUG nova.scheduler.filters.aggregate_image_properties_isolation [req-d177cadb-70c5-4b71-bfe9-6965f3c75dcb fb4114459eec4a039caaace6ad1c2583 a126a6b887d6452bb4c8ae99774a07ef - - -] RH_DEBUG: image prop = ImageMetaProps(hw_architecture=<?>,hw_auto_disk_config=<?>,hw_boot_menu=<?>,hw_cdrom_bus=<?>,hw_cpu_cores=<?>,hw_cpu_max_cores=<?>,hw_cpu_max_sockets=<?>,hw_cpu_max_threads=<?>,hw_cpu_policy=<?>,hw_cpu_realtime_mask=<?>,hw_cpu_sockets=<?>,hw_cpu_thread_policy=<?>,hw_cpu_threads=<?>,hw_device_id=<?>,hw_disk_bus=<?>,hw_disk_type=<?>,hw_firmware_type=<?>,hw_floppy_bus=<?>,hw_ipxe_boot=<?>,hw_machine_type=<?>,hw_mem_page_size=<?>,hw_numa_cpus=<?>,hw_numa_mem=<?>,hw_numa_nodes=<?>,hw_qemu_guest_agent=<?>,hw_rng_model=<?>,hw_scsi_model=<?>,hw_serial_port_count=<?>,hw_video_model=<?>,hw_video_ram=<?>,hw_vif_model=<?>,hw_vif_multiqueue_enabled=<?>,hw_vm_mode=<?>,hw_watchdog_action=<?>,img_bdm_v2=<?>,img_bittorrent=<?>,img_block_device_mapping=<?>,img_cache_in_nova=<?>,img_compression_level=<?>,img_config_drive=<?>,img_hv_requested_version=<?>,img_hv_type=<?>,img_linked_clone=<?>,img_mappings=<?>,img_owner_id=<?>,img_root_device_name=<?>,img_signature=<?>,img_signature_certificate_uuid=<?>,img_signature_hash_method=<?>,img_signature_key_type=<?>,img_use_agent=<?>,img_version=<?>,os_admin_user=<?>,os_command_line=<?>,os_distro=<?>,os_require_quiesce=<?>,os_skip_agent_inject_files_at_boot=<?>,os_skip_agent_inject_ssh=<?>,os_type=<?>) host_passes /usr/lib/python2.7/site-packages/nova/scheduler/filters/aggregate_image_properties_isolation.py:51
Got it working with e.g. property os_distro, so it seems we only get the list of pre-defined fields from the ImageMetaProps class
$ nova aggregate-set-metadata 9 os_distro=windows
$ nova aggregate-set-metadata 12 os_distro=rhel
$ glance image-update e0c662b1-0e51-4753-b38d-27f1a43a32b8 --property os_distro=rhel
$ glance image-update 05a631fd-b814-4617-890b-628a9535f538 --property os_distro=windows
2017-10-05 15:30:18.193 22369 DEBUG nova.scheduler.host_manager [req-fa7e50f4-c568-47d0-885c-38f3c9ba4da5 fb4114459eec4a039caaace6ad1c2583 a126a6b887d6452bb4c8ae99774a07ef - - -] Update host state with aggregates: [Aggregate(created_at=2017-10-05T14:54:25Z,deleted=False,deleted_at=None,hosts=['compute-0.localdomain'],id=12,metadata={os='linux',os_distro='rhel'},name='agr-one',updated_at=None,uuid=c28f6d7d-4661-4da5-9332-49b287efdc87)] _locked_update /usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py:171
2017-10-05 15:30:18.215 22369 DEBUG nova.scheduler.host_manager [req-fa7e50f4-c568-47d0-885c-38f3c9ba4da5 fb4114459eec4a039caaace6ad1c2583 a126a6b887d6452bb4c8ae99774a07ef - - -] Update host state with aggregates: [Aggregate(created_at=2017-10-05T14:53:37Z,deleted=False,deleted_at=None,hosts=['compute-1.localdomain'],id=9,metadata={os='windows',os_distro='windows'},name='agr-two',updated_at=None,uuid=51496acf-835f-4c1f-b0c7-d6b3c92f9605)] _locked_update /usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py:171
* here we see the os_distro=rhel
2017-10-05 15:30:18.224 22369 DEBUG nova.scheduler.filters.aggregate_image_properties_isolation [req-fa7e50f4-c568-47d0-885c-38f3c9ba4da5 fb4114459eec4a039caaace6ad1c2583 a126a6b887d6452bb4c8ae99774a07ef - - -] RH_DEBUG: image prop = ImageMetaProps(hw_architecture=<?>,hw_auto_disk_config=<?>,hw_boot_menu=<?>,hw_cdrom_bus=<?>,hw_cpu_cores=<?>,hw_cpu_max_cores=<?>,hw_cpu_max_sockets=<?>,hw_cpu_max_threads=<?>,hw_cpu_policy=<?>,hw_cpu_realtime_mask=<?>,hw_cpu_sockets=<?>,hw_cpu_thread_policy=<?>,hw_cpu_threads=<?>,hw_device_id=<?>,hw_disk_bus=<?>,hw_disk_type=<?>,hw_firmware_type=<?>,hw_floppy_bus=<?>,hw_ipxe_boot=<?>,hw_machine_type=<?>,hw_mem_page_size=<?>,hw_numa_cpus=<?>,hw_numa_mem=<?>,hw_numa_nodes=<?>,hw_qemu_guest_agent=<?>,hw_rng_model=<?>,hw_scsi_model=<?>,hw_serial_port_count=<?>,hw_video_model=<?>,hw_video_ram=<?>,hw_vif_model=<?>,hw_vif_multiqueue_enabled=<?>,hw_vm_mode=<?>,hw_watchdog_action=<?>,img_bdm_v2=<?>,img_bittorrent=<?>,img_block_device_mapping=<?>,img_cache_in_nova=<?>,img_compression_level=<?>,img_config_drive=<?>,img_hv_requested_version=<?>,img_hv_type=<?>,img_linked_clone=<?>,img_mappin
gs=<?>,img_owner_id=<?>,img_root_device_name=<?>,img_signature=<?>,img_signature_certificate_uuid=<?>,img_signature_hash_method=<?>,img_signature_key_type=<?>,img_use_agent=<?>,img_version=<?>,os_admin_user=<?>,os_command_line=<?>,os_distro='rhel',os_require_quiesce=<?>,os_skip_agent_inject_files_at_boot=<?>,os_skip_agent_inject_ssh=<?>,os_type=<?>) host_passes /usr/lib/python2.7/site-packages/nova/scheduler/filters/aggregate_image_properties_isolation.py:51
* filter just returned a single compute
2017-10-05 15:30:18.226 22369 DEBUG nova.filters [req-fa7e50f4-c568-47d0-885c-38f3c9ba4da5 fb4114459eec4a039caaace6ad1c2583 a126a6b887d6452bb4c8ae99774a07ef - - -] Filter AggregateImagePropertiesIsolation returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104
$ nova show 30ab311b-ed30-4b48-be19-d66665c8dea1 |grep OS-EXT-SRV-ATTR:host
| OS-EXT-SRV-ATTR:host | compute-0.localdomain |
| OS-EXT-SRV-ATTR:hostname | test-one-1 |
$ for i in `seq 2 10`; do openstack server create --flavor m1.tiny --image e0c662b1-0e51-4753-b38d-27f1a43a32b8 --nic net-id=8008d543-d9e3-464c-99c0-7b25a24c330d test-one-$i; done
$ for i in `seq 1 10`; do nova show test-one-$i |egrep "OS-EXT-SRV-ATTR:host| fault"; done
| OS-EXT-SRV-ATTR:host | compute-0.localdomain |
| OS-EXT-SRV-ATTR:hostname | test-one-1 |
| OS-EXT-SRV-ATTR:host | compute-0.localdomain |
| OS-EXT-SRV-ATTR:hostname | test-one-2 |
| OS-EXT-SRV-ATTR:host | compute-0.localdomain |
| OS-EXT-SRV-ATTR:hostname | test-one-3 |
| OS-EXT-SRV-ATTR:host | compute-0.localdomain |
| OS-EXT-SRV-ATTR:hostname | test-one-4 |
| OS-EXT-SRV-ATTR:host | compute-0.localdomain |
| OS-EXT-SRV-ATTR:hostname | test-one-5 |
| OS-EXT-SRV-ATTR:host | compute-0.localdomain |
| OS-EXT-SRV-ATTR:hostname | test-one-6 |
| OS-EXT-SRV-ATTR:host | compute-0.localdomain |
| OS-EXT-SRV-ATTR:hostname | test-one-7 |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hostname | test-one-8 |
| fault | {"message": "No valid host was found. There are not enough hosts available.", "code": 500, "details": " File \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 404, in build_instances |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hostname | test-one-9 |
| fault | {"message": "No valid host was found. There are not enough hosts available.", "code": 500, "details": " File \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 404, in build_instances |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hostname | test-one-10 |
| fault | {"message": "No valid host was found. There are not enough hosts available.", "code": 500, "details": " File \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 404, in build_instances |
OSP8 test - works with custom prop 'os'
[root@osp8-controller ~(keystone_admin)]# grep ^scheduler_default_filters /etc/nova/nova.conf
scheduler_default_filters=AggregateImagePropertiesIsolation,RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter
[root@osp8-controller ~(keystone_admin)]# nova aggregate-create agr-one
+----+---------+-------------------+-------+----------+
| Id | Name | Availability Zone | Hosts | Metadata |
+----+---------+-------------------+-------+----------+
| 4 | agr-one | - | | |
+----+---------+-------------------+-------+----------+
[root@osp8-controller ~(keystone_admin)]# nova aggregate-create agr-two
+----+---------+-------------------+-------+----------+
| Id | Name | Availability Zone | Hosts | Metadata |
+----+---------+-------------------+-------+----------+
| 5 | agr-two | - | | |
+----+---------+-------------------+-------+----------+
[root@osp8-controller ~(keystone_admin)]# nova aggregate-set-metadata agr-one os=linux
Metadata has been successfully updated for aggregate 4.
+----+---------+-------------------+-------+------------+
| Id | Name | Availability Zone | Hosts | Metadata |
+----+---------+-------------------+-------+------------+
| 4 | agr-one | - | | 'os=linux' |
+----+---------+-------------------+-------+------------+
[root@osp8-controller ~(keystone_admin)]# nova aggregate-set-metadata agr-two os=windows
Metadata has been successfully updated for aggregate 5.
+----+---------+-------------------+-------+--------------+
| Id | Name | Availability Zone | Hosts | Metadata |
+----+---------+-------------------+-------+--------------+
| 5 | agr-two | - | | 'os=windows' |
+----+---------+-------------------+-------+--------------+
[root@osp8-controller ~(keystone_admin)]# nova aggregate-add-host agr-one osp8-compute1
Host osp8-compute1 has been successfully added for aggregate 4
+----+---------+-------------------+-----------------+------------+
| Id | Name | Availability Zone | Hosts | Metadata |
+----+---------+-------------------+-----------------+------------+
| 4 | agr-one | - | 'osp8-compute1' | 'os=linux' |
+----+---------+-------------------+-----------------+------------+
[root@osp8-controller ~(keystone_admin)]# nova aggregate-add-host agr-two osp8-compute2
Host osp8-compute2 has been successfully added for aggregate 5
+----+---------+-------------------+-----------------+--------------+
| Id | Name | Availability Zone | Hosts | Metadata |
+----+---------+-------------------+-----------------+--------------+
| 5 | agr-two | - | 'osp8-compute2' | 'os=windows' |
+----+---------+-------------------+-----------------+--------------+
[root@osp8-controller filters(keystone_admin)]# nova aggregate-details 4
+----+---------+-------------------+-----------------+------------+
| Id | Name | Availability Zone | Hosts | Metadata |
+----+---------+-------------------+-----------------+------------+
| 4 | agr-one | - | 'osp8-compute1' | 'os=linux' |
+----+---------+-------------------+-----------------+------------+
[root@osp8-controller filters(keystone_admin)]# nova aggregate-details 5
+----+---------+-------------------+-----------------+--------------+
| Id | Name | Availability Zone | Hosts | Metadata |
+----+---------+-------------------+-----------------+--------------+
| 5 | agr-two | - | 'osp8-compute2' | 'os=windows' |
+----+---------+-------------------+-----------------+--------------+
[root@osp8-controller ~(keystone_admin)]# glance image-show baea1209-75c7-4daf-8a51-0f8061e2c715
+------------------+--------------------------------------------------------------------+
| Property | Value |
+------------------+--------------------------------------------------------------------+
| checksum | 56730d3091a764d5f8b38feeef0bfcef |
| container_format | bare |
| created_at | 2017-10-06T09:25:38Z |
| direct_url | file:///var/lib/glance/images/baea1209-75c7-4daf-8a51-0f8061e2c715 |
| disk_format | raw |
| id | baea1209-75c7-4daf-8a51-0f8061e2c715 |
| min_disk | 0 |
| min_ram | 0 |
| name | cirros-ceph1 |
| os | linux |
| owner | 6741168c17524c24b79ce5af6373a7c4 |
| protected | False |
| size | 41126400 |
| status | active |
| tags | [] |
| updated_at | 2017-10-06T09:25:56Z |
| virtual_size | None |
| visibility | private |
+------------------+--------------------------------------------------------------------+
[root@osp8-controller ~(keystone_admin)]# glance image-update 1cd02867-1428-4fc8-9a83-b4692da88cec --property os=windows
+------------------+--------------------------------------------------------------------+
| Property | Value |
+------------------+--------------------------------------------------------------------+
| checksum | 56730d3091a764d5f8b38feeef0bfcef |
| container_format | bare |
| created_at | 2017-10-06T09:24:15Z |
| direct_url | file:///var/lib/glance/images/1cd02867-1428-4fc8-9a83-b4692da88cec |
| disk_format | raw |
| id | 1cd02867-1428-4fc8-9a83-b4692da88cec |
| min_disk | 0 |
| min_ram | 0 |
| name | cirros-ceph2 |
| os | windows |
| owner | 6741168c17524c24b79ce5af6373a7c4 |
| protected | False |
| size | 41126400 |
| status | active |
| tags | [] |
| updated_at | 2017-10-06T09:26:06Z |
| virtual_size | None |
| visibility | private |
+------------------+--------------------------------------------------------------------+
[root@osp8-controller ~(keystone_admin)]# openstack server create --flavor m1.tiny --image baea1209-75c7-4daf-8a51-0f8061e2c715 --nic net-id=c026ee53-a02a-4ee3-aec6-f11fa6b64a68 test-one-1
2017-10-06 05:33:56.008 8686 DEBUG nova.scheduler.filters.aggregate_image_properties_isolation [req-eeb12f8c-e46a-4f4c-9f94-6b8c47a55f6a 7de1d918a88540ee9cf9a15d845dad8f 6741168c17524c24b79ce5af6373a7c4 - - -] RH_DEBUG: image prop = {u'os': u'linux'} host_passes /usr/lib/python2.7/site-packages/nova/scheduler/filters/aggregate_image_properties_isolation.py:57
[root@osp8-controller filters(keystone_admin)]# for i in `seq 2 10`; do openstack server create --flavor m1.tiny --image baea1209-75c7-4daf-8a51-0f8061e2c715 --nic net-id=c026ee53-a02a-4ee3-aec6-f11fa6b64a68 test-one-$i; done
[root@osp8-controller filters(keystone_admin)]# for i in `seq 1 10`; do nova show test-one-$i |egrep "OS-EXT-SRV-ATTR:host| fault"; done
| OS-EXT-SRV-ATTR:host | osp8-compute1 |
| OS-EXT-SRV-ATTR:host | osp8-compute1 |
| OS-EXT-SRV-ATTR:host | osp8-compute1 |
| OS-EXT-SRV-ATTR:host | osp8-compute1 |
| OS-EXT-SRV-ATTR:host | osp8-compute1 |
| OS-EXT-SRV-ATTR:host | osp8-compute1 |
| OS-EXT-SRV-ATTR:host | osp8-compute1 |
| OS-EXT-SRV-ATTR:host | osp8-compute1 |
| OS-EXT-SRV-ATTR:host | osp8-compute1 |
| OS-EXT-SRV-ATTR:host | osp8-compute1 |
|
Description of problem: I'm trying to use the AggregateImagePropertiesIsolation scheduler filter to provision all instance spawned from specific image to a single host (compute node 1 in my case). However its not working as expected. Please correct me if i have missed any steps or configuration. I have OSP7 setup, 1 controller and 2 compute node, I have created nova host aggregate "test" and add compute node1 to it, similarly created another aggregate and add compute node0 to it. I have update the glance image property by same key-value pair(cute=True) that i have used for host aggregate "test", then enable the "AggregateImagePropertiesIsolation" in nova.conf as mentioned below. >>scheduler_default_filters=AggregateInstanceExtraSpecsFilter,AggregateImagePropertiesIsolation,RetryFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter However, when i spawned the multiple instance using that image, some instance still gets provisioned on compute node0, however i expect all instance should be provisioned on compute node1 only. [stack@instack ~]$ nova aggregate-details test +----+------+-------------------+-----------------------------------+---------------------------------------+ | Id | Name | Availability Zone | Hosts | Metadata | +----+------+-------------------+-----------------------------------+---------------------------------------+ | 1 | test | nova | 'overcloud-compute-1.localdomain' | 'availability_zone=nova', 'cute=True' | +----+------+-------------------+-----------------------------------+---------------------------------------+ [stack@instack ~]$ glance image-show 768cc00e-9834-44f5-92de-22c707da8088 +------------------+--------------------------------------+ | Property | Value | +------------------+--------------------------------------+ | Property 'cute' | True | | checksum | 23a57eae58e8993e3ca22b2fe98ad1d4 | | container_format | bare | | created_at | 2017-03-09T16:12:27.000000 | | deleted | False | | disk_format | qcow2 | | id | 768cc00e-9834-44f5-92de-22c707da8088 | | is_public | True | | min_disk | 0 | | min_ram | 0 | | name | cirros | | owner | 3775ee3e76974f3c9596be37cec56bfb | | protected | False | | size | 15859712 | | status | active | | updated_at | 2017-03-23T15:33:39.000000 | +------------------+--------------------------------------+ Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: