+++ This bug was initially created as a clone of Bug #1851797 +++

Description of problem:
Failed to create VM with watchdog by setting flavor metadata

Version-Release number of selected component (if applicable):
openstack-nova-compute-20.2.1-0.20200528080027.1e95025.el8ost.noarch
libvirt-daemon-kvm-6.0.0-23.module+el8.2.1+6955+1e1fca42.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Create a image/volume in OSP16.1, set the image/volume metedata hw_watchdog_action: pause, create flavor: m2, start the VM from image/volume with watchdog successfully, the xml is as below:
    <watchdog model='i6300esb' action='pause'>
      <alias name='watchdog0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </watchdog>

2. Set the flavor:m2 with hw_watchdog_action: pause, try to start the VM from the same image/volume with m2 on same compute node, failed with message:
   "No valid host was found. There are not enough hosts available"
-----------------------------------------------------------------------
[heat-admin@overcloud-controller-0 ~]$ sudo tail -f /var/log/containers/nova/nova-scheduler.log
2020-06-29 02:24:10.513 47 INFO nova.filters [req-a8e4515d-44ef-4020-83d5-9888a5776b8b 317f1c1fe439476dbfb991ec7768d23f 63a5b71d92f74e0ba553e6acca2c747c - default default] Filter AggregateInstanceExtraSpecsFilter returned 0 hosts
2020-06-29 02:24:10.514 47 INFO nova.filters [req-a8e4515d-44ef-4020-83d5-9888a5776b8b 317f1c1fe439476dbfb991ec7768d23f 63a5b71d92f74e0ba553e6acca2c747c - default default] Filtering removed all hosts for the request with instance ID '554daf2f-88f1-4171-ad01-fc236958219d'. Filter results: ['AvailabilityZoneFilter: (start: 4, end: 1)', 'ComputeFilter: (start: 1, end: 1)', 'ComputeCapabilitiesFilter: (start: 1, end: 1)', 'ImagePropertiesFilter: (start: 1, end: 1)', 'ServerGroupAntiAffinityFilter: (start: 1, end: 1)', 'ServerGroupAffinityFilter: (start: 1, end: 1)', 'NUMATopologyFilter: (start: 1, end: 1)', 'AggregateInstanceExtraSpecsFilter: (start: 1, end: 0)']  
------------------------------------------------------------------------

Actual results:
In step2, failed to start VM with flavor metadata: hw_watchdog_action: pause

Expected results:
In step2, can start VM with flavor metadata: hw_watchdog_action: pause successfully

--- Additional comment from  on 2020-07-03 14:35:08 UTC ---

there is not enough info in this bug to triage it
when filing bug you should always include sosreports if available
the current failure is due to the AggregateInstanceExtraSpecsFilter which likely is related to how its configure

at a minimum  we need the metadata for the aggregate and the flavor info but really we need the controller  logs

--- Additional comment from  on 2020-07-06 09:45:48 UTC ---

Please see the information as below:

1.  Create a image in OSP16.1, set the image metedata hw_watchdog_action: pause, create flavor: m2, 
    start the VM from image/volume with watchdog successfully, the xml is as below:
    <watchdog model='i6300esb' action='pause'>
      <alias name='watchdog0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </watchdog>

- image: r8 metadata
| properties       | direct_url='file:///var/lib/glance/images/181ea5e5-c345-4200-b205-8a6dd8d66d91', hw_watchdog_action='pause', os_hash_algo='sha512', os_hash_value='527d0a141dc1968b39d5528cf10223bb7849c77d0bc3cc67ed8024789b4c61aec6feaa444cd3ad0a305d80fd4ed1ebdc8f164527dc3d1437d3c04321fd76a78d', os_hidden='False', stores='default_backend' |

- flavor: m2 metadata (no item)
$ openstack flavor show m2
+----------------------------+-------+
| Field                      | Value |
+----------------------------+-------+
| OS-FLV-DISABLED:disabled   | False |
......
| properties                 |       |
......
  
2. Set the flavor:m2 with hw_watchdog_action: pause, 
   try to start the VM from the same image with m2 on same compute node, failed with message in nova-conductor.log:
   "No valid host was found. There are not enough hosts available"

- flavor: m2 metadata
$ openstack flavor show m2
+----------------------------+---------------------------------+
| Field                      | Value                           |
+----------------------------+---------------------------------+
| OS-FLV-DISABLED:disabled   | False                           |
| OS-FLV-EXT-DATA:ephemeral  | 0                               |
| access_project_ids         | None                            |
| description                | None                            |
| disk                       | 10                              |
| extra_specs                | {'hw_watchdog_action': 'pause'} |
| id                         | 7                               |
| name                       | m2                              |
| os-flavor-access:is_public | True                            |
| properties                 | hw_watchdog_action='pause'      |
| ram                        | 2048                            |
| rxtx_factor                | 1.0                             |
| swap                       | 0                               |
| vcpus                      | 2                               |
+----------------------------+---------------------------------+

$ cat nova-scheduler.log 
2020-07-06 09:39:38.916 50 INFO nova.filters [req-b271c72f-99f6-4d83-927d-1448c2b7bf1e 317f1c1fe439476dbfb991ec7768d23f 63a5b71d92f74e0ba553e6acca2c747c - default default] Filter AggregateInstanceExtraSpecsFilter returned 0 hosts
2020-07-06 09:39:38.917 50 INFO nova.filters [req-b271c72f-99f6-4d83-927d-1448c2b7bf1e 317f1c1fe439476dbfb991ec7768d23f 63a5b71d92f74e0ba553e6acca2c747c - default default] Filtering removed all hosts for the request with instance ID '474ca5db-73f5-436f-9e78-0df943edfc4c'. Filter results: ['AvailabilityZoneFilter: (start: 4, end: 1)', 'ComputeFilter: (start: 1, end: 1)', 'ComputeCapabilitiesFilter: (start: 1, end: 1)', 'ImagePropertiesFilter: (start: 1, end: 1)', 'ServerGroupAntiAffinityFilter: (start: 1, end: 1)', 'ServerGroupAffinityFilter: (start: 1, end: 1)', 'NUMATopologyFilter: (start: 1, end: 1)', 'AggregateInstanceExtraSpecsFilter: (start: 1, end: 0)']

More details in attached log file. Thank you!

--- Additional comment from  on 2020-07-06 09:46:21 UTC ---



--- Additional comment from  on 2020-07-06 09:47:34 UTC ---



--- Additional comment from  on 2020-07-06 10:05:07 UTC ---

More information:

$ openstack aggregate list
+----+------+-------------------+
| ID | Name | Availability Zone |
+----+------+-------------------+
|  1 | numa | numa              |
|  2 | vgpu | vgpu              |
+----+------+-------------------+

In step 1 and 2, both try to start VM from image and select Availability Zone: nova in web console.

--- Additional comment from  on 2020-07-06 11:23:47 UTC ---

hw_watchdog_action is the way it would be set in an image not the valid form for a flavor.

flavors have namespaced extra specs in this case hw:watchdog_action  images do not support namespacs
so we flatten them converting hw:watchdog_action to hw_watchdog_action


so as specifed the hw_watchdog_action in the flaovr should be ignored with regrades to setting the values in the xml.

however as hw_watchdog_action is an unnamespaced extra spec it will be chekced by the AggregateInstanceExtraSpecsFilter
and if it is not listed on a host aggreate it will reject the host.

so im closing this as not a bug as the flavor is invaild and hte AggregateInstanceExtraSpecsFilter is correctly filtering the
host. The AggregateInstanceExtraSpecsFilter checks all unnamespaced extra specs or those that start with aggregate_instance_extra_specs:

if you change hw_watchdog_action to hw:watchdog_action it should work.

--- Additional comment from  on 2020-07-07 01:37:39 UTC ---

(In reply to smooney from comment #6)
> hw_watchdog_action is the way it would be set in an image not the valid form
> for a flavor.
> 
> flavors have namespaced extra specs in this case hw:watchdog_action  images
> do not support namespacs
> so we flatten them converting hw:watchdog_action to hw_watchdog_action
> 
> 
> so as specifed the hw_watchdog_action in the flaovr should be ignored with
> regrades to setting the values in the xml.
> 
> however as hw_watchdog_action is an unnamespaced extra spec it will be
> chekced by the AggregateInstanceExtraSpecsFilter
> and if it is not listed on a host aggreate it will reject the host.
> 

Hi, Sean

This is the problem, I updated the flavor metadata on OSP dashboard,
"Update Flavor Metadata -> Watchdog Behavior -> Watchdog Action -> Click + ",
it added the "hw_watchdog_action" in flavor.

I think at least the dashboard need fix, to add hw:watchdog_action but not hw_watchdog_action,
we can't provide a way to customer, and then say you can't do things like that.

Regards,
Chenli Hu

--- Additional comment from  on 2020-07-07 01:45:23 UTC ---



--- Additional comment from  on 2020-07-07 01:50:41 UTC ---



--- Additional comment from  on 2020-07-07 01:52:05 UTC ---



--- Additional comment from  on 2020-07-07 06:05:55 UTC ---

I move it to openstack-dashboard-theme, please correct it if it's not suitable, thank you!

--- Additional comment from  on 2020-07-07 12:49:12 UTC ---

this is a glance bug
https://github.com/openstack/glance/blob/master/etc/metadefs/compute-watchdog.json

    "resource_type_associations": [
        {
            "name": "OS::Glance::Image"
        },
        {
            "name": "OS::Cinder::Volume",
            "properties_target": "image"
        },
        {
            "name": "OS::Nova::Flavor"
        }
    ],

should be

   "resource_type_associations": [
        {
            "name": "OS::Glance::Image",
            "prefix": "hw_"
        },
        {
            "name": "OS::Nova::Flavor",
            "prefix": "hw:"
        }
    ],

and the hw_ prefix should be dropped from the property.

https://github.com/openstack/glance/blob/master/etc/metadefs/compute-watchdog.json#L20

horizon generates the ui form the glance metadefs.

--- Additional comment from Cyril Roelandt on 2020-07-09 17:48:46 UTC ---

Are we supposed to remove OS::Cinder::Volume as well? Or is the patch below OK?

To be honest, I'm not sure I understand the issue very well. Can you open a bug upstream? If so, I'll make sure the relevant patch gets backported.

diff --git a/etc/metadefs/compute-watchdog.json b/etc/metadefs/compute-watchdog.json
index a8e9e43a..cbe39696 100644
--- a/etc/metadefs/compute-watchdog.json
+++ b/etc/metadefs/compute-watchdog.json
@@ -6,18 +6,20 @@
     "protected": true,
     "resource_type_associations": [
         {
-            "name": "OS::Glance::Image"
+            "name": "OS::Glance::Image",
+            "prefix": "hw_"
         },
         {
             "name": "OS::Cinder::Volume",
             "properties_target": "image"
         },
         {
-            "name": "OS::Nova::Flavor"
+            "name": "OS::Nova::Flavor",
+            "prefix": "hw:"
         }
     ],
     "properties": {
-        "hw_watchdog_action": {
+        "watchdog_action": {
             "title": "Watchdog Action",
             "description": "For the libvirt driver, you can enable and set the behavior of a virtual hardware watchdog device for each flavor. Watchdog devices keep an eye on the guest server, and carry out the configured action, if the server hangs. The watchdog uses the i6300esb device (emulating a PCI Intel 6300ESB). If hw_watchdog_action is not specified, the watchdog is disabled. Watchdog behavior set using a specific image's properties will override behavior set using flavors.",
             "type": "string",

--- Additional comment from  on 2020-07-09 20:34:41 UTC ---

image metadata stored in the volume is valid but it would have the same prefix as image e.g. "hw_"
im not sure if you need to specify that or not.

basicaly horizon uses the glance metadta api to retirve a catalog of metadata keys and in some cases validations which it then uses to dynamically generate uis.

so if you ever wondered where the ui you get for setting image metadata or flavor extra specs comes form its glances metadef api.
glance was selected to maintain the catalog of metadata definitons just after i started working on openstack about 6 years ago.
https://github.com/openstack/glance/commit/1c242032fbb26fed3a82691abb030583b4f8940b
this file was added in that inital commit and it looks like its always been wrong.

the watchdog is a very rarely used feature and even more rare for the feature to be configured via horizon.

so basically since the metadef was incorrect the

based on the example response in the api docs i belive that

volume definition should look like this

{
          "name": "OS::Cinder::Volume",
          "prefix": "hw_",
          "properties_target": "image",
}

see https://docs.openstack.org/glance/latest/user/glancemetadefcatalogapi.html#retrieve-namespace

and yes feel free to file an upstream bug to track this.

this is a glance feature that many people dont know is there.
the orginial intent was that clients could hit that endpoint and automaticaly fiture out what metadtaa was valid and generate valiation, commandlines and documentation/ui elements.
which is why in correct behavior in horizon needs to be fixed in glance.

--- Additional comment from Cyril Roelandt on 2020-07-10 00:58:05 UTC ---

OK I sent a patch upstream (see https://review.opendev.org/#/c/740384) and added you to the reviewer list.

--- Additional comment from RHEL Program Management on 2020-09-24 20:39:29 UTC ---

This bugzilla has been removed from the release since it  does not have an acked release flag. For details, see https://mojo.redhat.com/docs/DOC-1144661#jive_content_id_OSP_Release_Planning.'

--- Additional comment from RHEL Program Management on 2020-10-01 19:35:46 UTC ---

This bugzilla has been removed from the release since it  does not have an acked release flag. For details, see https://mojo.redhat.com/docs/DOC-1144661#jive_content_id_OSP_Release_Planning.'

--- Additional comment from RHEL Program Management on 2020-10-01 19:43:42 UTC ---

This bugzilla has been removed from the release since it  does not have an acked release flag. For details, see https://mojo.redhat.com/docs/DOC-1144661#jive_content_id_OSP_Release_Planning.'

--- Additional comment from RHEL Program Management on 2020-10-05 15:24:12 UTC ---

This item has been properly Triaged and planned for the release, and Target Release is now set to match the release flag. For details, see https://mojo.redhat.com/docs/DOC-1195410

--- Additional comment from Giulio Fidente on 2020-11-03 09:55:58 UTC ---

Cyril, I don't see the stable/train backport, is that on purpose? If so I think we need the downstream cherry-pick

--- Additional comment from RHEL Program Management on 2020-11-09 17:02:09 UTC ---

This item has been properly Triaged and planned for the release, and Target Release is now set to match the release flag. For details, see https://mojo.redhat.com/docs/DOC-1195410

--- Additional comment from Cyril Roelandt on 2020-11-10 20:11:33 UTC ---

@Giulio: Yes, that is not going to be fixed upstream, so it's a downstream-only backport.