Bug 2242427

Description of problem: FFU from 16.2.5 to 17.1 failed at step 5 of `openstack overcloud upgrade run --yes --stack <stack> --debug --limit allovercloud,undercloud --playbook all` because neutron api was down. 

Looking at the container and neutron servers, it was crash because it was unable to load 2 extensions. (network_availability_zone and router_availability_zone). 

Looking at the code, I could see that they both required the extension `availability_zone`. 

Futherer up the logs we can see what look like neutron not loading the extenision. 
~~~
2023-10-06 01:06:22.981 7 DEBUG neutron.api.extensions [req-a0f02f38-42f1-4f56-80c7-25993ec6e599 - - - - -] Ext name="Availability Zone" alias="availability_zone" description="The availability zone extension." updated="2015-01-01T10:00:00-00:00" _check_extension /usr/lib/python3.9/site-packages/neutron/api/extensions.py:416
2023-10-06 01:06:22.981 7 INFO neutron.api.extensions [req-a0f02f38-42f1-4f56-80c7-25993ec6e599 - - - - -] Extension availability_zone not supported by any of loaded plugins
~~~

From what I can tell from the code all 3 should be support from the ovn core extensions


Version-Release number of selected component (if applicable):

registry.redhat.io/rhosp-rhel9/openstack-neutron-server:17.1.1-3

How reproducible:

Unsure have only reproduced during FFU so far 

Steps to Reproduce:
1. FFU from  16.2.5 with octaiva
(Not sure it necessary with Octavia, it failed with the deploy steps Octavia because the neutron api was down)

Actual results:

Neutron api are down because the neutron container is crashing

Expected results:

Neutron APIs to be up

Additional info:

# LOGS
egrep -r "(router_availability_zone|network_availability_zone)" /var/log/containers/neutron/server.log
2023-10-06 01:01:31.018 8 DEBUG neutron.api.extensions [req-874f95de-2952-4cc7-b48d-61e028978113 - - - - -] Loading extension file: network_availability_zone.py _load_all_extensions_from_path /usr/lib/python3.9/site-packages/neutron/api/extensions.py:450
2023-10-06 01:01:31.018 8 DEBUG neutron.api.extensions [req-874f95de-2952-4cc7-b48d-61e028978113 - - - - -] Ext name="Network Availability Zone" alias="network_availability_zone" description="Availability zone support for network." updated="2015-01-01T10:00:00-00:00" _check_extension /usr/lib/python3.9/site-packages/neutron/api/extensions.py:416
2023-10-06 01:01:31.019 8 INFO neutron.api.extensions [req-874f95de-2952-4cc7-b48d-61e028978113 - - - - -] Loaded extension: network_availability_zone
2023-10-06 01:01:31.057 8 DEBUG neutron.api.extensions [req-874f95de-2952-4cc7-b48d-61e028978113 - - - - -] Loading extension file: router_availability_zone.py _load_all_extensions_from_path /usr/lib/python3.9/site-packages/neutron/api/extensions.py:450
2023-10-06 01:01:31.057 8 DEBUG neutron.api.extensions [req-874f95de-2952-4cc7-b48d-61e028978113 - - - - -] Ext name="Router Availability Zone" alias="router_availability_zone" description="Availability zone support for router." updated="2015-01-01T10:00:00-00:00" _check_extension /usr/lib/python3.9/site-packages/neutron/api/extensions.py:416
2023-10-06 01:01:31.058 8 INFO neutron.api.extensions [req-874f95de-2952-4cc7-b48d-61e028978113 - - - - -] Loaded extension: router_availability_zone
2023-10-06 01:01:31.105 8 ERROR neutron.api.extensions [req-874f95de-2952-4cc7-b48d-61e028978113 - - - - -] Unable to process extensions (router_availability_zone, network_availability_zone) because the configured plugins do not satisfy their requirements. Some features will not work as expected.
2023-10-06 01:01:31.105 8 ERROR neutron.service [req-874f95de-2952-4cc7-b48d-61e028978113 - - - - -] Unrecoverable error: please check log for details.: neutron_lib.exceptions.ExtensionsNotFound: Extensions not found: ['router_availability_zone', 'network_availability_zone'].



2023-10-06 01:06:22.981 7 DEBUG neutron.api.extensions [req-a0f02f38-42f1-4f56-80c7-25993ec6e599 - - - - -] Ext name="Availability Zone" alias="availability_zone" description="The availability zone extension." updated="2015-01-01T10:00:00-00:00" _check_extension /usr/lib/python3.9/site-packages/neutron/api/extensions.py:416
2023-10-06 01:06:22.981 7 INFO neutron.api.extensions [req-a0f02f38-42f1-4f56-80c7-25993ec6e599 - - - - -] Extension availability_zone not supported by any of loaded plugins


# ML2 config

[ml2]
type_drivers=geneve,vxlan,vlan,flat
tenant_network_types=geneve,vxlan,vlan,flat
mechanism_drivers=ovn
path_mtu=0
extension_drivers=qos,port_security,dns
overlay_ip_version=4

[ml2_type_geneve]
max_header_size=38
vni_ranges=1:65536

[ml2_type_vxlan]
vxlan_group=224.0.0.1
vni_ranges=1:65536

[ml2_type_vlan]
network_vlan_ranges=tenant:1000:2000

[ml2_type_flat]
flat_networks=datacentre

[ovn]
ovn_nb_connection=ssl:172.17.1.89:6641,ssl:172.17.1.25:6641,ssl:172.17.1.34:6641
ovn_sb_connection=ssl:172.17.1.89:6642,ssl:172.17.1.25:6642,ssl:172.17.1.34:6642
ovn_nb_private_key=/etc/pki/tls/private/neutron_ovn.key
ovn_nb_certificate=/etc/pki/tls/certs/neutron_ovn.crt
ovn_nb_ca_cert=/etc/ipa/ca.crt
ovn_sb_private_key=/etc/pki/tls/private/neutron_ovn.key
ovn_sb_certificate=/etc/pki/tls/certs/neutron_ovn.crt
ovn_sb_ca_cert=/etc/ipa/ca.crt
ovsdb_connection_timeout=180
ovsdb_probe_interval=60000
neutron_sync_mode=log
ovn_metadata_enabled=True
enable_distributed_floating_ip=False
dns_servers=
ovn_emit_need_to_frag=False

[network_log]
rate_limit=100
burst_limit=25

# Code
/lib/python3.9/site-packages/neutron/extensions/network_availability_zone.py:

from neutron_lib.api.definitions import availability_zone as az_def
from neutron_lib.api.definitions import network

ALIAS = 'network_availability_zone'
IS_SHIM_EXTENSION = False
IS_STANDARD_ATTR_EXTENSION = False
NAME = 'Network Availability Zone'
API_PREFIX = ''
DESCRIPTION = 'Availability zone support for network.'
UPDATED_TIMESTAMP = '2015-01-01T10:00:00-00:00'
RESOURCE_NAME = network.RESOURCE_NAME
COLLECTION_NAME = network.COLLECTION_NAME
RESOURCE_ATTRIBUTE_MAP = {
    COLLECTION_NAME: {
        az_def.COLLECTION_NAME: {
            'allow_post': False, 'allow_put': False,
            'is_visible': True
        },
        az_def.AZ_HINTS: {
            'allow_post': True, 'allow_put': False, 'is_visible': True,
            'is_sort_key': True,
            'validate': {'type:availability_zone_hint_list': None},
            'default': []
        }
    }
}
SUB_RESOURCE_ATTRIBUTE_MAP = {}
ACTION_MAP = {}
REQUIRED_EXTENSIONS = [az_def.ALIAS]
OPTIONAL_EXTENSIONS = []
ACTION_STATUS = {}


/lib/python3.9/site-packages/neutron/extensions/router_availability_zone.py


from neutron_lib.api.definitions import availability_zone as az
from neutron_lib.api.definitions import l3


ALIAS = 'router_availability_zone'
IS_SHIM_EXTENSION = False
IS_STANDARD_ATTR_EXTENSION = False
NAME = 'Router Availability Zone'
API_PREFIX = ''
DESCRIPTION = 'Availability zone support for router.'
UPDATED_TIMESTAMP = '2015-01-01T10:00:00-00:00'
RESOURCE_NAME = l3.ROUTER
COLLECTION_NAME = l3.ROUTERS
RESOURCE_ATTRIBUTE_MAP = {
    COLLECTION_NAME: {
        az.COLLECTION_NAME: {
            'allow_post': False, 'allow_put': False,
            'is_visible': True
        },
        az.AZ_HINTS: {
            'allow_post': True, 'allow_put': False, 'is_visible': True,
            'validate': {'type:availability_zone_hint_list': None},
            'default': []
        }
    }
}
SUB_RESOURCE_ATTRIBUTE_MAP = {}
ACTION_MAP = {}
REQUIRED_EXTENSIONS = [l3.ALIAS, az.ALIAS]
OPTIONAL_EXTENSIONS = []
ACTION_STATUS = {}

/lib/python3.9/site-packages/neutron/extensions/availability_zone.py

import abc

from neutron_lib.api.definitions import availability_zone as az_def
from neutron_lib.api import extensions as api_extensions
from neutron_lib.plugins import directory

from neutron.api import extensions
from neutron.api.v2 import base


class Availability_zone(api_extensions.APIExtensionDescriptor):
    """Availability zone extension."""
    api_definition = az_def

    @classmethod
    def get_resources(cls):
        """Returns Ext Resources."""
        plugin = directory.get_plugin()
        params = az_def.RESOURCE_ATTRIBUTE_MAP.get(az_def.COLLECTION_NAME)
        controller = base.create_resource(az_def.COLLECTION_NAME,
                                          az_def.RESOURCE_NAME, plugin, params)

        ex = extensions.ResourceExtension(az_def.COLLECTION_NAME, controller)

        return [ex]


class AvailabilityZonePluginBase(object, metaclass=abc.ABCMeta):
    """REST API to operate the Availability Zone."""

    @abc.abstractmethod
    def get_availability_zones(self, context, filters=None, fields=None,
                               sorts=None, limit=None, marker=None,
                               page_reverse=False):
        """Return availability zones which a resource belongs to"""

    @abc.abstractmethod
    def validate_availability_zones(self, context, resource_type,
                                    availability_zones):
        """Verify that the availability zones exist."""



/lib/python3.9/site-packages/neutron/common/ovn/extensions.py

from neutron_lib.api.definitions import address_scope
from neutron_lib.api.definitions import agent as agent_def
from neutron_lib.api.definitions import allowedaddresspairs
from neutron_lib.api.definitions import auto_allocated_topology
from neutron_lib.api.definitions import availability_zone as az_def  <---------------- Availability zone extension
from neutron_lib.api.definitions import default_subnetpools
from neutron_lib.api.definitions import dns
from neutron_lib.api.definitions import dns_domain_ports
from neutron_lib.api.definitions import expose_port_forwarding_in_fip
from neutron_lib.api.definitions import external_net
from neutron_lib.api.definitions import extra_dhcp_opt
from neutron_lib.api.definitions import extraroute
from neutron_lib.api.definitions import filter_validation
from neutron_lib.api.definitions import fip_pf_description
from neutron_lib.api.definitions import fip_port_details
from neutron_lib.api.definitions import flavors
from neutron_lib.api.definitions import floating_ip_port_forwarding
from neutron_lib.api.definitions import floatingip_pools
from neutron_lib.api.definitions import l3
from neutron_lib.api.definitions import l3_ext_gw_mode
from neutron_lib.api.definitions import l3_flavors
from neutron_lib.api.definitions import logging
from neutron_lib.api.definitions import multiprovidernet
from neutron_lib.api.definitions import network_availability_zone <---------------- Network Availability Zone
from neutron_lib.api.definitions import network_ip_availability
from neutron_lib.api.definitions import network_mtu
from neutron_lib.api.definitions import network_mtu_writable
from neutron_lib.api.definitions import pagination
from neutron_lib.api.definitions import port_device_profile
from neutron_lib.api.definitions import port_mac_address_regenerate
from neutron_lib.api.definitions import port_numa_affinity_policy
from neutron_lib.api.definitions import port_resource_request
from neutron_lib.api.definitions import port_security
from neutron_lib.api.definitions import portbindings
from neutron_lib.api.definitions import portbindings_extended as pbe_ext
from neutron_lib.api.definitions import project_id
from neutron_lib.api.definitions import provider_net
from neutron_lib.api.definitions import qos
from neutron_lib.api.definitions import qos_bw_limit_direction
from neutron_lib.api.definitions import qos_default
from neutron_lib.api.definitions import qos_gateway_ip
from neutron_lib.api.definitions import qos_rule_type_details
from neutron_lib.api.definitions import qos_rules_alias
from neutron_lib.api.definitions import rbac_address_scope
from neutron_lib.api.definitions import rbac_security_groups
from neutron_lib.api.definitions import router_availability_zone as raz_def
from neutron_lib.api.definitions import security_groups_normalized_cidr
from neutron_lib.api.definitions import security_groups_remote_address_group
from neutron_lib.api.definitions import segment as seg_def
from neutron_lib.api.definitions import sorting
from neutron_lib.api.definitions import stateful_security_group
from neutron_lib.api.definitions import subnet_dns_publish_fixed_ip
from neutron_lib.api.definitions import trunk
from neutron_lib.api.definitions import vlantransparent
from neutron_lib import constants

from neutron.extensions import security_groups_shared_filtering_lib

# NOTE(russellb) This remains in its own file (vs constants.py) because we want
# to be able to easily import it and export the info without any dependencies
# on external imports.

# NOTE(russellb) If you update these lists, please also update
# doc/source/admin/ovn/features.rst and the current release note.
ML2_SUPPORTED_API_EXTENSIONS_OVN_L3 = [
    l3.ALIAS,
    extraroute.ALIAS,
    l3_ext_gw_mode.ALIAS,
    fip_port_details.ALIAS,
    floatingip_pools.ALIAS,
    pagination.ALIAS,
    'qos-fip',
    qos_gateway_ip.ALIAS,
    sorting.ALIAS,
    project_id.ALIAS,
    dns.ALIAS,
    dns_domain_ports.ALIAS,
    subnet_dns_publish_fixed_ip.ALIAS,
    agent_def.ALIAS,
    az_def.ALIAS,     <---------------- Availability zone extension
    raz_def.ALIAS,   <---------------- Router Availability Zone
    flavors.ALIAS,
    l3_flavors.ALIAS,
]
ML2_SUPPORTED_API_EXTENSIONS = [
    address_scope.ALIAS,
    agent_def.ALIAS,
    allowedaddresspairs.ALIAS,
    auto_allocated_topology.ALIAS,
    portbindings.ALIAS,
    pbe_ext.ALIAS,
    default_subnetpools.ALIAS,
    dns.ALIAS,
    external_net.ALIAS,
    extra_dhcp_opt.ALIAS,
    filter_validation.ALIAS,
    multiprovidernet.ALIAS,
    network_mtu.ALIAS,
    network_mtu_writable.ALIAS,
    network_availability_zone.ALIAS, <---------------- Network Availability Zone
    network_ip_availability.ALIAS,
    port_device_profile.ALIAS,
    port_mac_address_regenerate.ALIAS,
    port_numa_affinity_policy.ALIAS,
    port_security.ALIAS,
    provider_net.ALIAS,
    port_resource_request.ALIAS,
    qos.ALIAS,
    qos_bw_limit_direction.ALIAS,
    qos_default.ALIAS,
    qos_rule_type_details.ALIAS,
    qos_rules_alias.ALIAS,
    'quotas',
    rbac_address_scope.ALIAS,
    'rbac-policies',
    rbac_security_groups.ALIAS,
    'standard-attr-revisions',
    'security-group',
    security_groups_normalized_cidr.ALIAS,
    security_groups_remote_address_group.ALIAS,
    stateful_security_group.ALIAS,
    security_groups_shared_filtering_lib.ALIAS,
    'standard-attr-description',
    constants.SUBNET_ALLOCATION_EXT_ALIAS,
    'standard-attr-tag',
    'standard-attr-timestamp',
    trunk.ALIAS,
    'quota_details',
    seg_def.ALIAS,
    expose_port_forwarding_in_fip.ALIAS,
   fip_pf_description.ALIAS,
    floating_ip_port_forwarding.ALIAS,
    vlantransparent.ALIAS,
    logging.ALIAS,
]