Bug 2089420

Description of problem:
The first operation I performed on an Openstack 17.0 environment after the overcloud was installed is to create an external network (see task "create the external network with vlan transparency":
http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/staging/DFG-network-networking-ovn-17.0_director-rhel-virthost-3cont_2comp-ipv4-vlan-sriov/54/console_logs/ir-cloud-config.log

The following error was obtained:
Error while executing command: HttpException: 500, Request Failed: internal server error while processing your request.


The following exception is printed in the controller-1 neutron server logs:
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc [req-f6cfe61c-3179-451e-b6c7-339441ff18a5 d883b14b25014525867759a08c317401 84df20cacdd84d34ae3b9dac48dcf83d - - -] Exception while dispatching security_group events: dictionary changed size during iteration: RuntimeError: dictionary changed size during iteration
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc Traceback (most recent call last):
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/ovo_rpc.py", line 131, in dispatch_events
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc     self._resource_push_api.push(context, [obj], rpc_event)
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc   File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/resources_rpc.py", line 245, in push
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc     self._push(context, resource_type, type_resources, event_type)
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc   File "/usr/lib/python3.9/site-packages/neutron/api/rpc/handlers/resources_rpc.py", line 251, in _push
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc     for version in version_manager.get_resource_versions(resource_type):
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc   File "/usr/lib/python3.9/site-packages/neutron/api/rpc/callbacks/version_manager.py", line 250, in get_resource_versions
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc     return _get_cached_tracker().get_resource_versions(resource_type)
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc   File "/usr/lib/python3.9/site-packages/neutron/api/rpc/callbacks/version_manager.py", line 226, in get_resource_versions
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc     self._check_expiration()
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc   File "/usr/lib/python3.9/site-packages/neutron/api/rpc/callbacks/version_manager.py", line 222, in _check_expiration
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc     self._update_consumer_versions()
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc   File "/usr/lib/python3.9/site-packages/neutron/api/rpc/callbacks/version_manager.py", line 211, in _update_consumer_versions
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc     neutron_plugin.get_agents_resource_versions(new_tracker)
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc   File "/usr/lib/python3.9/site-packages/neutron/db/agents_db.py", line 467, in get_agents_resource_versions
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc     for agent in self._get_agents_considered_for_versions():
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc   File "/usr/lib/python3.9/site-packages/neutron/db/agents_db.py", line 453, in _get_agents_considered_for_versions
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc     up_agents = self.get_agents(context.get_admin_context(),
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 1255, in fn
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc     return op(results, new_method(*args, _driver=self, **kwargs))
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 1318, in get_agents
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc     for agent in n_agent.AgentCache():
2022-05-23 15:13:20.732 17 ERROR neutron.plugins.ml2.ovo_rpc RuntimeError: dictionary changed size during iteration



The event ChassisMetadataAgentWriteEvent was called right before this traceback, so maybe the number of agents changed.


I tried to reproduce it after that, and I couldn't. My guess is that either:
- it's only reproducible the first time a network is created after the overcloud installation or
- it's an unlikely race condition or
- the overcloud installation was not really completed, although the installation had finished some minutes before, according to infrared


This is the command that reproduced the issue:
openstack network create public    --external --transparent-vlan --share    --provider-network-type vlan    --provider-segment 319    --provider-physical-network datacentre


Version-Release number of selected component (if applicable):
RHOS-17.0-RHEL-9-20220519.n.1

How reproducible:
only once