Description of problem:
Tempest test test_cross_tenant_traffic failed during the creation of a VM instance with the following error message [1], which happens at 2021-08-11 16:49:49:
Details: {'code': 409, 'message': "Multiple security_group matches found for name 'tempest-secgroup_access--609865299', use an ID to be more specific."}


I have reviewed the tempest.log file and a security named like that is only created once, two seconds before the failed VM creation [2]:
2021-08-11 16:49:47.679 234999 INFO tempest.lib.common.rest_client [req-b6e5bb12-f490-489f-bc7a-daa45e624bbb ] Request (TestSecurityGroupsBasicOps:setUp): 201 POST http://10.0.0.112:9696/v2.0/security-groups 1.317s
2021-08-11 16:49:47.680 234999 DEBUG tempest.lib.common.rest_client [req-b6e5bb12-f490-489f-bc7a-daa45e624bbb ] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: {"security_group": {"name": "tempest-secgroup_access--609865299", "description": "tempest-secgroup_access--609865299 description", "tenant_id": "dd44686f5f974d5b81cd7c0a08728c68"}}
    Response - Headers: {'content-type': 'application/json', 'content-length': '1412', 'x-openstack-request-id': 'req-b6e5bb12-f490-489f-bc7a-daa45e624bbb', 'date': 'Wed, 11 Aug 2021 16:49:47 GMT', 'connection': 'close', 'status': '201', 'content-location': 'http://10.0.0.112:9696/v2.0/security-groups'}
        Body: b'{"security_group": {"id": "c55f7dfb-eb35-488c-918a-40f3d9c090a3", "name": "tempest-secgroup_access--609865299", "tenant_id": "dd44686f5f974d5b81cd7c0a08728c68", "description": "tempest-secgroup_access--609865299 description", "security_group_rules": [{"id": "2bd47bd7-e930-4200-bba7-eeb4be34670f", "tenant_id": "dd44686f5f974d5b81cd7c0a08728c68", "security_group_id": "c55f7dfb-eb35-488c-918a-40f3d9c090a3", "ethertype": "IPv4", "direction": "egress", "protocol": null, "port_range_min": null, "port_range_max": null, "remote_ip_prefix": null, "remote_group_id": null, "description": null, "tags": [], "created_at": "2021-08-11T16:49:47Z", "updated_at": "2021-08-11T16:49:47Z", "revision_number": 0, "project_id": "dd44686f5f974d5b81cd7c0a08728c68"}, {"id": "3ba58c3a-cab1-4020-bba9-d1c490250702", "tenant_id": "dd44686f5f974d5b81cd7c0a08728c68", "security_group_id": "c55f7dfb-eb35-488c-918a-40f3d9c090a3", "ethertype": "IPv6", "direction": "egress", "protocol": null, "port_range_min": null, "port_range_max": null, "remote_ip_prefix": null, "remote_group_id": null, "description": null, "tags": [], "created_at": "2021-08-11T16:49:47Z", "updated_at": "2021-08-11T16:49:47Z", "revision_number": 0, "project_id": "dd44686f5f974d5b81cd7c0a08728c68"}], "tags": [], "created_at": "2021-08-11T16:49:47Z", "updated_at": "2021-08-11T16:49:47Z", "revision_number": 1, "project_id": "dd44686f5f974d5b81cd7c0a08728c68"}}'





Apparently, the SG POST request generated two entries in the databases [3] (I'm not including all the logs):
2021-08-11 16:49:46.375 27 DEBUG neutron.api.v2.base [req-b6e5bb12-f490-489f-bc7a-daa45e624bbb 884c6d80284b471093e4caa27050de1c dd44686f5f974d5b81cd7c0a08728c68 - default default] Request body: {'security_group': {'name': 'tempest-secgroup_access--609865299', 'description': 'tempest-secgroup_access--609865299 description', 'tenant_id': 'dd44686f5f974d5b81cd7c0a08728c68'}} prepare_request_body /usr/lib/python3.6/site-packages/neutron/api/v2/base.py:719
2021-08-11 16:49:46.823 27 DEBUG networking_ovn.db.revision [req-b6e5bb12-f490-489f-bc7a-daa45e624bbb 884c6d80284b471093e4caa27050de1c dd44686f5f974d5b81cd7c0a08728c68 - default default] create_initial_revision uuid=3fe383b6-d6c3-4a1b-82ff-9a299852e2e3, type=security_groups, rev=-1 create_initial_revision /usr/lib/python3.6/site-packages/networking_ovn/db/revision.py:59
2021-08-11 16:49:46.925 27 DEBUG neutron_lib.db.api [req-b6e5bb12-f490-489f-bc7a-daa45e624bbb 884c6d80284b471093e4caa27050de1c dd44686f5f974d5b81cd7c0a08728c68 - default default] Retry wrapper got retriable exception: (pymysql.err.InternalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction')
2021-08-11 16:49:46.927 27 DEBUG neutron.api.rpc.handlers.resources_rpc [req-b6e5bb12-f490-489f-bc7a-daa45e624bbb 884c6d80284b471093e4caa27050de1c dd44686f5f974d5b81cd7c0a08728c68 - - -] Pushing event updated for resources: {'SecurityGroup': ['ID=3fe383b6-d6c3-4a1b-82ff-9a299852e2e3,revision_number=1']} push /usr/lib/python3.6/site-packages/neutron/api/rpc/handlers/resources_rpc.py:243

2021-08-11 16:49:47.426 27 DEBUG neutron.api.v2.base [req-b6e5bb12-f490-489f-bc7a-daa45e624bbb 884c6d80284b471093e4caa27050de1c dd44686f5f974d5b81cd7c0a08728c68 - default default] Request body: {'security_group': {'name': 'tempest-secgroup_access--609865299', 'description': 'tempest-secgroup_access--609865299 description', 'tenant_id': 'dd44686f5f974d5b81cd7c0a08728c68'}} prepare_request_body /usr/lib/python3.6/site-packages/neutron/api/v2/base.py:719
2021-08-11 16:49:47.573 27 DEBUG networking_ovn.db.revision [req-b6e5bb12-f490-489f-bc7a-daa45e624bbb 884c6d80284b471093e4caa27050de1c dd44686f5f974d5b81cd7c0a08728c68 - default default] create_initial_revision uuid=c55f7dfb-eb35-488c-918a-40f3d9c090a3, type=security_groups, rev=-1 create_initial_revision /usr/lib/python3.6/site-packages/networking_ovn/db/revision.py:59
2021-08-11 16:49:47.680 27 DEBUG neutron.api.rpc.handlers.resources_rpc [req-b6e5bb12-f490-489f-bc7a-daa45e624bbb 884c6d80284b471093e4caa27050de1c dd44686f5f974d5b81cd7c0a08728c68 - - -] Pushing event updated for resources: {'SecurityGroup': ['ID=c55f7dfb-eb35-488c-918a-40f3d9c090a3,revision_number=1']} push /usr/lib/python3.6/site-packages/neutron/api/rpc/handlers/resources_rpc.py:243


So, the SG is created with IDs 3fe383b6-d6c3-4a1b-82ff-9a299852e2e3 and c55f7dfb-eb35-488c-918a-40f3d9c090a3. The response to the SG POST reguest only included the second one, but then the VM creation fails because there are two SG named tempest-secgroup_access--609865299






[1] https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/network/view/networking-ovn/job/DFG-network-networking-ovn-16.1_director-rhel-virthost-3cont_2comp-ipv4-geneve/561/testReport/tempest.scenario.test_security_groups_basic_ops/TestSecurityGroupsBasicOps/test_cross_tenant_traffic_compute_id_e79f879e_debb_440c_a7e4_efeda05b6848_network_/
[2] http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-networking-ovn-16.1_director-rhel-virthost-3cont_2comp-ipv4-geneve/561/undercloud-0/home/stack/tempest-dir/tempest.log.gz
[3] http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-networking-ovn-16.1_director-rhel-virthost-3cont_2comp-ipv4-geneve/561/controller-2/var/log/containers/neutron/server.log.2.gz


Version-Release number of selected component (if applicable):
RHOS-16.1-RHEL-8-20210804.n.0


How reproducible:
It was reproduced once with RHOS-16.1-RHEL-8-20210804.n.0 and once with RHOS-16.1-RHEL-8-20210727.n.1 only and on different tests


Steps to Reproduce:
1. run the job (run the whole tempest neutron suite). I don't have a simpler way to reproduce, since it doesn't reproduce often and it can be reproduced by different tests
2.
3.