Bug 1993130
| Summary: | Security group wrongly created twice | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Eduardo Olivares <eolivare> | |
| Component: | openstack-neutron | Assignee: | Rodolfo Alonso <ralonsoh> | |
| Status: | CLOSED ERRATA | QA Contact: | Eduardo Olivares <eolivare> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 16.1 (Train) | CC: | chrisw, ralonsoh, scohen | |
| Target Milestone: | --- | Keywords: | Triaged | |
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-neutron-15.2.1-1.20210903133311.el8ost | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 2000848 (view as bug list) | Environment: | ||
| Last Closed: | 2022-03-24 10:56:13 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat OpenStack Platform 16.1 (openstack-neutron) security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0990 |
Description of problem: Tempest test test_cross_tenant_traffic failed during the creation of a VM instance with the following error message [1], which happens at 2021-08-11 16:49:49: Details: {'code': 409, 'message': "Multiple security_group matches found for name 'tempest-secgroup_access--609865299', use an ID to be more specific."} I have reviewed the tempest.log file and a security named like that is only created once, two seconds before the failed VM creation [2]: 2021-08-11 16:49:47.679 234999 INFO tempest.lib.common.rest_client [req-b6e5bb12-f490-489f-bc7a-daa45e624bbb ] Request (TestSecurityGroupsBasicOps:setUp): 201 POST http://10.0.0.112:9696/v2.0/security-groups 1.317s 2021-08-11 16:49:47.680 234999 DEBUG tempest.lib.common.rest_client [req-b6e5bb12-f490-489f-bc7a-daa45e624bbb ] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'} Body: {"security_group": {"name": "tempest-secgroup_access--609865299", "description": "tempest-secgroup_access--609865299 description", "tenant_id": "dd44686f5f974d5b81cd7c0a08728c68"}} Response - Headers: {'content-type': 'application/json', 'content-length': '1412', 'x-openstack-request-id': 'req-b6e5bb12-f490-489f-bc7a-daa45e624bbb', 'date': 'Wed, 11 Aug 2021 16:49:47 GMT', 'connection': 'close', 'status': '201', 'content-location': 'http://10.0.0.112:9696/v2.0/security-groups'} Body: b'{"security_group": {"id": "c55f7dfb-eb35-488c-918a-40f3d9c090a3", "name": "tempest-secgroup_access--609865299", "tenant_id": "dd44686f5f974d5b81cd7c0a08728c68", "description": "tempest-secgroup_access--609865299 description", "security_group_rules": [{"id": "2bd47bd7-e930-4200-bba7-eeb4be34670f", "tenant_id": "dd44686f5f974d5b81cd7c0a08728c68", "security_group_id": "c55f7dfb-eb35-488c-918a-40f3d9c090a3", "ethertype": "IPv4", "direction": "egress", "protocol": null, "port_range_min": null, "port_range_max": null, "remote_ip_prefix": null, "remote_group_id": null, "description": null, "tags": [], "created_at": "2021-08-11T16:49:47Z", "updated_at": "2021-08-11T16:49:47Z", "revision_number": 0, "project_id": "dd44686f5f974d5b81cd7c0a08728c68"}, {"id": "3ba58c3a-cab1-4020-bba9-d1c490250702", "tenant_id": "dd44686f5f974d5b81cd7c0a08728c68", "security_group_id": "c55f7dfb-eb35-488c-918a-40f3d9c090a3", "ethertype": "IPv6", "direction": "egress", "protocol": null, "port_range_min": null, "port_range_max": null, "remote_ip_prefix": null, "remote_group_id": null, "description": null, "tags": [], "created_at": "2021-08-11T16:49:47Z", "updated_at": "2021-08-11T16:49:47Z", "revision_number": 0, "project_id": "dd44686f5f974d5b81cd7c0a08728c68"}], "tags": [], "created_at": "2021-08-11T16:49:47Z", "updated_at": "2021-08-11T16:49:47Z", "revision_number": 1, "project_id": "dd44686f5f974d5b81cd7c0a08728c68"}}' Apparently, the SG POST request generated two entries in the databases [3] (I'm not including all the logs): 2021-08-11 16:49:46.375 27 DEBUG neutron.api.v2.base [req-b6e5bb12-f490-489f-bc7a-daa45e624bbb 884c6d80284b471093e4caa27050de1c dd44686f5f974d5b81cd7c0a08728c68 - default default] Request body: {'security_group': {'name': 'tempest-secgroup_access--609865299', 'description': 'tempest-secgroup_access--609865299 description', 'tenant_id': 'dd44686f5f974d5b81cd7c0a08728c68'}} prepare_request_body /usr/lib/python3.6/site-packages/neutron/api/v2/base.py:719 2021-08-11 16:49:46.823 27 DEBUG networking_ovn.db.revision [req-b6e5bb12-f490-489f-bc7a-daa45e624bbb 884c6d80284b471093e4caa27050de1c dd44686f5f974d5b81cd7c0a08728c68 - default default] create_initial_revision uuid=3fe383b6-d6c3-4a1b-82ff-9a299852e2e3, type=security_groups, rev=-1 create_initial_revision /usr/lib/python3.6/site-packages/networking_ovn/db/revision.py:59 2021-08-11 16:49:46.925 27 DEBUG neutron_lib.db.api [req-b6e5bb12-f490-489f-bc7a-daa45e624bbb 884c6d80284b471093e4caa27050de1c dd44686f5f974d5b81cd7c0a08728c68 - default default] Retry wrapper got retriable exception: (pymysql.err.InternalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 2021-08-11 16:49:46.927 27 DEBUG neutron.api.rpc.handlers.resources_rpc [req-b6e5bb12-f490-489f-bc7a-daa45e624bbb 884c6d80284b471093e4caa27050de1c dd44686f5f974d5b81cd7c0a08728c68 - - -] Pushing event updated for resources: {'SecurityGroup': ['ID=3fe383b6-d6c3-4a1b-82ff-9a299852e2e3,revision_number=1']} push /usr/lib/python3.6/site-packages/neutron/api/rpc/handlers/resources_rpc.py:243 2021-08-11 16:49:47.426 27 DEBUG neutron.api.v2.base [req-b6e5bb12-f490-489f-bc7a-daa45e624bbb 884c6d80284b471093e4caa27050de1c dd44686f5f974d5b81cd7c0a08728c68 - default default] Request body: {'security_group': {'name': 'tempest-secgroup_access--609865299', 'description': 'tempest-secgroup_access--609865299 description', 'tenant_id': 'dd44686f5f974d5b81cd7c0a08728c68'}} prepare_request_body /usr/lib/python3.6/site-packages/neutron/api/v2/base.py:719 2021-08-11 16:49:47.573 27 DEBUG networking_ovn.db.revision [req-b6e5bb12-f490-489f-bc7a-daa45e624bbb 884c6d80284b471093e4caa27050de1c dd44686f5f974d5b81cd7c0a08728c68 - default default] create_initial_revision uuid=c55f7dfb-eb35-488c-918a-40f3d9c090a3, type=security_groups, rev=-1 create_initial_revision /usr/lib/python3.6/site-packages/networking_ovn/db/revision.py:59 2021-08-11 16:49:47.680 27 DEBUG neutron.api.rpc.handlers.resources_rpc [req-b6e5bb12-f490-489f-bc7a-daa45e624bbb 884c6d80284b471093e4caa27050de1c dd44686f5f974d5b81cd7c0a08728c68 - - -] Pushing event updated for resources: {'SecurityGroup': ['ID=c55f7dfb-eb35-488c-918a-40f3d9c090a3,revision_number=1']} push /usr/lib/python3.6/site-packages/neutron/api/rpc/handlers/resources_rpc.py:243 So, the SG is created with IDs 3fe383b6-d6c3-4a1b-82ff-9a299852e2e3 and c55f7dfb-eb35-488c-918a-40f3d9c090a3. The response to the SG POST reguest only included the second one, but then the VM creation fails because there are two SG named tempest-secgroup_access--609865299 [1] https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/network/view/networking-ovn/job/DFG-network-networking-ovn-16.1_director-rhel-virthost-3cont_2comp-ipv4-geneve/561/testReport/tempest.scenario.test_security_groups_basic_ops/TestSecurityGroupsBasicOps/test_cross_tenant_traffic_compute_id_e79f879e_debb_440c_a7e4_efeda05b6848_network_/ [2] http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-networking-ovn-16.1_director-rhel-virthost-3cont_2comp-ipv4-geneve/561/undercloud-0/home/stack/tempest-dir/tempest.log.gz [3] http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-networking-ovn-16.1_director-rhel-virthost-3cont_2comp-ipv4-geneve/561/controller-2/var/log/containers/neutron/server.log.2.gz Version-Release number of selected component (if applicable): RHOS-16.1-RHEL-8-20210804.n.0 How reproducible: It was reproduced once with RHOS-16.1-RHEL-8-20210804.n.0 and once with RHOS-16.1-RHEL-8-20210727.n.1 only and on different tests Steps to Reproduce: 1. run the job (run the whole tempest neutron suite). I don't have a simpler way to reproduce, since it doesn't reproduce often and it can be reproduced by different tests 2. 3.