Bug 1933625 - Booting VM with a Floating IP and Pinging it fails with OVSDB Error: transaction failed messages
Summary: Booting VM with a Floating IP and Pinging it fails with OVSDB Error: transact...
Keywords:
Status: CLOSED DUPLICATE of bug 1503518
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-networking-ovn
Version: 16.1 (Train)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Jakub Libosvar
QA Contact: Eran Kuris
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-03-01 09:50 UTC by Asma Syed Hameed
Modified: 2021-12-22 20:00 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-12-22 20:00:05 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker OSP-499 0 None None None 2021-11-18 14:20:23 UTC

Description Asma Syed Hameed 2021-03-01 09:50:36 UTC
Description of problem:
When performing neutron scale testing on OSP16.1, we see OVSDB Error: transaction failed messages in neutron/server.log and even the ovn is dead.

https://gist.github.com/asyedham/ad441ce7ce7c46c2dd0e6d6aec1bd096

This is the rally plugin used(create network, subnet, boot server with fip and ping)for a total of 2000 times at a concurrency of 16:
https://github.com/openstack/browbeat/blob/master/rally/rally-plugins/netcreate-boot-ping/netcreate_nova-boot-fip-ping.py

Note: Neutron resources are still persistent where as nova is able to delete the resource after the workload is run.

Version-Release number of selected component (if applicable):
Red Hat OpenStack Platform release 16.1.4 GA (Train)

How reproducible:
100%

Steps to Reproduce:
1. Run the scenario netcreate-boot-ping
2. Observe the results and neutron logs


Rally test and errors log is at http://perf1.perf.lab.eng.bos.redhat.com/pub/asyedham/neutron-scale-testing/results/20210225-094725/rally/simple-plugins/netcreate-boot-ping/20210225-094725-browbeat-netcreate-boot-ping-16-1-iteration-0.log

PFA attached neutron, openvswitch and nova logs captured during rally run 
http://perf1.perf.lab.eng.bos.redhat.com/pub/asyedham/neutron-scale-testing/neutron-scale-test/

grafana snapshot: https://snapshot.raintank.io/dashboard/snapshot/VaU3D8L7o5HPnCYkim0iXrNblkRYJlZT

Additional info:
ovn2.13 scratch build based on ovn2.13-20.12.0-19 with logical datapath groups enabled by default:

http://brew-task-repos.usersys.redhat.com/repos/scratch/dceara/ovn2.13/20.12.0/20pvt_dp_groups.el8fdp/


Also, we have increased the timeouts from 300s to 3000s
nova_server_boot_timeout = 3000.0
vm_ping_timeout = 3000.0


Note You need to log in before you can comment on or make changes to this bug.