Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1885592

Summary: [OVN] Some neutron requests answered with 504 after overcloud reboot
Product: Red Hat OpenStack Reporter: Eduardo Olivares <eolivare>
Component: python-networking-ovnAssignee: Jakub Libosvar <jlibosva>
Status: CLOSED NOTABUG QA Contact: Eran Kuris <ekuris>
Severity: high Docs Contact:
Priority: urgent    
Version: 16.1 (Train)CC: apevec, jlibosva, ksambor, lhh, majopela, scohen
Target Milestone: z3Keywords: AutomationBlocker, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1984305 (view as bug list) Environment:
Last Closed: 2021-07-27 06:25:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1984305    

Description Eduardo Olivares 2020-10-06 13:00:20 UTC 
Description of problem:
This issue is reproduced by the OVN 16.1 to 16.1 update jobs:
https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/network/view/networking-ovn/job/DFG-network-networking-ovn-update-16.1_director-rhel-virthost-3cont_2comp_2net-ipv4-geneve-composable/
or
https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/network/view/networking-ovn/job/DFG-network-networking-ovn-update-16.1_director-rhel-virthost-3cont_2comp-ipv4-vlan-sriov/

These jobs run tempest before update (1), after update (2) and after an overcloud reboot (3).
The issue occurs at (3). Most of the test fail because some requests are answered with 504 error.

After some debugging, we determined that the issue happens when:
- creating a network resource (it can be a network, a security group, etc)
- when the request is received by the neutron-api running on a certain container (in our case, we saw it only failed with controller-1 and ovn master dbs were running on controller-1 too)


This is what happens when controller-0's neutron-api receives a POST to create a security-group-rule (SUCCESS):
http://pastebin.test.redhat.com/908090


This is what happens when controller-1's neutron-api receives a POST to create a security-group-rule (FAIL):
http://pastebin.test.redhat.com/908097



LsAddCommand is not colled from controller-1.






Version-Release number of selected component (if applicable):
RHOS-16.1-RHEL-8-20201005.n.0
python3-networking-ovn-7.3.1-1.20200902233412.4e24f4c.1.el8ost.noarch
python3-neutron-15.2.1-1.20200905043434.40d217c.el8ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Run OVN 16.1 update job