Bug 2203372

Summary: [OSP 16.2][neutron][ml2/ovs] - router gateway port stuck in down state
Product: Red Hat OpenStack Reporter: Flavio Piccioni <fpiccion>
Component: openstack-neutronAssignee: ldenny
Status: CLOSED INSUFFICIENT_DATA QA Contact: Eran Kuris <ekuris>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 16.2 (Train)CC: chrisbro, chrisw, ekuris, jmarti, ldavidde, ldenny, ltamagno, mlavalle, njohnston, scohen, skaplons, ykarel
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-06-26 14:00:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Flavio Piccioni 2023-05-12 09:35:25 UTC 
Description of problem:
in a RHOSP 16.2.4 - NonDVR l3-ha OVS environment floating ips stopped working (not reachable) suddenly.

Workaround was about deleting and re-creating "affected" routers.

Currently, number of no-working routers keeps increasing.


Version-Release number of selected component (if applicable):
RHOSP 16.2.4 - NonDVR l3-ha OVS environment

Additional info:
more logs and info will be attached as comments asap.
Comment 23 ldenny 2023-06-26 14:00:08 UTC 
Hi Team, 

As the customer has closed the case now and we are unable to reproduce, I will be closing this bugzilla now.

If you suspect you are hitting this same issue please check router status on all 3 controllers with something like [1]

If you see the route status for a given router is master on all 3 controllers, you can test recreating it and checking the status once more.

Some things that may be relevant: 
- neutron l3-agent-list-hosting-router <UUID> output were void for the failing routers
- Environment was recently updated from 13.0.16 to 16.2.4
- Issue didn't reoccur after routers were recreated.

If your issue matches some of the above, please reopen this bugzilla.

Thanks! 


[1] for i in $(ls *2023-06-06*/*/var/lib/neutron/ha_confs/ | grep -v '\.' |sort -u); do echo -n "$i "; rg master *2023-06-06*/*/var/lib/neutron/ha_confs/$i/state | wc -l | tr -d '\n' ;echo -n " "; rg backup *2023-06-06*/*/var/lib/neutron/ha_confs/$i/state | wc -l ;done