Bug 2295757

Summary:	[FFU] After OVN controllers are upgraded we need to confirm that they are running properly before upgrading OVN control plane
Product:	Red Hat OpenStack	Reporter:	Alex Stupnikov <astupnik>
Component:	openstack-tripleo-heat-templates	Assignee:	Lukas Bezdicka <lbezdick>
Status:	CLOSED ERRATA	QA Contact:	Archana Singh <arcsingh>
Severity:	high	Docs Contact:
Priority:	high
Version:	17.1 (Wallaby)	CC:	averdagu, froyo, jbadiapa, jelynch, jlibosva, jpretori, kgilliga, lbezdick, mariel, mburns, mtomaska, parthee, ramishra, schhabdi
Target Milestone:	z4	Keywords:	Triaged
Target Release:	17.1
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-14.3.1-17.1.20240919130751.e7c7ce3.el9ost	Doc Type:	Bug Fix
Doc Text:	Before this update, an issue with the `systemd` file caused a restart loop of ovn-controllers during an upgrade from RHOSP 16.2 to 17.1. This issue caused an outage for DHCP and DNS services on the workloads. With this update, the issue has been fixed.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2024-11-21 09:30:38 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Alex Stupnikov 2024-07-04 09:33:33 UTC

Description of problem:
One of our customers faced network outage during FFU steps after upgrading and restarting OVN controllers: controllers were unable to process existing OVN DB contents and connectivity was lost. Lukas Bezdicka helped to address the problem from DF perspective.

OVN issue was reported to find a root cause of this problem: https://issues.redhat.com/browse/FDP-692 . This bug is reported to react when OVN team provides a feedback. 

Version-Release number of selected component (if applicable):
RHOSP 16.2/17.1

How reproducible: no clear reproducer, problem was faced during OVN upgrade after OVN controllers were updated and restarted, but old RHOSP 16.2 OVN controlplane was still running.

Comment 2 Jakub Libosvar 2024-07-19 14:21:39 UTC

The issue is in the way TripleO generates pid file paths for the systemd service file and conmon - they are not the same which leads to situation conmon generates a pid file that is not seen by the systemd. Then the systemd keeps restaring ovn-controller. Since the restarts are periodic at some point ovn-controller is restarted while OVN databases are being upgraded and hence down. ovn-controller loses the data about services such as DNS and DHCP. Then workloads using it can't resolve hostnames or if the DHCP lease expires they lose connectivity completely.

Comment 3 Jakub Libosvar 2024-07-19 14:22:54 UTC

*** Bug 2298388 has been marked as a duplicate of this bug. ***

Comment 22 errata-xmlrpc 2024-11-21 09:30:38 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: RHOSP 17.1.4 (openstack-tripleo-heat-templates) security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:9978

Comment 23 Lukas Bezdicka 2024-11-27 11:38:11 UTC

*** Bug 2321885 has been marked as a duplicate of this bug. ***

Comment 24 Red Hat Bugzilla 2025-03-28 04:25:44 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days