Bug 2091686

Summary:	[RAFT] OVN DB container unable to restart
Product:	Red Hat OpenStack	Reporter:	Alex Katz <akatz>
Component:	openstack-neutron	Assignee:	Terry Wilson <twilson>
Status:	CLOSED ERRATA	QA Contact:	Alex Katz <akatz>
Severity:	high	Docs Contact:
Priority:	high
Version:	17.0 (Wallaby)	CC:	apevec, bcafarel, chrisw, jlibosva, jschluet, lhh, majopela, mtomaska, scohen, skaplons, spower, twilson
Target Milestone:	ga	Keywords:	TestOnly, Triaged
Target Release:	17.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2092976 (view as bug list)		Environment:
Last Closed:	2022-09-21 12:21:38 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2092976
Bug Blocks:	1503518

Description Alex Katz 2022-05-30 17:39:13 UTC

Description of problem:
OVN DB containers (ovn_cluster_south_db_server, ovn_cluster_north_db_server) are not restarted correctly because PID file remains in the /var/lib/openvswitch/ovn/ folder after containers are stopped using podman or systemctl commands.


Version-Release number of selected component (if applicable):
RHOS-17.0-RHEL-9-20220526.n.0
ovn-2021-21.12.0-59


How reproducible:


Steps to Reproduce:
systemctl restart tripleo_ovn_cluster_south_db_server
or
podman restart ovn_cluster_south_db_server


Actual results:


Expected results:


Additional info:

Comment 1 Terry Wilson 2022-05-31 21:32:08 UTC

Interestingly, it looks like on controller-0, restarting the services works as expected. On controller-1 and controller-2 it fails the pidfile_is_running() check.

What looks like is happening is that on controller-0, which is set up to initially create the cluster, we're hitting ovn-ctl here:

  https://github.com/ovn-org/ovn/blob/915e6e0c2b4cb46d16db62fb6155eacdb3a0cb89/utilities/ovn-ctl#L311

which, even though the command we've built is run with 'exec', in this case it is run in the background with an '&'. This means ovn-ctl is running with pid=6, and ovsdb-server is running with pid=66.

BUT, on the other two servers, we are hitting ovn-ctl here:

  https://github.com/ovn-org/ovn/blob/915e6e0c2b4cb46d16db62fb6155eacdb3a0cb89/utilities/ovn-ctl#L314

which ends up running the command in the foreground, so using exec means that ovsdb-server ends up running with pid=6...which is the pid that ovn-ctl will get when the container is restarted, so when pidfile_is_running() reads the .pid file and then checks /proc/6, it will find that it *does* exist, and so it thinks that ovsdb-server is running.

One way to fix might be to pass in an optional binary to pidfile_is_running and then check if `readlink /proc/$pid/exe` == $binary.

Comment 2 Terry Wilson 2022-06-02 16:55:06 UTC

Fix here: https://patchwork.ozlabs.org/project/ovn/list/?series=303093

Comment 16 errata-xmlrpc 2022-09-21 12:21:38 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:6543