Description of problem: OVN DB containers (ovn_cluster_south_db_server, ovn_cluster_north_db_server) are not restarted correctly because PID file remains in the /var/lib/openvswitch/ovn/ folder after containers are stopped using podman or systemctl commands. Version-Release number of selected component (if applicable): RHOS-17.0-RHEL-9-20220526.n.0 ovn-2021-21.12.0-59 How reproducible: Steps to Reproduce: systemctl restart tripleo_ovn_cluster_south_db_server or podman restart ovn_cluster_south_db_server Actual results: Expected results: Additional info:
Interestingly, it looks like on controller-0, restarting the services works as expected. On controller-1 and controller-2 it fails the pidfile_is_running() check. What looks like is happening is that on controller-0, which is set up to initially create the cluster, we're hitting ovn-ctl here: https://github.com/ovn-org/ovn/blob/915e6e0c2b4cb46d16db62fb6155eacdb3a0cb89/utilities/ovn-ctl#L311 which, even though the command we've built is run with 'exec', in this case it is run in the background with an '&'. This means ovn-ctl is running with pid=6, and ovsdb-server is running with pid=66. BUT, on the other two servers, we are hitting ovn-ctl here: https://github.com/ovn-org/ovn/blob/915e6e0c2b4cb46d16db62fb6155eacdb3a0cb89/utilities/ovn-ctl#L314 which ends up running the command in the foreground, so using exec means that ovsdb-server ends up running with pid=6...which is the pid that ovn-ctl will get when the container is restarted, so when pidfile_is_running() reads the .pid file and then checks /proc/6, it will find that it *does* exist, and so it thinks that ovsdb-server is running. One way to fix might be to pass in an optional binary to pidfile_is_running and then check if `readlink /proc/$pid/exe` == $binary.
Fix here: https://patchwork.ozlabs.org/project/ovn/list/?series=303093
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2022:6543