This is almost certainly unrelated, but there are lots of instances of this:

    Feb 15 00:36:28 controller-2 pacemaker-controld[42561]: warning: Cannot execute '/usr/lib/ocf/resource.d/ovn/ovndb-servers': No such file or directory
    Feb 15 00:36:28 controller-2 pacemaker-controld[42561]: error: Failed to retrieve meta-data for ocf:ovn:ovndb-servers

Other resource actions succeed fine, so the file must actually be there. The controller (running as hacluster) runs meta-data actions, while the executor (running as root) runs other resource actions, so I'm guessing hacluster lacks access to one of the parent directories (via permissions or SELinux). If that's the case, whatever package that creates the directory should be updated.

The only impact would be that the cluster wouldn't know which parameters, if any, are reloadable or sensitive, which shouldn't have any effect on the problem here, but it's worth fixing. There's a long-term Pacemaker to-do to make meta-data actions go through the executor, which would avoid this issue, but I don't know when that will happen.

(In reply to Ken Gaillot from comment #2)
> This is almost certainly unrelated, but there are lots of instances of this:
> 
>     Feb 15 00:36:28 controller-2 pacemaker-controld[42561]: warning: Cannot
> execute '/usr/lib/ocf/resource.d/ovn/ovndb-servers': No such file or
> directory
>     Feb 15 00:36:28 controller-2 pacemaker-controld[42561]: error: Failed to
> retrieve meta-data for ocf:ovn:ovndb-servers
> 
> Other resource actions succeed fine, so the file must actually be there. 

Right the file is there inside the ovn-dbs container but not on the host.
That is actually why at the time I had filed https://bugzilla.redhat.com/show_bug.cgi?id=1850506 (but as you mention
it is not a trivial amount of work).
They are just a bit annoying or scary if you do not know they are benign. Reason for that is that this Ovn OCF RA
is maintained by other folks and ends up in the ovn package and not in resource-agents rpm.

(In reply to Michele Baldessari from comment #3)
> (In reply to Ken Gaillot from comment #2)
> > This is almost certainly unrelated, but there are lots of instances of this:
> > 
> >     Feb 15 00:36:28 controller-2 pacemaker-controld[42561]: warning: Cannot
> > execute '/usr/lib/ocf/resource.d/ovn/ovndb-servers': No such file or
> > directory
> >     Feb 15 00:36:28 controller-2 pacemaker-controld[42561]: error: Failed to
> > retrieve meta-data for ocf:ovn:ovndb-servers
> > 
> > Other resource actions succeed fine, so the file must actually be there. 
> 
> Right the file is there inside the ovn-dbs container but not on the host.
> That is actually why at the time I had filed
> https://bugzilla.redhat.com/show_bug.cgi?id=1850506 (but as you mention
> it is not a trivial amount of work).
> They are just a bit annoying or scary if you do not know they are benign.
> Reason for that is that this Ovn OCF RA
> is maintained by other folks and ends up in the ovn package and not in
> resource-agents rpm.

Ah right, I forgot about that. Never mind then. One of these days I'll get to that ...

This is unrelated to the crm_resource regression in -6, don't worry about the agents.

The cancel action failure is a scheduling bug and the apparent cause of the shutdown hang as well, since the scheduler just keeps incorrectly rescheduling the cancel action each time it fails.

The problem starts at 01:40:51 when an instance of ovn-dbs-bundle needs to be moved from controller-2 to controller-0. The first thing that needs to happen is that the recurring monitor for the ovndb_servers instance inside it needs to be cancelled, so ovndb_servers can be stopped, and so on. The problem is the scheduler schedules the cancel action on controller-0 instead of controller-2. Since controller-0 doesn't yet have the connection to the bundle, it immediately returns an error.

At this point I haven't determined whether it's a regression. Has this same procedure worked on earlier builds?

I'll try to get a fix asap -- if we go beyond the end of this week, we'll need to get exception. Can OpenStack QA ack this?

(In reply to Ken Gaillot from comment #5)
> This is unrelated to the crm_resource regression in -6, don't worry about
> the agents.
> 
> The cancel action failure is a scheduling bug and the apparent cause of the
> shutdown hang as well, since the scheduler just keeps incorrectly
> rescheduling the cancel action each time it fails.
> 
> The problem starts at 01:40:51 when an instance of ovn-dbs-bundle needs to
> be moved from controller-2 to controller-0. The first thing that needs to
> happen is that the recurring monitor for the ovndb_servers instance inside
> it needs to be cancelled, so ovndb_servers can be stopped, and so on. The
> problem is the scheduler schedules the cancel action on controller-0 instead
> of controller-2. Since controller-0 doesn't yet have the connection to the
> bundle, it immediately returns an error.
> 
> At this point I haven't determined whether it's a regression. Has this same
> procedure worked on earlier builds?

Thanks Ken. It's hard to say tbh, I need to discuss this with Damien tomorrow.
Previously we were affected by the systemd stop pacemaker_remote quite often, so it
is a bit difficult to get hard data.

> I'll try to get a fix asap -- if we go beyond the end of this week, we'll
> need to get exception. Can OpenStack QA ack this?

Let me ask our QE for this in the meantime.

Thanks!

Fix merged upstream as of commit 8f5b73c0

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (pacemaker bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:1782

*** Bug 1969419 has been marked as a duplicate of this bug. ***

*** Bug 1969419 has been marked as a duplicate of this bug. ***

*** Bug 1971650 has been marked as a duplicate of this bug. ***

*** Bug 1973660 has been marked as a duplicate of this bug. ***