Description of problem:
We seem to be hitting occurrences of:

# dnf install --allowerasing -y <package list>

Malformed lock file found: /var/cache/dnf/metadata_lock.pid.
Ensure no other dnf process is running and remove the lock file manually or run systemd-tmpfiles --remove dnf.conf.

This seems to happen most often when a machine loses power unexpectedly.  This might seem like it should be a rare event but it happens more frequently in HA configurations where nodes need to be STONITHed for the integrity of the cluster.

I suspect that what happens is that the lock is taken by a running dnf process but before the data (i.e. the pid) can be flushed to the file, the machine is powered off and on again.  Once booted up, the filesystem journal is replayed creating the file but without it's contents.

What results is an empty lock file which DNF simply refuses to handle (https://github.com/rpm-software-management/dnf/blob/e4f3e91480647146b3bce542cccd2f675a5bc8e2/dnf/lock.py#L86-L104):

    def _try_read_lock(self):
        try:
            with open(self.target, 'r') as f:
                return int(f.readline())
        except IOError:
            return -1
        except ValueError:
            time.sleep(2)
            try:
                with open(self.target, 'r') as f:
                    return int(f.readline())
            except IOError:
                return -1
            except ValueError:
                msg = _('Malformed lock file found: %s.\n'
                        'Ensure no other dnf process is running and '
                        'remove the lock file manually or run '
                        'systemd-tmpfiles --remove dnf.conf.' % (self.target))
                raise LockError(msg)

I wonder if this is the best we can do.  Perhaps a system boot job to remove any stale locks is needed?

Any other ideas?