Bug 1525171

Nir, it sounds a regression, maybe related to subprocess32 usage?  Can you verify that this is not behavior change? NFS storage access happens in external process afaik, once vdsmd dies, this child process is taken care of by systemd.
we used to use lock file before using systemd services - and after moving to systemd we never needed to take care for being zombies

(In reply to Yaniv Bronhaim from comment #1)
> Nir, it sounds a regression, maybe related to subprocess32 usage?  

No, we see it in 4.1, using cpopen. Please check this, explaning the issue:
https://bugzilla.redhat.com/show_bug.cgi?id=1518676#c25

I will file a systemd bug about restarting service after kill fail to terminate
it, but I think we should take care of this regardless of systemd.

> Can you
> verify that this is not behavior change? NFS storage access happens in
> external process afaik, 

We still have code accessing NFS directly, we are in the process of removing this
code.

> once vdsmd dies, this child process is taken care of
> by systemd.
> we used to use lock file before using systemd services - and after moving to
> systemd we never needed to take care for being zombies

This is not about zombies, but about process hang in D state. Killing it does
nothing since the process is uninterruptible.

ok, so avoiding accessing IO that can be unreachable in main process is the solution for such cases? we should use ioprocess for those accesses.. otherwise we can't really escape from the D state when setting reaching unreachable nfs address, no?

This bug is about adding flock mechanism to prevent running vdsmd instance again - https://gerrit.ovirt.org/85665

I think that systemd should take care of this. Do we have a matching bug there?

Bug #1530556 is about running new instance although systemd failed to kill original service pid. I didn't set it as Blocks\Depends on for this bug, as in this bugzilla the expected result is that vdsmd will fail immediately on startup with a clear error about another running instance.

Adding the bug dependency. the fix is more as workaround for vdsmd more than actually solving the bug in systemd

Using lock file doesn't help. Actually all fds of the origin process are removed once sigkill sent by systemd. This means that another instance can start normally and the original pid stacks until storage is reachable again.

Moving back to POST, patch is not included in ovirt-4.2 branch

Verification blocked due to Bug 1530556 which is in "NEW" state .

putting Bug 1530556 as "depends on" is wrong. It's additional bug that its fix can make the fix we do here redundant. It doesn't block the verification for running vdsmd process twice. start vdsmd service, change to vdsm user and run "/usr/share/vdsm/vdsmd" again. /var/lib/vdsm/vdsmd.lock file should lock the second process and fail the run

On host running vdsm
vdsm-4.20.20-1.el7ev.x86_64

2nd try to run vdsm - failed with nice error

# /usr/share/vdsm/vdsmd
Traceback (most recent call last):
File "/usr/share/vdsm/vdsmd", line 12, in <module>
  vdsmd.main()
File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 268, in main
  __assertSingleInstance()
File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 214, in __assertSingleInstance
    raise FatalError(str(e))
vdsm.vdsmd.FatalError: Instance already running

Lock file /run/vdsm/vdsmd.lock exists

This bugzilla is included in oVirt 4.2.2 release, published on March 28th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.2 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.