systemd daemons usually assumes that systemd is ensuring that only one instance of the process is running. However if the process is in D state when restarted it, systemd seems to ignore the kill failure, and try to start the service twice (actually infinite times).

Restart vdsmd service while the vdsmd process is in D state, accessing non-responsive NFS server- Systemd fail to kill vdsmd, and start it again.

You can find example flow here:
https://bugzilla.redhat.com/show_bug.cgi?id=1518676#c25

Also attach the log from:
https://bugzilla.redhat.com/attachment.cgi?id=1361489

Version: systemd 231


Steps to Reproduce:
1. Create a service  with the attach script (single)

#!/usr/bin/python

import io
import mmap
import os
import time

fd = os.open("/tmp/mnt/test", os.O_RDWR | os.O_CREAT | os.O_DIRECT)
buf = mmap.mmap(-1, 512)

check = 0

while True:
    buf[:] = b"\0" * len(buf)
    buf.seek(0)
    buf.write("pid: %d\ncheck: %d\n" % (os.getpid(), check))

    with io.FileIO(fd, "w", closefd=False) as f:
        f.seek(0)
        f.write(buf)
        os.fsync(f)

    time.sleep(10)
    check += 1

2. Install the service with the attached unit file (single.service)

[Unit]
Description=single

[Service]
Type=simple
ExecStart=/root/single
Restart=always


3. Mount nfs on /tmp/mnt

# mkdir /tmp/mnt

# mount myserver:/path /tmp/mnt


4. Start the service

# systemctl start single

# systemctl status single
● single.service - single
   Loaded: loaded (/root/single.service; linked; vendor preset: disabled)
   Active: active (running) since Sun 2017-12-31 20:15:52 IST; 2s ago
 Main PID: 18110 (single)
   CGroup: /system.slice/single.service
           └─18110 /usr/bin/python /root/single

5. Block access to nfs server:

# iptables -A OUTPUT -p tcp -d myserver --dport 2049 -j DROP


6. Wait 10 seconds until the single service move to D state:

# ps auxf
...
root     18110  0.0  0.1 131808  4816 ?        Ds   20:15   0:00 /usr/bin/python /root/single


7. Restart the service

# systemctl restart single


8. In another shell, run journalctl -f

Dec 31 20:17:04 voodoo6.tlv.redhat.com systemd[1]: Stopping single...
Dec 31 20:18:34 voodoo6.tlv.redhat.com systemd[1]: single.service stop-sigterm timed out. Killing.
Dec 31 20:20:01 voodoo6.tlv.redhat.com systemd[1]: Started Session 15 of user root.
Dec 31 20:20:01 voodoo6.tlv.redhat.com systemd[1]: Starting Session 15 of user root.
Dec 31 20:20:01 voodoo6.tlv.redhat.com CROND[18165]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Dec 31 20:20:02 voodoo6.tlv.redhat.com kernel: nfs: server dumbo.tlv.redhat.com not responding, timed out
Dec 31 20:20:04 voodoo6.tlv.redhat.com kernel: nfs: server dumbo.tlv.redhat.com not responding, still trying
Dec 31 20:20:04 voodoo6.tlv.redhat.com systemd[1]: single.service still around after SIGKILL. Ignoring.
Dec 31 20:21:34 voodoo6.tlv.redhat.com systemd[1]: single.service stop-final-sigterm timed out. Killing.
Dec 31 20:23:04 voodoo6.tlv.redhat.com systemd[1]: single.service still around after final SIGKILL. Entering failed mode.
Dec 31 20:23:04 voodoo6.tlv.redhat.com systemd[1]: Unit single.service entered failed state.
Dec 31 20:23:04 voodoo6.tlv.redhat.com systemd[1]: single.service failed.
Dec 31 20:23:04 voodoo6.tlv.redhat.com systemd[1]: Started single.
Dec 31 20:23:04 voodoo6.tlv.redhat.com systemd[1]: Starting single...


9. Check ps output again

# ps auxf
...
root     18110  0.0  0.0      0     0 ?        Ds   20:15   0:00 [single]
root     18183  0.0  0.1 131804  4820 ?        Ds   20:23   0:00 /usr/bin/python /root/single

# systemctl status single
● single.service - single
   Loaded: loaded (/root/single.service; linked; vendor preset: disabled)
   Active: active (running) since Sun 2017-12-31 20:23:04 IST; 9min ago
 Main PID: 18183 (single)
   CGroup: /system.slice/single.service
           └─18183 /usr/bin/python /root/single


Actual result:
two instances running in the same time.

Expected result:
Only single instance is started.