Bug 1082129

Description of problem:

The gridengine sge_qmaster does not create a PID file on startup (e.g. /run/sgemaster.pid).  The current systemd configuration runs it as a SysV-style forking daemon (Type=forking in sgemaster.service).  Without a PID file, systemd tries to guess the main PID of a forking service, but this process is not reliable (search GuessMainPID in http://www.freedesktop.org/software/systemd/man/systemd.service.html).

On one of our servers, this reliably segfaulted (others work ok).  After significant effort tracing the error, I found it to be that the incorrect main PID was identified, systemd then thought the daemon was finished and sent a SIGTERM to the other processes to tidy up.  Unfortunately this signal arrived during initialisation and the daemon blew up with a seg fault due to insufficiently defensive coding.

This is a combination of a misidentification of the main PID and a race condition on receiving a SIGTERM during initialisation.  I'm unsure of the systemd algorithm for guessing the main PID, but it probably just waits for a couple of forks or a short time period, then picks the current top process, so this may be racey too.  The guessing process is explicitly described as unreliable in the systemd documentation.


Version-Release number of selected component (if applicable):
 gridengine-qmaster-2011.11p1-15.fc19.x86_64

How reproducible:
 100% on one server, 0% on another!

Steps to Reproduce:
1. create a default Grid Engine master:
  dnf install gridengine-qmaster
  cd /usr/share/gridengine/
  ./install_qmaster  # pretty much just take defaults
2. systemctl start sgemaster.service

Actual results:
 Depends on luck - can hit a race condition (=segfault) or a controlled shutdown due to misidentification of the main process (=master is stopped cleanly), or be lucky and have it start up.

Expected results:
 Daemon starts up.

Additional info:
 Aside from being luckier, I found two fixes to the problem.

One can patch the daemons to write out a PID file and add a PIDFile option to the systemd service unit file.  I can provide an example patch for sge_qmaster if that's helpful.

Less invasively, one can add the following lines to /etc/sysconfig/gridengine, which prevent the daemons from forking:
------
# prevent SGE from daemonising qmaster, shadowd, execd
# required for systemd to control this as a "Type=simple" service
# see bugzilla #????
SGE_ND=true
-----
and change the unit file type to "Type=simple", where system expects the process to continue in the foreground rather than daemonising.  This does result in minor spam to /var/log/messages (about 20 lines every 3 mins on my system).  One has to change the unit files for sgemaster.service, sge_shadowd.service and sge_execd.service.

Lennart Poettering recommends the foreground approach in another discussion here - http://lists.freedesktop.org/archives/systemd-devel/2011-June/002677.html