Description of problem:

Several times in oslab we have ended up with multiple neutron daemons running at the same time.

I'm fairly sure the sequence of events starts with /var filling up, causing various odd high-level problems such as scheduling errors or networking failures; but nothing blatantly obvious as an out-of-disk error unless you're digging in the right logs on the right host at the right time.

So without noticing this, as part of trouble-shooting the admin tries restarting various daemons.  

At that point you probably see something like

---
[root@rhel init.d]# service neutron-dhcp-agent start
Starting neutron-dhcp-agent: bash: line 0: echo: write error: No space left on device
                                                           [FAILED]
---

which will probably lead you to realise the full disk situation, even if it could be much more helpful in the error output.

However, the neutron-dhcp-agent daemon was actually started.  The problem is that the daemon gets started under a 'daemon' function that returns the pid, e.g.

---
    echo -n $"Starting $prog: "
    daemon --user neutron --pidfile $pidfile "$exec --log-file /var/log/$proj/$plugin.log ${configs[@]/#/--config-file } &>/dev/null & echo \$! > $pidfile"
    retval=$?
---

Even if $pidfile can't be written, the daemon has started anyway.

So likely you restart the daemon again, which leads to replacing one problem with another, much harder to debug, one.  The two (or more) agents start racing with each other to consume RPC calls, etc.