Description of problem:

I found out that when a peer probe is performed by the user, mgmt/glusterd write a file named after the hostname of the peer in question. On successful probes, this file is replaced with a file named after the UUID of the glusterd instance on the peer, while a failed probe causes the temp file to simply get deleted.

Here's an illustration:

root@someserver:/var/lib/glusterd/peers] gluster peer probe some_non_host &
[1] 25918
root@someserver:/var/lib/glusterd/peers] cat some_non_host
uuid=00000000-0000-0000-0000-000000000000
state=0
hostname1=ksome_non_host
root@someserver:/var/lib/glusterd/peers]
root@someserver:/var/lib/glusterd/peers] peer probe: failed: Probe returned with unknown errno 107

[1]+  Exit 1                  gluster peer probe some_non_host
root@someserver:/var/lib/glusterd/peers] ls
root@someserver:/var/lib/glusterd/peers] 
 
Here's the deal. When, for some reason, glulsterd is killed off before it get a chance to clean up on the temp file (say for a peer that really doesn't exist), and then, if you reboot your machine, the temporary file will really break mgmt/glusterd's recovery graph, and glusterd will be unable to initialize any of the existing volumes without having to delete the tmp file manually.

Version-Release number of selected component (if applicable):
Observed this on 2 releases: 3.4.0 and 3.4.2.

How reproducible:
100%

Steps to Reproduce:
1. Probe for a peer (preferably one that does not exist)
2. Parallelly, kill off glusterd as soon as temp file is created
3. Once glusterd is dead, reboot the machine.

Actual results:
When the machine comes back up, the brick processes for none of the glulster volumes created prior to reboot would be started automatically by glusterd.

Expected results:
mgmt/glusterd should have the intelligence to distinguish between a genuine peer and a temp file created during probe. The temp file should not affect the recovery graph after reboot. Something like a <peer-name>.tmp?

Additional info:
Preferably, also delete any temp file discovered during recovery at startup?