Description of problem:
During an rpm upgrade of glusterfs-server, preun will run "/sbin/service glusterfsd condrestart". This, of course, succeeds in killing all glusterfsd instances as status reports that glusterfsd is running so stop kills them all and start does nothing since this isn't a legacy configuration. Normally this is a desired effect as it allows the brick instances to load the new version when glusterd is next restarted.

If, however, we just upgraded one server in a replica set and the self-heal hasn't completed, upgrading the next server will cause a split-brain as the stale files on the first server will be the only files available to afr and will be updated making the files on the second server to also be considered stale.

How reproducible:
As long as there's disk activity on the same file during both brick restarts, and that file is large enough to not complete the self heal in time, always.

Expected results:
Perhaps a check should be done to ensure a clean self-heal state before doing the condrestart in preun or in the init/systemctl scripts.