Description of problem: From time to time, the job fail with: remote: fatal: [builder109.cloud.gluster.org]: FAILED! => {"changed": false, "msg": "Unable to start service rpc-statd: Job for rpc-statd.service failed because the control process exited with error code. See \"systemctl status rpc-statd.service\" and \"journalctl -xe\" for details.\n"} However, it fix itself after a while. While invetigating, I did see this was caused by selinux: type=AVC msg=audit(1521031389.894:2003194): avc: denied { write } for pid=5686 comm="rpc.statd" name="rpc.statd.pid" dev="tmpfs" ino=107552625 scontext=system_u:system_r:rpcd_t:s0 tcontext=unconfined_u:object_r:var_run_t:s0 tclass=file but once the issue is fixed, I can't reproduce it anymore, even after removing the file. So tracking progress on the issue until it happen again.
There should be no need to start rpc.statd on builders. Gluster/NFS will try to start rpc.statd directly (without init/systemd). The cleanup of the tests kill rpc.statd in preparation for the next test.
The rpc.statd start is a canary in the coal mine for IPV6 issues. It fails to start when IPV6 is disabled without doing a `dracut -v -f`. So without this step the entire process is un-predictable.
So, if we do not need them, I can safely remove this then.
Related : https://bugzilla.redhat.com/show_bug.cgi?id=1623410
This bug is moved to https://api.github.com/repos/gluster/project-infrastructure/issues/1, and will be tracked there from now on. Visit GitHub issues URL for further details