Bug 1555315

Summary: ansible role to deploy jenkins_builder sometime fail
Product: [Community] GlusterFS Reporter: Michael S. <misc>
Component: project-infrastructureAssignee: M. Scherer <mscherer>
Status: CLOSED UPSTREAM QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: mainlineCC: bugs, gluster-infra, mscherer
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-11 12:12:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Michael S. 2018-03-14 13:14:18 UTC
Description of problem:

From time to time, the job fail with:

remote: fatal: [builder109.cloud.gluster.org]: FAILED! => {"changed": false, "msg": "Unable to start service rpc-statd: Job for rpc-statd.service 
                 failed because the control process exited with error code. See \"systemctl status rpc-statd.service\" and \"journalctl -xe\" for details.\n"}

However, it fix itself after a while.

While invetigating, I did see this was caused by selinux:

type=AVC msg=audit(1521031389.894:2003194): avc:  denied  { write } for  pid=5686 comm="rpc.statd" name="rpc.statd.pid" dev="tmpfs" ino=107552625 
                 scontext=system_u:system_r:rpcd_t:s0 tcontext=unconfined_u:object_r:var_run_t:s0 tclass=file

but once the issue is fixed, I can't reproduce it anymore, even after removing the file.

So tracking progress on the issue until it happen again.

Comment 1 Niels de Vos 2018-03-14 14:17:07 UTC
There should be no need to start rpc.statd on builders. Gluster/NFS will try to start rpc.statd directly (without init/systemd). The cleanup of the tests kill rpc.statd in preparation for the next test.

Comment 2 Nigel Babu 2018-03-14 14:20:46 UTC
The rpc.statd start is a canary in the coal mine for IPV6 issues. It fails to start when IPV6 is disabled without doing a `dracut -v -f`. So without this step the entire process is un-predictable.

Comment 3 Michael S. 2018-03-14 16:45:46 UTC
So, if we do not need them, I can safely remove this then.

Comment 4 M. Scherer 2018-09-03 13:56:09 UTC
Related : https://bugzilla.redhat.com/show_bug.cgi?id=1623410

Comment 5 Worker Ant 2020-03-11 12:12:59 UTC
This bug is moved to https://api.github.com/repos/gluster/project-infrastructure/issues/1, and will be tracked there from now on. Visit GitHub issues URL for further details