+++ This bug is a downstream clone. The original bug is: +++ +++ bug 1688052 +++ ====================================================================== Description of problem: In case gluster fencing policy check fails, _getVolumeStatus returns an empty dict[1]. Then, can_fence_host iterates through the returned empty dict with the method .get("bricks"), which is None. This throws an ugly exception[3] in the logs. If it happens frequently, it can spam significantly. In addition, in the related code there are two typos ("realted" instead of "related") here[4] and here[5]. [1] https://github.com/oVirt/vdsm/blob/master/lib/vdsm/gluster/fence.py#L112 [2] https://github.com/oVirt/vdsm/blob/master/lib/vdsm/gluster/fence.py#L39 [3] 2019-02-28 14:43:21,324+0000 WARN (jsonrpc/1) [Gluster] Failed to check gluster realted fencing policies: Command execution failed error: Error : Request timed out return code: 1 (fence:109) 2019-02-28 14:43:21,324+0000 ERROR (jsonrpc/1) [DynamicBridge] TypeError raised by dispatched function (Bridge:199) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 193, in _dynamicMethod result = fn(*methodArgs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1265, in fenceNode if action != 'status' and not should_fence(policy): File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1193, in should_fence result = check_gluster_fencing_policies(policy) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1250, in check_gluster_fencing_policies skipFencingIfGlusterQuorumNotMet) File "/usr/lib/python2.7/site-packages/vdsm/gluster/fence.py", line 37, in can_fence_host for brick in volStatus.get('bricks'): TypeError: 'NoneType' object is not iterable [4] https://github.com/oVirt/vdsm/blob/master/lib/vdsm/gluster/fence.py#L101 [5] https://github.com/oVirt/vdsm/blob/master/lib/vdsm/gluster/fence.py#L110 (Originally by Germano Veit Michel)
Any update? (Originally by Sahina Bose)
As discussed with Gobinda, the fix for this bug will be targeted for ovirt-4.3.6 (Originally by Satheesaran Sundaramoorthi)
sync2jira (Originally by Daniel Gur)
Tested with RHV 4.3.6 build with vdsm-4.30.29 1. HC env configured with HE VM running on gluster replica 3 volume 2. Fencing policies are enabled on the volume 3. Server quorum is disabled on the engine volume, to make sure stopping glusterd doesn't pause HE VM 4. Stop glusterd on all the nodes. 5. Move the host to maintenance. 6. No errors - 'non iterable' see with engine logs or vdsm logs
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:3009