+++ This bug was initially created as a clone of Bug #1688052 +++ Description of problem: In case gluster fencing policy check fails, _getVolumeStatus returns an empty dict[1]. Then, can_fence_host iterates through the returned empty dict with the method .get("bricks"), which is None. This throws an ugly exception[3] in the logs. If it happens frequently, it can spam significantly. In addition, in the related code there are two typos ("realted" instead of "related") here[4] and here[5]. [1] https://github.com/oVirt/vdsm/blob/master/lib/vdsm/gluster/fence.py#L112 [2] https://github.com/oVirt/vdsm/blob/master/lib/vdsm/gluster/fence.py#L39 [3] 2019-02-28 14:43:21,324+0000 WARN (jsonrpc/1) [Gluster] Failed to check gluster realted fencing policies: Command execution failed error: Error : Request timed out return code: 1 (fence:109) 2019-02-28 14:43:21,324+0000 ERROR (jsonrpc/1) [DynamicBridge] TypeError raised by dispatched function (Bridge:199) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 193, in _dynamicMethod result = fn(*methodArgs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1265, in fenceNode if action != 'status' and not should_fence(policy): File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1193, in should_fence result = check_gluster_fencing_policies(policy) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1250, in check_gluster_fencing_policies skipFencingIfGlusterQuorumNotMet) File "/usr/lib/python2.7/site-packages/vdsm/gluster/fence.py", line 37, in can_fence_host for brick in volStatus.get('bricks'): TypeError: 'NoneType' object is not iterable [4] https://github.com/oVirt/vdsm/blob/master/lib/vdsm/gluster/fence.py#L101 [5] https://github.com/oVirt/vdsm/blob/master/lib/vdsm/gluster/fence.py#L110 --- Additional comment from RHEL Product and Program Management on 2019-03-18 07:09:16 UTC --- Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone. --- Additional comment from RHEL Product and Program Management on 2019-03-21 12:08:44 UTC --- Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone. --- Additional comment from RHEL Product and Program Management on 2019-03-21 12:23:36 UTC --- Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone. --- Additional comment from RHEL Product and Program Management on 2019-03-21 13:20:44 UTC --- Setting RHEV Target Release to 4.3.0 when oVirt Target Milestone is set to any of these: 4.3.1, 4.3.2, 4.3.3. --- Additional comment from Gobinda Das on 2019-03-25 07:12:40 UTC --- As this is not marked as blocker and next build of ovirt-4.3.3 accepts only blockers, so moving this to ovirt-4.3.4 --- Additional comment from Sahina Bose on 2019-04-23 07:04:53 UTC --- Any update?
Dependent ovirt bug is targeted for ovirt-4.3.6
https://bugzilla.redhat.com/show_bug.cgi?id=1746718 is the cloned bug which is ON_QA.
Tested with RHV 4.3.6 build with vdsm-4.30.29 1. HC env configured with HE VM running on gluster replica 3 volume 2. Fencing policies are enabled on the volume 3. Server quorum is disabled on the engine volume, to make sure stopping glusterd doesn't pause HE VM 4. Stop glusterd on all the nodes. 5. Move the host to maintenance. 6. No errors - 'non iterable' see with engine logs or vdsm logs
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2963