Previously, the gluster fencing policy check failed due to a non-iterable object and threw an exception. The code also contained a minor typo. The current release fixes these issues.
DescriptionGermano Veit Michel
2019-03-13 02:06:59 UTC
Description of problem:
In case gluster fencing policy check fails, _getVolumeStatus returns an empty dict[1]. Then, can_fence_host iterates through the returned empty dict with the method .get("bricks"), which is None. This throws an ugly exception[3] in the logs. If it happens frequently, it can spam significantly.
In addition, in the related code there are two typos ("realted" instead of "related") here[4] and here[5].
[1] https://github.com/oVirt/vdsm/blob/master/lib/vdsm/gluster/fence.py#L112
[2] https://github.com/oVirt/vdsm/blob/master/lib/vdsm/gluster/fence.py#L39
[3]
2019-02-28 14:43:21,324+0000 WARN (jsonrpc/1) [Gluster] Failed to check
gluster realted fencing policies: Command execution failed
error: Error : Request timed out
return code: 1 (fence:109)
2019-02-28 14:43:21,324+0000 ERROR (jsonrpc/1) [DynamicBridge] TypeError
raised by dispatched function (Bridge:199)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 193,
in _dynamicMethod
result = fn(*methodArgs)
File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1265, in
fenceNode
if action != 'status' and not should_fence(policy):
File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1193, in
should_fence
result = check_gluster_fencing_policies(policy)
File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1250, in
check_gluster_fencing_policies
skipFencingIfGlusterQuorumNotMet)
File "/usr/lib/python2.7/site-packages/vdsm/gluster/fence.py", line
37, in can_fence_host
for brick in volStatus.get('bricks'):
TypeError: 'NoneType' object is not iterable
[4] https://github.com/oVirt/vdsm/blob/master/lib/vdsm/gluster/fence.py#L101
[5] https://github.com/oVirt/vdsm/blob/master/lib/vdsm/gluster/fence.py#L110
This issue is already verified in 4.3.6, do we still need same for 4.4.0?
Comment 13Germano Veit Michel
2019-11-21 22:16:45 UTC
(In reply to Gobinda Das from comment #12)
> This issue is already verified in 4.3.6, do we still need same for 4.4.0?
Hi Gobinda,
Not sure if I understand your question correctly...
If you are asking if this BZ should be kept open to be in 4.4 errata, I think the answer is yes, that's the process. Or if you are asking if QE should also verify this BZ in 4.4, then this is more of a question to QE than me. It sounds like a simple fix and maybe it can be skipped, if the process allows?
Comment 15Kaustav Majumder
2020-01-28 09:13:41 UTC
Description of problem: In case gluster fencing policy check fails, _getVolumeStatus returns an empty dict[1]. Then, can_fence_host iterates through the returned empty dict with the method .get("bricks"), which is None. This throws an ugly exception[3] in the logs. If it happens frequently, it can spam significantly. In addition, in the related code there are two typos ("realted" instead of "related") here[4] and here[5]. [1] https://github.com/oVirt/vdsm/blob/master/lib/vdsm/gluster/fence.py#L112 [2] https://github.com/oVirt/vdsm/blob/master/lib/vdsm/gluster/fence.py#L39 [3] 2019-02-28 14:43:21,324+0000 WARN (jsonrpc/1) [Gluster] Failed to check gluster realted fencing policies: Command execution failed error: Error : Request timed out return code: 1 (fence:109) 2019-02-28 14:43:21,324+0000 ERROR (jsonrpc/1) [DynamicBridge] TypeError raised by dispatched function (Bridge:199) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 193, in _dynamicMethod result = fn(*methodArgs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1265, in fenceNode if action != 'status' and not should_fence(policy): File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1193, in should_fence result = check_gluster_fencing_policies(policy) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1250, in check_gluster_fencing_policies skipFencingIfGlusterQuorumNotMet) File "/usr/lib/python2.7/site-packages/vdsm/gluster/fence.py", line 37, in can_fence_host for brick in volStatus.get('bricks'): TypeError: 'NoneType' object is not iterable [4] https://github.com/oVirt/vdsm/blob/master/lib/vdsm/gluster/fence.py#L101 [5] https://github.com/oVirt/vdsm/blob/master/lib/vdsm/gluster/fence.py#L110