Bug 1746718 - [downstream clone - 4.3.6] Typo and exception due to non-iterable object on gluster fencing testing
Summary: [downstream clone - 4.3.6] Typo and exception due to non-iterable object on g...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 4.3.1
Hardware: x86_64
OS: Linux
low
low
Target Milestone: ovirt-4.3.6
: 4.3.6
Assignee: Kaustav Majumder
QA Contact: SATHEESARAN
URL:
Whiteboard:
Depends On: 1688052
Blocks: 1723678
TreeView+ depends on / blocked
 
Reported: 2019-08-29 07:23 UTC by RHV bug bot
Modified: 2020-08-03 15:19 UTC (History)
9 users (show)

Fixed In Version: vdsm-4.30.29-1.el7ev
Doc Type: Bug Fix
Doc Text:
Previously, the gluster fencing policy check failed due to a non-iterable object and threw an exception. The code also contained a minor typo. The current release fixes these issues.
Clone Of: 1688052
Environment:
Last Closed: 2019-10-10 15:36:50 UTC
oVirt Team: Gluster
Target Upstream Version:
Embargoed:
lsvaty: testing_plan_complete-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:3009 0 None None None 2019-10-10 15:37:06 UTC
oVirt gerrit 98613 0 'None' MERGED gluster: Fix for typo and exception due to empty dict returned from _getVolumeStatus 2020-07-16 06:22:49 UTC
oVirt gerrit 101422 0 'None' MERGED gluster: Fix for typo and exception due to empty dict returned from _getVolumeStatus 2020-07-16 06:22:49 UTC

Description RHV bug bot 2019-08-29 07:23:44 UTC
+++ This bug is a downstream clone. The original bug is: +++
+++   bug 1688052 +++
======================================================================

Description of problem:

In case gluster fencing policy check fails, _getVolumeStatus returns an empty dict[1]. Then, can_fence_host iterates through the returned empty dict with the method .get("bricks"), which is None. This throws an ugly exception[3] in the logs. If it happens frequently, it can spam significantly.

In addition, in the related code there are two typos ("realted" instead of "related") here[4] and here[5].

[1] https://github.com/oVirt/vdsm/blob/master/lib/vdsm/gluster/fence.py#L112
[2] https://github.com/oVirt/vdsm/blob/master/lib/vdsm/gluster/fence.py#L39
[3]
2019-02-28 14:43:21,324+0000 WARN  (jsonrpc/1) [Gluster] Failed to check
gluster realted fencing policies: Command execution failed
error: Error : Request timed out
return code: 1 (fence:109)
2019-02-28 14:43:21,324+0000 ERROR (jsonrpc/1) [DynamicBridge] TypeError
raised by dispatched function (Bridge:199)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 193,
in _dynamicMethod
    result = fn(*methodArgs)
  File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1265, in
fenceNode
    if action != 'status' and not should_fence(policy):
  File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1193, in
should_fence
    result = check_gluster_fencing_policies(policy)
  File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1250, in
check_gluster_fencing_policies
    skipFencingIfGlusterQuorumNotMet)
  File "/usr/lib/python2.7/site-packages/vdsm/gluster/fence.py", line
37, in can_fence_host
    for brick in volStatus.get('bricks'):
TypeError: 'NoneType' object is not iterable
[4] https://github.com/oVirt/vdsm/blob/master/lib/vdsm/gluster/fence.py#L101
[5] https://github.com/oVirt/vdsm/blob/master/lib/vdsm/gluster/fence.py#L110

(Originally by Germano Veit Michel)

Comment 6 RHV bug bot 2019-08-29 07:23:56 UTC
Any update?

(Originally by Sahina Bose)

Comment 7 RHV bug bot 2019-08-29 07:23:57 UTC
As discussed with Gobinda, the fix for this bug will be targeted for ovirt-4.3.6

(Originally by Satheesaran Sundaramoorthi)

Comment 9 RHV bug bot 2019-08-29 07:24:02 UTC
sync2jira

(Originally by Daniel Gur)

Comment 10 RHV bug bot 2019-08-29 07:24:03 UTC
sync2jira

(Originally by Daniel Gur)

Comment 12 SATHEESARAN 2019-09-04 10:43:43 UTC
Tested with RHV 4.3.6 build with vdsm-4.30.29

1. HC env configured with HE VM running on gluster replica 3 volume
2. Fencing policies are enabled on the volume
3. Server quorum is disabled on the engine volume, to make sure stopping glusterd doesn't pause HE VM
4. Stop glusterd on all the nodes.
5. Move the host to maintenance.
6. No errors - 'non iterable' see with engine logs or vdsm logs

Comment 15 errata-xmlrpc 2019-10-10 15:36:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3009


Note You need to log in before you can comment on or make changes to this bug.