Bug 1687249

Summary: Error handling in /usr/sbin/gluster-eventsapi produces IndexError: tuple index out of range
Product: [Community] GlusterFS Reporter: Aravinda VK <avishwan>
Component: eventsapiAssignee: Aravinda VK <avishwan>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 5CC: dahorak, rhs-bugs, sanandpa, sankarshan, sheggodu
Target Milestone: ---Keywords: EasyFix, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-5.5 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1685027 Environment:
Last Closed: 2019-03-15 12:46:27 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1600459, 1685027    
Bug Blocks: 1687248    

Description Aravinda VK 2019-03-11 03:42:33 UTC
+++ This bug was initially created as a clone of Bug #1685027 +++

+++ This bug was initially created as a clone of Bug #1600459 +++

Description of problem:
  During testing of RHGS WA, I've found following traceback raised from 
  /usr/sbin/gluster-eventsapi script:

  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  File "/usr/sbin/gluster-eventsapi", line 666, in <module>
    runcli()
  File "/usr/lib/python2.7/site-packages/gluster/cliutils/cliutils.py", line 224, in runcli
    cls.run(args)
  File "/usr/sbin/gluster-eventsapi", line 329, in run
    sync_to_peers(args)
  File "/usr/sbin/gluster-eventsapi", line 177, in sync_to_peers
    "{1}".format(e[0], e[2]),
  IndexError: tuple index out of range
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  The prospective real issue is hidden beside this traceback.

Version-Release number of selected component (if applicable):
  glusterfs-events-3.12.2-13.el7rhgs.x86_64

How reproducible:
  100% if you will be able to cause the raise of GlusterCmdException

Steps to Reproduce:
I'm not sure, how to reproduce it from scratch, as my knowledge related
to gluster-eventsapi is very limited, but the problem is quite well
visible from the source code:

Open /usr/sbin/gluster-eventsapi script and look for function sync_to_peers
around line 171:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
171 def sync_to_peers(args):
172     if os.path.exists(WEBHOOKS_FILE):
173         try:
174             sync_file_to_peers(WEBHOOKS_FILE_TO_SYNC)
175         except GlusterCmdException as e:
176             handle_output_error("Failed to sync Webhooks file: [Error: {0}]"
177                                 "{1}".format(e[0], e[2]),
178                                 errcode=ERROR_WEBHOOK_SYNC_FAILED,
179                                 json_output=args.json)
180 
181     if os.path.exists(CUSTOM_CONFIG_FILE):
182         try:
183             sync_file_to_peers(CUSTOM_CONFIG_FILE_TO_SYNC)
184         except GlusterCmdException as e:
185             handle_output_error("Failed to sync Config file: [Error: {0}]"
186                                 "{1}".format(e[0], e[2]),
187                                 errcode=ERROR_CONFIG_SYNC_FAILED,
188                                 json_output=args.json)
189 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Important lines are 177 and 186:
  "{1}".format(e[0], e[2]),

The problem is, that the GlusterCmdException is raised this way[1]:
  raise GlusterCmdException((rc, out, err))
So all three parameters rc, out and err are supplied as one parameter (of type tuple).

Actual results:
  Any problem leading to raise of GlusterCmdException is hidden beside
  above mentioned exception.

Expected results:
  There shouldn't be any such traceback.

Additional info:
[1] file /usr/lib/python2.7/site-packages/gluster/cliutils/cliutils.py

--- Additional comment from Daniel HorĂ¡k on 2018-07-13 08:23:29 UTC ---

Possible Reproduction scenario might be, to remove (rename) /var/lib/glusterd/events/ directory on one Gluster Storage Node and try to add webhook from another storage node:

On Gluster node 5:
  # mv /var/lib/glusterd/events/ /var/lib/glusterd/events_BACKUP

On Gluster node 1:
  # gluster-eventsapi webhook-add http://0.0.0.0:8697/test
  Traceback (most recent call last):
    File "/usr/sbin/gluster-eventsapi", line 666, in <module>
      runcli()
    File "/usr/lib/python2.7/site-packages/gluster/cliutils/cliutils.py", line 224, in runcli
      cls.run(args)
    File "/usr/sbin/gluster-eventsapi", line 329, in run
      sync_to_peers(args)
    File "/usr/sbin/gluster-eventsapi", line 177, in sync_to_peers
      "{1}".format(e[0], e[2]),
  IndexError: tuple index out of range

--- Additional comment from Worker Ant on 2019-03-04 08:17:52 UTC ---

REVIEW: https://review.gluster.org/22294 (eventsapi: Fix error while handling GlusterCmdException) posted (#1) for review on master by Aravinda VK

--- Additional comment from Worker Ant on 2019-03-06 13:22:53 UTC ---

REVIEW: https://review.gluster.org/22294 (eventsapi: Fix error while handling GlusterCmdException) merged (#2) on master by Amar Tumballi

Comment 1 Worker Ant 2019-03-11 03:44:04 UTC
REVIEW: https://review.gluster.org/22332 (eventsapi: Fix error while handling GlusterCmdException) posted (#1) for review on release-5 by Aravinda VK

Comment 2 Worker Ant 2019-03-15 12:46:27 UTC
REVIEW: https://review.gluster.org/22332 (eventsapi: Fix error while handling GlusterCmdException) merged (#2) on release-5 by Shyamsundar Ranganathan

Comment 3 Shyamsundar 2019-03-27 13:46:38 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-5.5, please open a new bug report.

glusterfs-5.5 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://lists.gluster.org/pipermail/announce/2019-March/000119.html
[2] https://www.gluster.org/pipermail/gluster-users/