Description of problem: During testing of RHGS WA, I've found following traceback raised from /usr/sbin/gluster-eventsapi script: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ File "/usr/sbin/gluster-eventsapi", line 666, in <module> runcli() File "/usr/lib/python2.7/site-packages/gluster/cliutils/cliutils.py", line 224, in runcli cls.run(args) File "/usr/sbin/gluster-eventsapi", line 329, in run sync_to_peers(args) File "/usr/sbin/gluster-eventsapi", line 177, in sync_to_peers "{1}".format(e[0], e[2]), IndexError: tuple index out of range ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The prospective real issue is hidden beside this traceback. Version-Release number of selected component (if applicable): glusterfs-events-3.12.2-13.el7rhgs.x86_64 How reproducible: 100% if you will be able to cause the raise of GlusterCmdException Steps to Reproduce: I'm not sure, how to reproduce it from scratch, as my knowledge related to gluster-eventsapi is very limited, but the problem is quite well visible from the source code: Open /usr/sbin/gluster-eventsapi script and look for function sync_to_peers around line 171: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 171 def sync_to_peers(args): 172 if os.path.exists(WEBHOOKS_FILE): 173 try: 174 sync_file_to_peers(WEBHOOKS_FILE_TO_SYNC) 175 except GlusterCmdException as e: 176 handle_output_error("Failed to sync Webhooks file: [Error: {0}]" 177 "{1}".format(e[0], e[2]), 178 errcode=ERROR_WEBHOOK_SYNC_FAILED, 179 json_output=args.json) 180 181 if os.path.exists(CUSTOM_CONFIG_FILE): 182 try: 183 sync_file_to_peers(CUSTOM_CONFIG_FILE_TO_SYNC) 184 except GlusterCmdException as e: 185 handle_output_error("Failed to sync Config file: [Error: {0}]" 186 "{1}".format(e[0], e[2]), 187 errcode=ERROR_CONFIG_SYNC_FAILED, 188 json_output=args.json) 189 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Important lines are 177 and 186: "{1}".format(e[0], e[2]), The problem is, that the GlusterCmdException is raised this way[1]: raise GlusterCmdException((rc, out, err)) So all three parameters rc, out and err are supplied as one parameter (of type tuple). Actual results: Any problem leading to raise of GlusterCmdException is hidden beside above mentioned exception. Expected results: There shouldn't be any such traceback. Additional info: [1] file /usr/lib/python2.7/site-packages/gluster/cliutils/cliutils.py
Possible Reproduction scenario might be, to remove (rename) /var/lib/glusterd/events/ directory on one Gluster Storage Node and try to add webhook from another storage node: On Gluster node 5: # mv /var/lib/glusterd/events/ /var/lib/glusterd/events_BACKUP On Gluster node 1: # gluster-eventsapi webhook-add http://0.0.0.0:8697/test Traceback (most recent call last): File "/usr/sbin/gluster-eventsapi", line 666, in <module> runcli() File "/usr/lib/python2.7/site-packages/gluster/cliutils/cliutils.py", line 224, in runcli cls.run(args) File "/usr/sbin/gluster-eventsapi", line 329, in run sync_to_peers(args) File "/usr/sbin/gluster-eventsapi", line 177, in sync_to_peers "{1}".format(e[0], e[2]), IndexError: tuple index out of range
Patch sent to upstream to fix the issue https://review.gluster.org/#/c/glusterfs/+/22294/
Hi, Proceeding with the same reproducer the fix is not working as expected: 1. Remove /var/lib/glusterd/events folder from one of the nodes 2. Perform `gluster-eventsapi webhook-add http://0.0.0.0:8697/test` on any node expect above node Actual: # gluster-eventsapi webhook-add http://0.0.0.0:8697/test Traceback (most recent call last): File "/usr/sbin/gluster-eventsapi", line 174, in sync_to_peers sync_file_to_peers(WEBHOOKS_FILE_TO_SYNC) File "/usr/lib/python3.6/site-packages/gluster/cliutils/cliutils.py", line 172, in sync_file_to_peers raise GlusterCmdException((rc, out, err)) gluster.cliutils.cliutils.GlusterCmdException: (1, '', 'Commit failed on 10.70.46.154. Error: Unable to open /var/lib/glusterd//events/webhooks.json\n') During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/sbin/gluster-eventsapi", line 670, in <module> runcli() File "/usr/lib/python3.6/site-packages/gluster/cliutils/cliutils.py", line 225, in runcli cls.run(args) File "/usr/sbin/gluster-eventsapi", line 333, in run sync_to_peers(args) File "/usr/sbin/gluster-eventsapi", line 177, in sync_to_peers errmsg = e.message[2] if e.message[2] else e.message[1] AttributeError: 'GlusterCmdException' object has no attribute 'message' Expected: Correct error message Additional Info: # rpm -qa glusterfs-events glusterfs-events-6.0-51.el8rhgs.x86_64 # /usr/libexec/platform-python -V Python 3.6.8 From pdb, there's is no attr `message`: # gluster-eventsapi webhook-add http://0.0.0.0:8697/test3 > /usr/sbin/gluster-eventsapi(179)sync_to_peers() -> errmsg = e.message[2] if e.message[2] else e.message[1] (Pdb) e GlusterCmdException((1, '', 'Commit failed on 10.70.46.154. Error: Unable to open /var/lib/glusterd//events/webhooks.json\n'),) (Pdb) dir(e) ['__cause__', '__class__', '__context__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__suppress_context__', '__traceback__', '__weakref__', 'args', 'with_traceback'] (Pdb) c Traceback (most recent call last): ... [Above Traceback] AttributeError: 'GlusterCmdException' object has no attribute 'message' Possible fix: Replace `message` with `args[0]` in [0] and [1] [0] https://github.com/gluster/glusterfs/blob/devel/events/src/peer_eventsapi.py#L175-L181 [1] https://github.com/gluster/glusterfs/blob/devel/events/src/peer_eventsapi.py#L188-L192 With above fix: # gluster-eventsapi webhook-add http://0.0.0.0:8697/test6 Failed to sync Webhooks file: [Error: 1]Commit failed on 10.70.46.154. Error: Unable to open /var/lib/glusterd//events/webhooks.json Another obeservation: ------------------ Even the webhook-add operation failed, status states the creation of webhooks (below shouldn't have registered afaik) # gluster-eventsapi status Webhooks: http://0.0.0.0:8697/test http://0.0.0.0:8697/test2 http://0.0.0.0:8697/test3 http://0.0.0.0:8697/test4 http://0.0.0.0:8697/test5 http://0.0.0.0:8697/test6 Thanks, Leela.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (glusterfs bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:1462