Bug 1357754 - Delayed Events if any one Webhook is slow
Summary: Delayed Events if any one Webhook is slow
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: eventsapi
Version: mainline
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Aravinda VK
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks: 1395613 1401261
TreeView+ depends on / blocked
 
Reported: 2016-07-19 05:46 UTC by Aravinda VK
Modified: 2017-03-06 17:21 UTC (History)
0 users

Fixed In Version: glusterfs-3.10.0
Clone Of:
: 1395613 1401261 (view as bug list)
Environment:
Last Closed: 2017-03-06 17:21:32 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Aravinda VK 2016-07-19 05:46:50 UTC
Description of problem:
If Webhook takes more time to consume the events, it may affect the further pushing of Events. Add configurable Timeout option while calling Webhook.

# Example, Set Timeout in seconds
gluster-eventsapi config-set webhook-timeout 30

Comment 1 Aravinda VK 2016-11-16 09:50:13 UTC
Also observed that, if one webhook is slow then future events gets delayed. Webhooks should be called asynchronously to avoid delays.

Comment 2 Worker Ant 2016-11-29 11:13:16 UTC
REVIEW: http://review.gluster.org/15966 (eventsapi: Push Messages to Webhooks in parallel) posted (#1) for review on master by Aravinda VK (avishwan)

Comment 3 Aravinda VK 2016-11-29 11:16:38 UTC
Separate threads maintained for each Webhook so that slow Webhook will not affect the other webhooks. With this change we do not need configurable option for Timeout.

Comment 4 Worker Ant 2016-12-02 06:25:41 UTC
REVIEW: http://review.gluster.org/15966 (eventsapi: Push Messages to Webhooks in parallel) posted (#2) for review on master by Aravinda VK (avishwan)

Comment 5 Worker Ant 2016-12-02 14:52:59 UTC
COMMIT: http://review.gluster.org/15966 committed in master by Aravinda VK (avishwan) 
------
commit 99458974b7d12bd12d78d4b9a19adfb62d771b5c
Author: Aravinda VK <avishwan>
Date:   Tue Nov 29 16:32:54 2016 +0530

    eventsapi: Push Messages to Webhooks in parallel
    
    With this patch, glustereventsd will maintain one thread per
    webhook. If a webhook is slow, then all the events to that worker
    will be delayed but it will not affect the other webhooks.
    
    Note: Webhook in transit may get missed if glustereventsd reloads due to
    new Webhook addition or due configuration changes.
    
    BUG: 1357754
    Change-Id: I2d11e01c7ac434355bc356ff75396252f51b339b
    Signed-off-by: Aravinda VK <avishwan>
    Reviewed-on: http://review.gluster.org/15966
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: Prashanth Pai <ppai>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>

Comment 6 Shyamsundar 2017-03-06 17:21:32 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.0, please open a new bug report.

glusterfs-3.10.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2017-February/030119.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.