Bug 1401261 - Delayed Events if any one Webhook is slow
Summary: Delayed Events if any one Webhook is slow
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: eventsapi
Version: 3.9
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Aravinda VK
QA Contact:
URL:
Whiteboard:
Depends On: 1357754
Blocks: 1395613
TreeView+ depends on / blocked
 
Reported: 2016-12-04 04:38 UTC by Atin Mukherjee
Modified: 2017-03-08 10:20 UTC (History)
1 user (show)

Fixed In Version: glusterfs-3.9.1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1357754
Environment:
Last Closed: 2017-03-08 10:20:58 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Atin Mukherjee 2016-12-04 04:38:16 UTC
+++ This bug was initially created as a clone of Bug #1357754 +++

Description of problem:
If Webhook takes more time to consume the events, it may affect the further pushing of Events. Add configurable Timeout option while calling Webhook.

# Example, Set Timeout in seconds
gluster-eventsapi config-set webhook-timeout 30

--- Additional comment from Aravinda VK on 2016-11-16 04:50:13 EST ---

Also observed that, if one webhook is slow then future events gets delayed. Webhooks should be called asynchronously to avoid delays.

--- Additional comment from Worker Ant on 2016-11-29 06:13:16 EST ---

REVIEW: http://review.gluster.org/15966 (eventsapi: Push Messages to Webhooks in parallel) posted (#1) for review on master by Aravinda VK (avishwan)

--- Additional comment from Aravinda VK on 2016-11-29 06:16:38 EST ---

Separate threads maintained for each Webhook so that slow Webhook will not affect the other webhooks. With this change we do not need configurable option for Timeout.

--- Additional comment from Worker Ant on 2016-12-02 01:25:41 EST ---

REVIEW: http://review.gluster.org/15966 (eventsapi: Push Messages to Webhooks in parallel) posted (#2) for review on master by Aravinda VK (avishwan)

--- Additional comment from Worker Ant on 2016-12-02 09:52:59 EST ---

COMMIT: http://review.gluster.org/15966 committed in master by Aravinda VK (avishwan) 
------
commit 99458974b7d12bd12d78d4b9a19adfb62d771b5c
Author: Aravinda VK <avishwan>
Date:   Tue Nov 29 16:32:54 2016 +0530

    eventsapi: Push Messages to Webhooks in parallel
    
    With this patch, glustereventsd will maintain one thread per
    webhook. If a webhook is slow, then all the events to that worker
    will be delayed but it will not affect the other webhooks.
    
    Note: Webhook in transit may get missed if glustereventsd reloads due to
    new Webhook addition or due configuration changes.
    
    BUG: 1357754
    Change-Id: I2d11e01c7ac434355bc356ff75396252f51b339b
    Signed-off-by: Aravinda VK <avishwan>
    Reviewed-on: http://review.gluster.org/15966
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: Prashanth Pai <ppai>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>

Comment 1 Worker Ant 2016-12-04 04:39:17 UTC
REVIEW: http://review.gluster.org/16021 (eventsapi: Push Messages to Webhooks in parallel) posted (#1) for review on release-3.9 by Atin Mukherjee (amukherj)

Comment 2 Worker Ant 2016-12-07 12:23:37 UTC
COMMIT: http://review.gluster.org/16021 committed in release-3.9 by Aravinda VK (avishwan) 
------
commit 00aa042e6183ef3cb5386f587c2481cd978d9dcf
Author: Aravinda VK <avishwan>
Date:   Tue Nov 29 16:32:54 2016 +0530

    eventsapi: Push Messages to Webhooks in parallel
    
    With this patch, glustereventsd will maintain one thread per
    webhook. If a webhook is slow, then all the events to that worker
    will be delayed but it will not affect the other webhooks.
    
    Note: Webhook in transit may get missed if glustereventsd reloads due to
    new Webhook addition or due configuration changes.
    
    >Reviewed-on: http://review.gluster.org/15966
    >Smoke: Gluster Build System <jenkins.org>
    >Reviewed-by: Prashanth Pai <ppai>
    >NetBSD-regression: NetBSD Build System <jenkins.org>
    >CentOS-regression: Gluster Build System <jenkins.org>
    
    BUG: 1401261
    Change-Id: I2d11e01c7ac434355bc356ff75396252f51b339b
    Signed-off-by: Aravinda VK <avishwan>
    Reviewed-on: http://review.gluster.org/16021
    Tested-by: Atin Mukherjee <amukherj>
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Prashanth Pai <ppai>

Comment 3 Kaushal 2017-03-08 10:20:58 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.9.1, please open a new bug report.

glusterfs-3.9.1 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2017-January/029725.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.