Bug 1401261

Summary:	Delayed Events if any one Webhook is slow
Product:	[Community] GlusterFS	Reporter:	Atin Mukherjee <amukherj>
Component:	eventsapi	Assignee:	Aravinda VK <avishwan>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	3.9	CC:	avishwan
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.9.1	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1357754	Environment:
Last Closed:	2017-03-08 10:20:58 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1357754
Bug Blocks:	1395613

Description Atin Mukherjee 2016-12-04 04:38:16 UTC

+++ This bug was initially created as a clone of Bug #1357754 +++

Description of problem:
If Webhook takes more time to consume the events, it may affect the further pushing of Events. Add configurable Timeout option while calling Webhook.

# Example, Set Timeout in seconds
gluster-eventsapi config-set webhook-timeout 30

--- Additional comment from Aravinda VK on 2016-11-16 04:50:13 EST ---

Also observed that, if one webhook is slow then future events gets delayed. Webhooks should be called asynchronously to avoid delays.

--- Additional comment from Worker Ant on 2016-11-29 06:13:16 EST ---

REVIEW: http://review.gluster.org/15966 (eventsapi: Push Messages to Webhooks in parallel) posted (#1) for review on master by Aravinda VK (avishwan)

--- Additional comment from Aravinda VK on 2016-11-29 06:16:38 EST ---

Separate threads maintained for each Webhook so that slow Webhook will not affect the other webhooks. With this change we do not need configurable option for Timeout.

--- Additional comment from Worker Ant on 2016-12-02 01:25:41 EST ---

REVIEW: http://review.gluster.org/15966 (eventsapi: Push Messages to Webhooks in parallel) posted (#2) for review on master by Aravinda VK (avishwan)

--- Additional comment from Worker Ant on 2016-12-02 09:52:59 EST ---

COMMIT: http://review.gluster.org/15966 committed in master by Aravinda VK (avishwan) 
------
commit 99458974b7d12bd12d78d4b9a19adfb62d771b5c
Author: Aravinda VK <avishwan>
Date:   Tue Nov 29 16:32:54 2016 +0530

    eventsapi: Push Messages to Webhooks in parallel
    
    With this patch, glustereventsd will maintain one thread per
    webhook. If a webhook is slow, then all the events to that worker
    will be delayed but it will not affect the other webhooks.
    
    Note: Webhook in transit may get missed if glustereventsd reloads due to
    new Webhook addition or due configuration changes.
    
    BUG: 1357754
    Change-Id: I2d11e01c7ac434355bc356ff75396252f51b339b
    Signed-off-by: Aravinda VK <avishwan>
    Reviewed-on: http://review.gluster.org/15966
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: Prashanth Pai <ppai>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>

Comment 1 Worker Ant 2016-12-04 04:39:17 UTC

REVIEW: http://review.gluster.org/16021 (eventsapi: Push Messages to Webhooks in parallel) posted (#1) for review on release-3.9 by Atin Mukherjee (amukherj)

Comment 2 Worker Ant 2016-12-07 12:23:37 UTC

COMMIT: http://review.gluster.org/16021 committed in release-3.9 by Aravinda VK (avishwan) 
------
commit 00aa042e6183ef3cb5386f587c2481cd978d9dcf
Author: Aravinda VK <avishwan>
Date:   Tue Nov 29 16:32:54 2016 +0530

    eventsapi: Push Messages to Webhooks in parallel
    
    With this patch, glustereventsd will maintain one thread per
    webhook. If a webhook is slow, then all the events to that worker
    will be delayed but it will not affect the other webhooks.
    
    Note: Webhook in transit may get missed if glustereventsd reloads due to
    new Webhook addition or due configuration changes.
    
    >Reviewed-on: http://review.gluster.org/15966
    >Smoke: Gluster Build System <jenkins.org>
    >Reviewed-by: Prashanth Pai <ppai>
    >NetBSD-regression: NetBSD Build System <jenkins.org>
    >CentOS-regression: Gluster Build System <jenkins.org>
    
    BUG: 1401261
    Change-Id: I2d11e01c7ac434355bc356ff75396252f51b339b
    Signed-off-by: Aravinda VK <avishwan>
    Reviewed-on: http://review.gluster.org/16021
    Tested-by: Atin Mukherjee <amukherj>
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Prashanth Pai <ppai>

Comment 3 Kaushal 2017-03-08 10:20:58 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.9.1, please open a new bug report.

glusterfs-3.9.1 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2017-January/029725.html
[2] https://www.gluster.org/pipermail/gluster-users/