Bug 1357754

Summary:	Delayed Events if any one Webhook is slow
Product:	[Community] GlusterFS	Reporter:	Aravinda VK <avishwan>
Component:	eventsapi	Assignee:	Aravinda VK <avishwan>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	mainline
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.10.0	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1395613 1401261 (view as bug list)		Environment:
Last Closed:	2017-03-06 17:21:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1395613, 1401261

Description Aravinda VK 2016-07-19 05:46:50 UTC

Description of problem:
If Webhook takes more time to consume the events, it may affect the further pushing of Events. Add configurable Timeout option while calling Webhook.

# Example, Set Timeout in seconds
gluster-eventsapi config-set webhook-timeout 30

Comment 1 Aravinda VK 2016-11-16 09:50:13 UTC

Also observed that, if one webhook is slow then future events gets delayed. Webhooks should be called asynchronously to avoid delays.

Comment 2 Worker Ant 2016-11-29 11:13:16 UTC

REVIEW: http://review.gluster.org/15966 (eventsapi: Push Messages to Webhooks in parallel) posted (#1) for review on master by Aravinda VK (avishwan)

Comment 3 Aravinda VK 2016-11-29 11:16:38 UTC

Separate threads maintained for each Webhook so that slow Webhook will not affect the other webhooks. With this change we do not need configurable option for Timeout.

Comment 4 Worker Ant 2016-12-02 06:25:41 UTC

REVIEW: http://review.gluster.org/15966 (eventsapi: Push Messages to Webhooks in parallel) posted (#2) for review on master by Aravinda VK (avishwan)

Comment 5 Worker Ant 2016-12-02 14:52:59 UTC

COMMIT: http://review.gluster.org/15966 committed in master by Aravinda VK (avishwan) 
------
commit 99458974b7d12bd12d78d4b9a19adfb62d771b5c
Author: Aravinda VK <avishwan>
Date:   Tue Nov 29 16:32:54 2016 +0530

    eventsapi: Push Messages to Webhooks in parallel
    
    With this patch, glustereventsd will maintain one thread per
    webhook. If a webhook is slow, then all the events to that worker
    will be delayed but it will not affect the other webhooks.
    
    Note: Webhook in transit may get missed if glustereventsd reloads due to
    new Webhook addition or due configuration changes.
    
    BUG: 1357754
    Change-Id: I2d11e01c7ac434355bc356ff75396252f51b339b
    Signed-off-by: Aravinda VK <avishwan>
    Reviewed-on: http://review.gluster.org/15966
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: Prashanth Pai <ppai>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>

Comment 6 Shyamsundar 2017-03-06 17:21:32 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.0, please open a new bug report.

glusterfs-3.10.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2017-February/030119.html
[2] https://www.gluster.org/pipermail/gluster-users/