Bug 840813

Summary:	CTDB: Noticeable delay in CTDB failover(ranging between 6-14 mins)
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Scott Haines <shaines>
Component:	glusterfs	Assignee:	krishnan parthasarathi <kparthas>
Status:	CLOSED ERRATA	QA Contact:	Sudhir D <sdharane>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	2.0	CC:	amarts, fharshav, gluster-bugs, nsathyan, racpatel, rfortier, sbhaloth
Target Milestone:	---
Target Release:	RHGS 2.1.0
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.4.0qa5-1	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	840655	Environment:
Last Closed:	2013-09-23 22:32:50 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	836448, 840655
Bug Blocks:

Description Scott Haines 2012-07-17 09:43:15 UTC

+++ This bug was initially created as a clone of Bug #840655 +++

+++ This bug was initially created as a clone of Bug #836448 +++

Description of problem:
CTDB: Noticeable delay in CTDB failover(ranging between 6-14 mins)

Version-Release number of selected component (if applicable):
CTDB version: 1.0.114.5-1.el6
glusterfs 3.3.0

How reproducible:
Often

Steps to Reproduce:
1.Configure Automated IP Fail-over for NFS and CIFS as per the Doc RHS 2.0 Administration Guide -(section -Configuring Automated IP Fail-over for NFS and CIFS)
2. now create Volumes and (NFS or CIFS)mount them using VIPs.
3. create few dir and files on mounted volume.
4. check the node status using 'ctdb status' and check ip mapping using 'ctdb ip'
5. Power off any one storage server who is part of that CTDB failover configuration.
6. Check the ctdb status from storage server using 'ctdb status' 

  
Actual results:
when one server is brought down, many times delay has been found in CTDB 
failover. ranging between 6-14 mins. For significantly large delays (~40mins, observed only once), CTDB's 'cluster membership' algorithm bans the node failing to 'recover'

Expected results:
Time taken for failover should not be in minutes.

Additional info:
--'ls' on /gluster/lock takes time 'proportional' to the delay noticed in 'failover'
-- It takes time in identifying that one node/storage server is down.

--- Additional comment from kparthas on 2012-07-02 00:57:23 EDT ---

This issue is seen due to a bug in the ping timer logic. The purpose of the ping timer is to assert the absence of any evidence the server is possibly alive.

The current implementation updates the 'last_sent' timer in the following
points in time,
- rpc_clnt_submit: when rpc messages are being queued at the transport
layer. (wrong!, since we have no way to determine if server actually
received the message)
- rpc_clnt_notify: when client receives pollout event on sending a message
on the 'wire'. (correct, since it indicates ACK from server)

The fix is to remove the 'incorrect' update of 'last_sent'. The fix is already pushed into the master branch, http://review.gluster.com/3625.

--- Additional comment from amarts on 2012-07-11 02:06:19 EDT ---

patch accepted in master.

Comment 1 Amar Tumballi 2012-07-18 09:36:36 UTC

KP, let me know your thoughts on this, will need to do a devel ack

Comment 4 surabhi 2013-07-02 11:13:19 UTC

verified in version:
glusterfs 3.4.0.12rhs.beta1 built on Jun 28 2013 06:41:38

Comment 6 Scott Haines 2013-09-23 22:32:50 UTC

Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html