Bug 840655 - CTDB: Noticeable delay in CTDB failover(ranging between 6-14 mins)
CTDB: Noticeable delay in CTDB failover(ranging between 6-14 mins)
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterfs (Show other bugs)
x86_64 Linux
high Severity medium
: ---
: ---
Assigned To: krishnan parthasarathi
Depends On: 836448
Blocks: 840813
  Show dependency treegraph
Reported: 2012-07-16 16:06 EDT by Scott Haines
Modified: 2015-11-03 18:04 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 836448
: 840813 (view as bug list)
Last Closed: 2012-09-11 10:23:40 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Scott Haines 2012-07-16 16:06:23 EDT
+++ This bug was initially created as a clone of Bug #836448 +++

Description of problem:
CTDB: Noticeable delay in CTDB failover(ranging between 6-14 mins)

Version-Release number of selected component (if applicable):
CTDB version:
glusterfs 3.3.0

How reproducible:

Steps to Reproduce:
1.Configure Automated IP Fail-over for NFS and CIFS as per the Doc RHS 2.0 Administration Guide -(section -Configuring Automated IP Fail-over for NFS and CIFS)
2. now create Volumes and (NFS or CIFS)mount them using VIPs.
3. create few dir and files on mounted volume.
4. check the node status using 'ctdb status' and check ip mapping using 'ctdb ip'
5. Power off any one storage server who is part of that CTDB failover configuration.
6. Check the ctdb status from storage server using 'ctdb status' 

Actual results:
when one server is brought down, many times delay has been found in CTDB 
failover. ranging between 6-14 mins. For significantly large delays (~40mins, observed only once), CTDB's 'cluster membership' algorithm bans the node failing to 'recover'

Expected results:
Time taken for failover should not be in minutes.

Additional info:
--'ls' on /gluster/lock takes time 'proportional' to the delay noticed in 'failover'
-- It takes time in identifying that one node/storage server is down.

--- Additional comment from kparthas@redhat.com on 2012-07-02 00:57:23 EDT ---

This issue is seen due to a bug in the ping timer logic. The purpose of the ping timer is to assert the absence of any evidence the server is possibly alive.

The current implementation updates the 'last_sent' timer in the following
points in time,
- rpc_clnt_submit: when rpc messages are being queued at the transport
layer. (wrong!, since we have no way to determine if server actually
received the message)
- rpc_clnt_notify: when client receives pollout event on sending a message
on the 'wire'. (correct, since it indicates ACK from server)

The fix is to remove the 'incorrect' update of 'last_sent'. The fix is already pushed into the master branch, http://review.gluster.com/3625.

--- Additional comment from amarts@redhat.com on 2012-07-11 02:06:19 EDT ---

patch accepted in master.
Comment 1 Amar Tumballi 2012-07-18 05:37:42 EDT
Need to check if that patch, which got accepted in master is enough for this. KP, please take care of this bug.
Comment 2 Ujjwala 2012-08-28 04:33:27 EDT
Verified it on RHS 2.0.z update 2 and the ctdb failover time varies between 70 to 90 seconds.
Tested it for Distribute, Replicate and Distributed-Replicate volumes on both cifs and nfs mounts.

# glusterfs -V
glusterfs 3.3.0rhs built on Aug 17 2012 07:06:58
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
You may redistribute copies of GlusterFS under the terms of the GNU General
Comment 4 errata-xmlrpc 2012-09-11 10:23:40 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.