Bug 840813 - CTDB: Noticeable delay in CTDB failover(ranging between 6-14 mins)
CTDB: Noticeable delay in CTDB failover(ranging between 6-14 mins)
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterfs (Show other bugs)
x86_64 Linux
unspecified Severity medium
: ---
: RHGS 2.1.0
Assigned To: krishnan parthasarathi
Sudhir D
Depends On: 836448 840655
  Show dependency treegraph
Reported: 2012-07-17 05:43 EDT by Scott Haines
Modified: 2015-11-03 18:04 EST (History)
7 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0qa5-1
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 840655
Last Closed: 2013-09-23 18:32:50 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Scott Haines 2012-07-17 05:43:15 EDT
+++ This bug was initially created as a clone of Bug #840655 +++

+++ This bug was initially created as a clone of Bug #836448 +++

Description of problem:
CTDB: Noticeable delay in CTDB failover(ranging between 6-14 mins)

Version-Release number of selected component (if applicable):
CTDB version:
glusterfs 3.3.0

How reproducible:

Steps to Reproduce:
1.Configure Automated IP Fail-over for NFS and CIFS as per the Doc RHS 2.0 Administration Guide -(section -Configuring Automated IP Fail-over for NFS and CIFS)
2. now create Volumes and (NFS or CIFS)mount them using VIPs.
3. create few dir and files on mounted volume.
4. check the node status using 'ctdb status' and check ip mapping using 'ctdb ip'
5. Power off any one storage server who is part of that CTDB failover configuration.
6. Check the ctdb status from storage server using 'ctdb status' 

Actual results:
when one server is brought down, many times delay has been found in CTDB 
failover. ranging between 6-14 mins. For significantly large delays (~40mins, observed only once), CTDB's 'cluster membership' algorithm bans the node failing to 'recover'

Expected results:
Time taken for failover should not be in minutes.

Additional info:
--'ls' on /gluster/lock takes time 'proportional' to the delay noticed in 'failover'
-- It takes time in identifying that one node/storage server is down.

--- Additional comment from kparthas@redhat.com on 2012-07-02 00:57:23 EDT ---

This issue is seen due to a bug in the ping timer logic. The purpose of the ping timer is to assert the absence of any evidence the server is possibly alive.

The current implementation updates the 'last_sent' timer in the following
points in time,
- rpc_clnt_submit: when rpc messages are being queued at the transport
layer. (wrong!, since we have no way to determine if server actually
received the message)
- rpc_clnt_notify: when client receives pollout event on sending a message
on the 'wire'. (correct, since it indicates ACK from server)

The fix is to remove the 'incorrect' update of 'last_sent'. The fix is already pushed into the master branch, http://review.gluster.com/3625.

--- Additional comment from amarts@redhat.com on 2012-07-11 02:06:19 EDT ---

patch accepted in master.
Comment 1 Amar Tumballi 2012-07-18 05:36:36 EDT
KP, let me know your thoughts on this, will need to do a devel ack
Comment 4 surabhi 2013-07-02 07:13:19 EDT
verified in version:
glusterfs built on Jun 28 2013 06:41:38
Comment 6 Scott Haines 2013-09-23 18:32:50 EDT
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.