Bug 1105102 - CTDB:Adding volume set option in hook script causes delay in glusterd operations.
Summary: CTDB:Adding volume set option in hook script causes delay in glusterd operati...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: samba
Version: rhgs-3.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: RHGS 3.0.0
Assignee: Raghavendra Talur
QA Contact: surabhi
URL:
Whiteboard:
Depends On: 1092242 1294224
Blocks: 1105118
TreeView+ depends on / blocked
 
Reported: 2014-06-05 11:48 UTC by surabhi
Modified: 2015-12-25 15:18 UTC (History)
6 users (show)

Fixed In Version: glusterfs-3.6.0.17-1.el6rhs
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1105118 (view as bug list)
Environment:
Last Closed: 2014-09-22 19:40:40 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2014:1278 0 normal SHIPPED_LIVE Red Hat Storage Server 3.0 bug fix and enhancement update 2014-09-22 23:26:55 UTC

Description surabhi 2014-06-05 11:48:06 UTC
Description of problem:
In a ctdb setup when we start a volume the hook scripts are supposed to mount the ctdb vol on /gluster/lock and add the entry to fstab. 

With the changes in hook script as follows  where we are adding volume set option for ping timeout for ctdb volume , and then do the mount and add entry in fstab ,it is causing delay in mount and showing inconsistent behaviour.
script location:
/var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh

function add_ping_timeout () {
    volname=$1
    value=$2
    gluster volume set $volname network.ping-timeout $value
}

sleep 5
    # Make sure ping-timeout is not default for CTDB volume
    add_ping_timeout $VOL $PING_TIMEOUT_SECS;
    mount -t glusterfs `hostname`:$VOL "$CTDB_MNT" && \
        add_fstab_entry $VOL $CTDB_MNT

As in ctdb customer scenario we run these hook scripts on multiple nodes(scaling)together and running these gluster volume set option causes delay in gluster operations and we get following error.

ctdb: failed: Another transaction is in progress. Please try again after sometime.

*****************
If we need ping timeout value to be set for ctdb volume it would be better to add it manually after we create the ctdb volume instead of handling it in hook script.

Version-Release number of selected component (if applicable):
glusterfs-geo-replication-3.6.0.11-1.el6rhs.x86_64
glusterfs-api-3.6.0.11-1.el6rhs.x86_64
glusterfs-server-3.6.0.11-1.el6rhs.x86_64
glusterfs-debuginfo-3.6.0.11-1.el6rhs.x86_64
glusterfs-libs-3.6.0.11-1.el6rhs.x86_64
glusterfs-fuse-3.6.0.11-1.el6rhs.x86_64
samba-glusterfs-3.6.9-168.2.el6rhs.x86_64
glusterfs-devel-3.6.0.11-1.el6rhs.x86_64
glusterfs-rdma-3.6.0.11-1.el6rhs.x86_64
glusterfs-api-devel-3.6.0.11-1.el6rhs.x86_64
glusterfs-cli-3.6.0.11-1.el6rhs.x86_64
glusterfs-3.6.0.11-1.el6rhs.x86_64


How reproducible:
Always

Steps to Reproduce:
1.Create a ctdb volume
2.start the volume


Actual results:
ctdb volume starts but the mount is taking long time and any gluster operation
gets delayed and give following error:

ctdb: failed: Another transaction is in progress. Please try again after sometime.

Expected results:
On all the nodes the mount of ctdb volume happens immediately the volume is started and none of the glusterd operations should be delayed.


Additional info:
We can add the ping time out option manually for ctdb volume or should handle it out of hook scripts because it may cause issues in multi-node environment.

Comment 3 Raghavendra Talur 2014-06-12 10:55:50 UTC
Patch posted at https://code.engineering.redhat.com/gerrit/26746.

Comment 4 surabhi 2014-06-17 09:38:02 UTC
With the fix in glusterfs-server-3.6.0.17-1.el6rhs.x86_64 where ping-time-out is part of mount option,IP:/ctdb_vol /gluster/lock glusterfs _netdev,defaults,transport=tcp,xlator-option=*client*.ping-timeout=10 0 0
I tried creating a new ctdb volume and started it.There is no delay seen in gluster operations and the volume is mounted on /gluster/lock on all the nodes.
Marking the BZ Verified.

Comment 6 errata-xmlrpc 2014-09-22 19:40:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2014-1278.html


Note You need to log in before you can comment on or make changes to this bug.