Bug 822827 - [glusterfs-3.3.0beta4]: quota remove command makes glusterfs client process segfault
[glusterfs-3.3.0beta4]: quota remove command makes glusterfs client process s...
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: quota (Show other bugs)
mainline
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Raghavendra Bhat
:
Depends On:
Blocks: 817967
  Show dependency treegraph
 
Reported: 2012-05-18 05:36 EDT by Raghavendra Bhat
Modified: 2013-07-24 13:09 EDT (History)
2 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-07-24 13:09:20 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions: glusterfs-3.3.0
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Raghavendra Bhat 2012-05-18 05:36:53 EDT
Description of problem:
after setting a quota limit on a certain directory, if that limit is removed via cli, then the glusterfs client segfaults while reconfiguring according to the new changes.

This is the backtrace of the core generated.

Core was generated by `/usr/local/sbin/glusterfs --volfile-id=mirror --volfile-server=hyperspace /mnt/'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f9445fac8b7 in reconfigure (this=0x7f9438068e10, options=0x7f9445573aa0) at ../../../../../xlators/features/quota/src/quota.c:3254
3254	                                __quota_reconfigure (this, top->itable, limit);
(gdb) bt
#0  0x00007f9445fac8b7 in reconfigure (this=0x7f9438068e10, options=0x7f9445573aa0) at ../../../../../xlators/features/quota/src/quota.c:3254
#1  0x00007f94476b0e5c in xlator_reconfigure_rec (old_xl=0x7f9438068e10, new_xl=0x7f943846c7b0) at ../../../libglusterfs/src/options.c:942
#2  0x00007f94476b0d87 in xlator_reconfigure_rec (old_xl=0x7f94380697d0, new_xl=0x7f943846d170) at ../../../libglusterfs/src/options.c:925
#3  0x00007f94476b0d87 in xlator_reconfigure_rec (old_xl=0x7f943805ea00, new_xl=0x7f943846db30) at ../../../libglusterfs/src/options.c:925
#4  0x00007f94476b0d87 in xlator_reconfigure_rec (old_xl=0x7f943805f3c0, new_xl=0x7f943846e4f0) at ../../../libglusterfs/src/options.c:925
#5  0x00007f94476b0d87 in xlator_reconfigure_rec (old_xl=0x7f943805fd80, new_xl=0x7f943846eeb0) at ../../../libglusterfs/src/options.c:925
#6  0x00007f94476b0d87 in xlator_reconfigure_rec (old_xl=0x7f9438061070, new_xl=0x7f943846f870) at ../../../libglusterfs/src/options.c:925
#7  0x00007f94476b0d87 in xlator_reconfigure_rec (old_xl=0x7f9438061a30, new_xl=0x7f9438470230) at ../../../libglusterfs/src/options.c:925
#8  0x00007f94476b0f66 in xlator_option_info_list (list=0x7f9438061a30, key=0x7f9438470230 "p\264\001\070\224\177", def_val=0x7f94476b0f66, 
    descr=0x7fff71037890) at ../../../libglusterfs/src/options.c:979
#9  0x00007f94476a0eec in uuid_copy (
    dst=0x7f94476a0eec "\213E\330H\211E\360H\213E\320H\211E\350\353\033H\213E\350\017\266\020H\213E\360\210\020H\203E\360\001H\203E\350\001\203E\374\001\203}\374\017~\337\311\303UH\211\345SH\203\354(\213\005\265\026#", src=0x7fff710378c0 "\020y\003q\377\177")
    at ../../../contrib/uuid/copy.c:43
#10 0x000000000040d1ec in glusterfs_volfile_reconfigure (newvolfile_fp=0x7f943800c920) at ../../../glusterfsd/src/glusterfsd-mgmt.c:1514
#11 0x000000000040d4d1 in mgmt_getspec_cbk (req=0x1510df8, iov=0x1510e38, count=1, myframe=0x7f944572ad18)
    at ../../../glusterfsd/src/glusterfsd-mgmt.c:1586
#12 0x00007f944743ac91 in rpc_clnt_handle_reply (clnt=0x15102d0, pollin=0x7f943800fd30) at ../../../../rpc/rpc-lib/src/rpc-clnt.c:788
#13 0x00007f944743b009 in rpc_clnt_notify (trans=0x1515040, mydata=0x1510300, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7f943800fd30)
    at ../../../../rpc/rpc-lib/src/rpc-clnt.c:907
#14 0x00007f9447437021 in rpc_transport_notify (this=0x1515040, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7f943800fd30)
    at ../../../../rpc/rpc-lib/src/rpc-transport.c:489
#15 0x00007f94437553a5 in socket_event_poll_in (this=0x1515040) at ../../../../../rpc/rpc-transport/socket/src/socket.c:1677
#16 0x00007f9443755919 in socket_event_handler (fd=6, idx=0, data=0x1515040, poll_in=1, poll_out=0, poll_err=0)
    at ../../../../../rpc/rpc-transport/socket/src/socket.c:1792
#17 0x00007f9447691cf1 in event_dispatch_epoll (event_pool=0x1508cd0) at ../../../libglusterfs/src/event.c:801
#18 0x00007f9447691f0b in event_pool_new (count=0) at ../../../libglusterfs/src/event.c:876
#19 0x00007f944769227d in list_add (new=0xffffffff47af82e8, head=0x1255) at ../../../libglusterfs/src/list.h:33
#20 0x0000000000408858 in main (argc=4, argv=0x7fff71037e78) at ../../../glusterfsd/src/glusterfsd.c:1674
(gdb)  f 0
#0  0x00007f9445fac8b7 in reconfigure (this=0x7f9438068e10, options=0x7f9445573aa0) at ../../../../../xlators/features/quota/src/quota.c:3254
3254	                                __quota_reconfigure (this, top->itable, limit);
(gdb) p this
$1 = (xlator_t *) 0x7f9438068e10
(gdb) p *this
$2 = {name = 0x7f943801d260 "mirror-quota", type = 0x7f943801d5e0 "features/quota", next = 0x7f9438068450, prev = 0x7f94380697d0, 
  parents = 0x7f94380503f0, children = 0x7f943802f0c0, options = 0x7f9445572ae0, dlhandle = 0x7f9438056070, fops = 0x7f94461b09c0, 
  cbks = 0x7f94461b0c60, dumpops = 0x0, volume_options = {next = 0x7f9438076160, prev = 0x7f9438076160}, fini = 0x7f9445fac968 <fini>, 
  init = 0x7f9445fac0f1 <init>, reconfigure = 0x7f9445fac6a0 <reconfigure>, mem_acct_init = 0x7f9445fabba0 <mem_acct_init>, 
  notify = 0x7f9447670f6f <default_notify>, loglevel = GF_LOG_NONE, latencies = {{min = 0, max = 0, total = 0, std = 0, mean = 0, 
      count = 0} <repeats 46 times>}, history = 0x0, ctx = 0x14ec010, graph = 0x7f9438011ab0, itable = 0x0, init_succeeded = 1 '\001', 
  private = 0x7f943804bb70, mem_acct = {num_types = 99, rec = 0x7f9438089ce0}, winds = 0, switched = 0 '\000', local_pool = 0x7f9438022e90}
(gdb) p top
$3 = (xlator_t *) 0x0
(gdb) l quota.c:3205
3200	
3201	out:
3202	        return;
3203	}
3204	
3205	
3206	int
3207	reconfigure (xlator_t *this, dict_t *options)
3208	{
3209	        int32_t           ret   = -1;
(gdb) 
3210	        quota_priv_t     *priv  = NULL;
3211	        limits_t         *limit = NULL, *next = NULL, *new = NULL;
3212	        struct list_head  head  = {0, };
3213	        xlator_t         *top   = NULL;
3214	        char              found = 0;
3215	
3216	        priv = this->private;
3217	
3218	        INIT_LIST_HEAD (&head);
3219	
(gdb) 
3220	        LOCK (&priv->lock);
3221	        {
3222	                list_splice_init (&priv->limit_head, &head);
3223	        }
3224	        UNLOCK (&priv->lock);
3225	
3226	        ret = quota_parse_limits (priv, this, options, &head);
3227	        if (ret == -1) {
3228	                gf_log ("quota", GF_LOG_WARNING,
3229	                        "quota reconfigure failed, "
(gdb) 
3230	                        "new changes will not take effect");
3231	                goto out;
3232	        }
3233	
3234	        LOCK (&priv->lock);
3235	        {
3236	                list_for_each_entry (limit, &priv->limit_head, limit_list) {
3237	                        top = ((glusterfs_ctx_t *)this->ctx)->active->top;
3238	
3239	                        __quota_reconfigure (this, top->itable, limit);
(gdb) info thr
  7 Thread 4697  0x00007f9446fff4bd in nanosleep () at ../sysdeps/unix/syscall-template.S:82
  6 Thread 4701  0x00007f9446ffecbd in read () at ../sysdeps/unix/syscall-template.S:82
  5 Thread 4696  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
  4 Thread 4700  0x00007f94469aa4f9 in __libc_readv (fd=5, vector=0x7f9440c0fe50, count=2) at ../sysdeps/unix/sysv/linux/readv.c:56
  3 Thread 4695  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
  2 Thread 4694  do_sigwait (set=<value optimized out>, sig=0x7f9445162eb8)
    at ../nptl/sysdeps/unix/sysv/linux/../../../../../sysdeps/unix/sysv/linux/sigwait.c:65
* 1 Thread 4693  0x00007f9445fac8b7 in reconfigure (this=0x7f9438068e10, options=0x7f9445573aa0)
    at ../../../../../xlators/features/quota/src/quota.c:3254
(gdb) 





Version-Release number of selected component (if applicable):


How reproducible:

always
Steps to Reproduce:
1. create a volume, start volume and mount it
2. enable quota on the volume and set a limit on the volume
3. remove the limit set on the volume
  
Actual results:
glusterfs process crashed

Expected results:
glusterfs process should not crash

Additional info:

+------------------------------------------------------------------------------+
[2012-05-18 14:50:20.330683] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mirror-client-0: changing port to 24053 (from 0)
[2012-05-18 14:50:20.330844] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mirror-client-1: changing port to 24043 (from 0)
[2012-05-18 14:50:24.322354] I [client-handshake.c:1628:select_server_supported_programs] 0-mirror-client-0: Using Program GlusterFS 3.3.0beta
4, Num (1298437), Version (330)
[2012-05-18 14:50:24.323297] I [client-handshake.c:1425:client_setvolume_cbk] 0-mirror-client-0: Connected to 127.0.1.1:24053, attached to rem
ote volume '/tmp/dir3'.
[2012-05-18 14:50:24.323377] I [client-handshake.c:1437:client_setvolume_cbk] 0-mirror-client-0: Server and Client lk-version numbers are not 
same, reopening the fds
[2012-05-18 14:50:24.323580] I [afr-common.c:3620:afr_notify] 0-mirror-replicate-0: Subvolume 'mirror-client-0' came back up; going online.
[2012-05-18 14:50:24.323910] I [client-handshake.c:1628:select_server_supported_programs] 0-mirror-client-1: Using Program GlusterFS 3.3.0beta
4, Num (1298437), Version (330)
[2012-05-18 14:50:24.324227] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mirror-client-0: Server lk version = 1
[2012-05-18 14:50:24.324562] I [client-handshake.c:1425:client_setvolume_cbk] 0-mirror-client-1: Connected to 127.0.1.1:24043, attached to rem
ote volume '/tmp/dir6'.
[2012-05-18 14:50:24.324620] I [client-handshake.c:1437:client_setvolume_cbk] 0-mirror-client-1: Server and Client lk-version numbers are not 
same, reopening the fds
[2012-05-18 14:50:24.336644] I [fuse-bridge.c:4138:fuse_graph_setup] 0-fuse: switched to graph 0
[2012-05-18 14:50:24.336782] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mirror-client-1: Server lk version = 1
[2012-05-18 14:50:24.336845] I [fuse-bridge.c:3376:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.16
[2012-05-18 14:50:24.337640] I [afr-common.c:1962:afr_set_root_inode_on_first_lookup] 0-mirror-replicate-0: added root inode
[2012-05-18 14:50:33.670473] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2012-05-18 14:50:34.697266] I [quota.c:3060:quota_parse_limits] 0-mirror-quota: /:23622320128
[2012-05-18 14:50:34.697288] I [quota.c:3092:quota_parse_limits] 0-mirror-quota: /:23622320128
[2012-05-18 14:50:34.697360] I [io-cache.c:1549:check_cache_size_ok] 0-mirror-io-cache: Max cache size is 4066508800
[2012-05-18 14:50:34.697403] I [io-cache.c:1549:check_cache_size_ok] 0-mirror-quick-read: Max cache size is 4066508800
[2012-05-18 14:50:40.647930] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2012-05-18 14:50:41.676985] I [quota.c:3085:quota_parse_limits] 0-mirror-quota: no "limit-set" option provided
pending frames:

patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 2012-05-18 14:50:41
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
:



gluster volume info
 
Volume Name: mirror
Type: Replicate
Volume ID: d4d776ee-2695-4470-8e12-21526b6ca72d
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: hyperspace:/tmp/dir3
Brick2: hyperspace:/tmp/dir6
Options Reconfigured:
features.quota: on
root@hyperspace:~#
Comment 1 Anand Avati 2012-05-18 22:31:44 EDT
CHANGE: http://review.gluster.com/3364 (features/quota: get the top xlator from the active graph outside the list) merged in master by Anand Avati (avati@redhat.com)
Comment 2 Vijay Bellur 2012-05-19 02:14:43 EDT
Can we backport this to release-3.3 please?
Comment 3 Raghavendra Bhat 2012-06-11 03:13:09 EDT
http://review.gluster.com/3379 fixes the issue in release-3.3 as well. Checked with glusterfs-3.3.0.
Comment 4 Raghavendra Bhat 2012-06-11 03:13:32 EDT
http://review.gluster.com/3379 fixes the issue in release-3.3 as well. Checked with glusterfs-3.3.0.

Note You need to log in before you can comment on or make changes to this bug.