Description of problem: glusterfs client crashed with the below backtrace while migrating the fds from old graph to the newer graph. (gdb) bt #0 0x00007f00bf7a51c6 in dht_fsync (frame=0x7f00c9351250, this=<value optimized out>, fd=0x366242c, datasync=0, xdata=0x0) at dht-inode-read.c:818 #1 0x00007f00bf560359 in wb_fsync (frame=0x7f00c93513a8, this=0x31b9a50, fd=0x366242c, datasync=0, xdata=0x0) at write-behind.c:1523 #2 0x00007f00bf34fa9a in ra_fsync (frame=0x7f00c93515ac, this=0x31ba4b0, fd=0x366242c, datasync=0, xdata=0x0) at read-ahead.c:628 #3 0x0000003c7e0265cb in default_fsync (frame=0x7f00c93515ac, this=0x31bb000, fd=0x366242c, flags=0, xdata=<value optimized out>) at defaults.c:1795 #4 0x0000003c7e0265cb in default_fsync (frame=0x7f00c93515ac, this=0x31bbc10, fd=0x366242c, flags=0, xdata=<value optimized out>) at defaults.c:1795 #5 0x0000003c7e0265cb in default_fsync (frame=0x7f00c93515ac, this=0x31bc7a0, fd=0x366242c, flags=0, xdata=<value optimized out>) at defaults.c:1795 #6 0x00007f00beb21722 in mdc_fsync (frame=0x7f00c9351d10, this=0x31bd3a0, fd=0x366242c, datasync=0, xdata=0x0) at md-cache.c:1654 #7 0x0000003c7e0265cb in default_fsync (frame=0x7f00c9351d10, this=0x31becc0, fd=0x366242c, flags=0, xdata=<value optimized out>) at defaults.c:1795 #8 0x00007f00be6fd915 in io_stats_fsync (frame=0x7f00c9351908, this=0x31bfa40, fd=0x366242c, flags=0, xdata=0x0) at io-stats.c:2194 #9 0x0000003c7e0265cb in default_fsync (frame=0x7f00c9351908, this=0x31c0760, fd=0x366242c, flags=0, xdata=<value optimized out>) at defaults.c:1795 #10 0x0000003c7e0684e2 in syncop_fsync (subvol=0x31c0760, fd=0x366242c, dataonly=0) at syncop.c:1921 #11 0x00007f00c2ac43e5 in fuse_migrate_fd (this=0x12ec8e0, basefd=0x366242c, old_subvol=0x31c0760, new_subvol=0x368a540) at fuse-bridge.c:4382 #12 0x00007f00c2ac459c in fuse_handle_opened_fds (this=0x12ec8e0, old_subvol=0x31c0760, new_subvol=0x368a540) at fuse-bridge.c:4467 #13 0x00007f00c2ac4649 in fuse_graph_switch_task (data=<value optimized out>) at fuse-bridge.c:4518 #14 0x0000003c7e05c222 in synctask_wrap (old_task=<value optimized out>) at syncop.c:333 #15 0x000000304ac438f0 in ?? () from /lib64/libc.so.6 #16 0x0000000000000000 in ?? () f 0 #0 0x00007f00bf7a51c6 in dht_fsync (frame=0x7f00c9351250, this=<value optimized out>, fd=0x366242c, datasync=0, xdata=0x0) at dht-inode-read.c:818 818 STACK_WIND (frame, dht_fsync_cbk, subvol, subvol->fops->fsync, (gdb) l 813 local->call_cnt = 1; 814 local->rebalance.flags = datasync; 815 816 subvol = local->cached_subvol; 817 818 STACK_WIND (frame, dht_fsync_cbk, subvol, subvol->fops->fsync, 819 fd, datasync, xdata); 820 821 return 0; 822 (gdb) p local $1 = <value optimized out> (gdb) p local->cached_subvol $1 = (xlator_t *) 0x0 f 6 #6 0x00007f522eaa5dc3 in default_fsync (frame=0x7f521000122c, this=0xfb1280, fd=0x7f5218001dec, flags=0, xdata=0x0) at ../../../libglusterfs/src/defaults.c:1795 1795 STACK_WIND_TAIL (frame, FIRST_CHILD(this), (gdb) p *fd->inode->fd_ctx There is no member named fd_ctx. (gdb) p *this $2 = {name = 0xfb0ac0 "vol-snapview-client", type = 0xfb1da0 "features/snapview-client", next = 0xfaff00, prev = 0xfb2820, parents = 0xfb3b50, children = 0xfb2760, options = 0xfb1cfc, dlhandle = 0xfb1ea0, fops = 0x7f521fdedc00 <fops>, cbks = 0x7f521fdedee0 <cbks>, dumpops = 0x0, volume_options = {next = 0xfb25f0, prev = 0xfb25f0}, fini = 0x7f521fbea2aa <svc_forget+228>, init = 0x7f521fbe9f69 <svc_flush+729>, reconfigure = 0x7f521fbe9e49 <svc_flush+441>, mem_acct_init = 0x7f521fbe9ef2 <svc_flush+610>, notify = 0x7f521fbea31c <reconfigure+56>, loglevel = GF_LOG_NONE, latencies = {{min = 0, max = 0, total = 0, std = 0, mean = 0, count = 0} <repeats 49 times>}, history = 0x0, ctx = 0xf5f010, graph = 0xf9fad0, itable = 0x0, init_succeeded = 1 '\001', private = 0xfb9eb0, mem_acct = {num_types = 118, rec = 0xfb8c10}, winds = 0, switched = 0 '\000', local_pool = 0xfb9f00, is_autoloaded = _gf_false} (gdb) p *fd->inode->_ctx $3 = {{key = 16454272, xl_key = 0xfb1280}, {value1 = 2, ptr1 = 0x2}, {value2 = 0, ptr2 = 0x0}} (gdb) p *fd->inode->_ctx->xl_key $4 = {name = 0xfb0ac0 "vol-snapview-client", type = 0xfb1da0 "features/snapview-client", next = 0xfaff00, prev = 0xfb2820, parents = 0xfb3b50, children = 0xfb2760, options = 0xfb1cfc, dlhandle = 0xfb1ea0, fops = 0x7f521fdedc00 <fops>, cbks = 0x7f521fdedee0 <cbks>, dumpops = 0x0, volume_options = {next = 0xfb25f0, prev = 0xfb25f0}, fini = 0x7f521fbea2aa <svc_forget+228>, init = 0x7f521fbe9f69 <svc_flush+729>, reconfigure = 0x7f521fbe9e49 <svc_flush+441>, mem_acct_init = 0x7f521fbe9ef2 <svc_flush+610>, notify = 0x7f521fbea31c <reconfigure+56>, loglevel = GF_LOG_NONE, latencies = {{min = 0, max = 0, total = 0, std = 0, mean = 0, count = 0} <repeats 49 times>}, history = 0x0, ctx = 0xf5f010, graph = 0xf9fad0, itable = 0x0, init_succeeded = 1 '\001', private = 0xfb9eb0, mem_acct = {num_types = 118, rec = 0xfb8c10}, winds = 0, switched = 0 '\000', local_pool = 0xfb9f00, is_autoloaded = _gf_false} p *fd->inode->_ctx $5 = {{key = 16454272, xl_key = 0xfb1280}, {value1 = 2, ptr1 = 0x2}, {value2 = 0, ptr2 = 0x0}} Version-Release number of selected component (if applicable): How reproducible: keep doing ls in .snaps directory when USS is enabled and do graph changes (all it has to do is a fd has to be opened and then graph switch has to happen) Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: snapview-client does not implement fsync fop.
REVIEW: http://review.gluster.org/9258 (features/snapview-client: handle fsync fop) posted (#1) for review on master by Raghavendra Bhat (raghavendra)
REVIEW: http://review.gluster.org/9258 (features/snapview-client: handle fsync fop) posted (#2) for review on master by Raghavendra Bhat (raghavendra)
REVIEW: http://review.gluster.org/9258 (features/snapview-client: handle fsync fop) posted (#3) for review on master by Raghavendra Bhat (raghavendra)
REVIEW: http://review.gluster.org/9258 (features/snapview-client: handle fsync fop) posted (#4) for review on master by Raghavendra Bhat (raghavendra)
REVIEW: http://review.gluster.org/9258 (features/snapview-client: handle fsync fop) posted (#5) for review on master by Raghavendra Bhat (raghavendra)
COMMIT: http://review.gluster.org/9258 committed in master by Vijay Bellur (vbellur) ------ commit 1a048844117fe13af0b1eb70eab07d304a23a358 Author: Raghavendra Bhat <raghavendra> Date: Tue Dec 9 23:20:18 2014 +0530 features/snapview-client: handle fsync fop Change-Id: Idaf46bd7497266af837789b09a0c62698f56ee4e BUG: 1172262 Signed-off-by: Raghavendra Bhat <raghavendra> Reviewed-on: http://review.gluster.org/9258 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Sachin Pandit <spandit> Reviewed-by: Vijay Bellur <vbellur>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report. glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user