Bug 1225323
| Summary: | Glusterfs client crash during fd migration after graph switch | |||
|---|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Raghavendra G <rgowdapp> | |
| Component: | unclassified | Assignee: | Raghavendra G <rgowdapp> | |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | ||
| Severity: | unspecified | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | mainline | CC: | bugs | |
| Target Milestone: | --- | |||
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | glusterfs-3.8rc2 | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1225859 (view as bug list) | Environment: | ||
| Last Closed: | 2016-06-16 13:04:41 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1225859 | |||
REVIEW: http://review.gluster.org/10929 (meta: implement fsync(dir)) posted (#1) for review on master by Raghavendra G (rgowdapp) COMMIT: http://review.gluster.org/10929 committed in master by Pranith Kumar Karampuri (pkarampu) ------ commit d6fc353afce03095c98d67d377eb7ddf334fd42e Author: Raghavendra G <rgowdapp> Date: Wed May 27 12:08:54 2015 +0530 meta: implement fsync(dir) Change-Id: I707c608a9803fe6ef86860ca5578d4d3f63fd2aa BUG: 1225323 Signed-off-by: Raghavendra G <rgowdapp> Reviewed-on: http://review.gluster.org/10929 Tested-by: NetBSD Build System Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Pranith Kumar Karampuri <pkarampu> This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report. glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/ [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user |
Description of problem: (gdb) bt #0 0x00007fe56cb27503 in dht_fsync (frame=0x7fe576d701c4, this=<optimized out>, fd=0x7fe560002cec, datasync=0, xdata=0x0) at ../../../../../xlators/cluster/dht/src/dht-inode-read.c:818 #1 0x00007fe56c8db4b8 in wb_fsync (frame=0x7fe576d70118, this=0x7fe56800a2e0, fd=0x7fe560002cec, datasync=0, xdata=0x0) at ../../../../../xlators/performance/write-behind/src/write-behind.c:1523 #2 0x00007fe56c6ccb3a in ra_fsync (frame=0x7fe576d7006c, this=0x7fe56800b6b0, fd=0x7fe560002cec, datasync=0, xdata=0x0) at ../../../../../xlators/performance/read-ahead/src/read-ahead.c:628 #3 0x00007fe577eefb1d in default_fsync (frame=0x7fe576d7006c, this=0x7fe56800c9e0, fd=0x7fe560002cec, flags=0, xdata=0x0) at ../../../libglusterfs/src/defaults.c:1817 #4 0x00007fe577eefb1d in default_fsync (frame=0x7fe576d7006c, this=0x7fe56800dde0, fd=0x7fe560002cec, flags=0, xdata=0x0) at ../../../libglusterfs/src/defaults.c:1817 #5 0x00007fe577eefb1d in default_fsync (frame=0x7fe576d7006c, this=0x7fe56800f0d0, fd=0x7fe560002cec, flags=0, xdata=0x0) at ../../../libglusterfs/src/defaults.c:1817 #6 0x00007fe577efab3b in default_fsync_resume (frame=0x7fe576d70a80, this=0x7fe568010540, fd=0x7fe560002cec, flags=0, xdata=0x0) at ../../../libglusterfs/src/defaults.c:1376 #7 0x00007fe577f14fdd in call_resume (stub=0x7fe5767f7dd4) at ../../../libglusterfs/src/call-stub.c:2576 #8 0x00007fe567dfa578 in open_and_resume (this=this@entry=0x7fe568010540, fd=fd@entry=0x7fe560002cec, stub=0x7fe5767f7dd4) at ../../../../../xlators/performance/open-behind/src/open-behind.c:241 #9 0x00007fe567dfa922 in ob_fsync (frame=0x7fe576d70a80, this=0x7fe568010540, fd=0x7fe560002cec, flag=<optimized out>, xdata=<optimized out>) at ../../../../../xlators/performance/open-behind/src/open-behind.c:498 #10 0x00007fe567becbaf in mdc_fsync (frame=0x7fe576d709d4, this=0x7fe5680118d0, fd=0x7fe560002cec, datasync=0, xdata=0x0) at ../../../../../xlators/performance/md-cache/src/md-cache.c:1669 #11 0x00007fe5679d0f76 in io_stats_fsync (frame=0x7fe576d70270, this=0x7fe568012c90, fd=0x7fe560002cec, flags=0, xdata=0x0) at ../../../../../xlators/debug/io-stats/src/io-stats.c:2207 #12 0x00007fe577eefb1d in default_fsync (frame=0x7fe576d70270, this=0x7fe568014160, fd=0x7fe560002cec, flags=0, xdata=0x0) at ../../../libglusterfs/src/defaults.c:1817 #13 0x00007fe577f3040a in syncop_fsync (subvol=subvol@entry=0x7fe568014160, fd=fd@entry=0x7fe560002cec, dataonly=dataonly@entry=0, xdata_in=xdata_in@entry=0x0, xdata_out=xdata_out@entry=0x0) at ../../../libglusterfs/src/syncop.c:2280 #14 0x00007fe56fe0257a in fuse_migrate_fd (this=this@entry=0x1661700, basefd=basefd@entry=0x7fe560002cec, old_subvol=old_subvol@entry=0x7fe568014160, new_subvol=new_subvol@entry=0x7fe560030be0) at ../../../../../xlators/mount/fuse/src/fuse-bridge.c:4423 #15 0x00007fe56fe0271d in fuse_handle_opened_fds (this=0x1661700, old_subvol=0x7fe568014160, new_subvol=0x7fe560030be0) at ../../../../../xlators/mount/fuse/src/fuse-bridge.c:4508 #16 0x00007fe56fe027c9 in fuse_graph_switch_task (data=<optimized out>) at ../../../../../xlators/mount/fuse/src/fuse-bridge.c:4559 ---Type <return> to continue, or q <return> to quit---q Quit (gdb) f 0 #0 0x00007fe56cb27503 in dht_fsync (frame=0x7fe576d701c4, this=<optimized out>, fd=0x7fe560002cec, datasync=0, xdata=0x0) at ../../../../../xlators/cluster/dht/src/dht-inode-read.c:818 818 STACK_WIND (frame, dht_fsync_cbk, subvol, subvol->fops->fsync, (gdb) l 813 local->call_cnt = 1; 814 local->rebalance.flags = datasync; 815 816 subvol = local->cached_subvol; 817 818 STACK_WIND (frame, dht_fsync_cbk, subvol, subvol->fops->fsync, 819 fd, datasync, xdata); 820 821 return 0; 822 (gdb) p subvol $1 = (xlator_t *) 0x0 Version-Release number of selected component (if applicable): master How reproducible: Fairly consistently in some setups Steps to Reproduce: 1. run ./tests/performance/open-behind.t in a loop. Sometimes Gluster mount crashes after we switch open-behind off. The crash is seen when when "num_graphs" is executed. 2. 3. Actual results: Expected results: Additional info: RCA: Its a race. If a graph switch happens when we have an fd opened under ".meta" subtree, crash is seen when migrating that fd. meta-fsync is not implemented and hence call is wound to real volume. Other translators have no idea of handling this virtual inode. Fix is to implement fsync in meta. As a complete fix, meta should implement all fops, so that it can check whether an inode belongs to it or not and wind the call down only if it does not belong to itself.