Bug 1211749
| Summary: | glusterd crashes when brick option validation fails | |||
|---|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Jeff Darcy <jdarcy> | |
| Component: | glusterd | Assignee: | Jeff Darcy <jdarcy> | |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | ||
| Severity: | urgent | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | mainline | CC: | amukherj, bugs | |
| Target Milestone: | --- | Keywords: | Triaged | |
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | glusterfs-3.8rc2 | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1219026 1220057 (view as bug list) | Environment: | ||
| Last Closed: | 2016-06-16 12:51:10 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1219026, 1220057 | |||
REVIEW: http://review.gluster.org/10238 (core/glusterd: avoid crash when option validation fails) posted (#1) for review on master by Jeff Darcy (jdarcy) REVIEW: http://review.gluster.org/10238 (core/glusterd: avoid crash when option validation fails) posted (#2) for review on master by Jeff Darcy (jdarcy) REVIEW: http://review.gluster.org/10238 (core/glusterd: avoid crash when option validation fails) posted (#3) for review on master by Jeff Darcy (jdarcy) REVIEW: http://review.gluster.org/10417 (core: use reference counting for mem_acct structures) posted (#1) for review on master by Jeff Darcy (jdarcy) REVIEW: http://review.gluster.org/10417 (core: use reference counting for mem_acct structures) posted (#2) for review on master by Jeff Darcy (jdarcy) REVIEW: http://review.gluster.org/10417 (core: use reference counting for mem_acct structures) posted (#3) for review on master by Jeff Darcy (jdarcy) COMMIT: http://review.gluster.org/10417 committed in master by Vijay Bellur (vbellur) ------ commit c085871e3919df2b309b76633e75d5449899437a Author: Jeff Darcy <jdarcy> Date: Tue Apr 28 04:40:00 2015 -0400 core: use reference counting for mem_acct structures When freeing memory, our memory-accounting code expects to be able to dereference from the (previously) allocated block to its owning translator. However, as we have already found once in option validation and twice in logging, that translator might itself have been freed and the dereference attempt causes on of our daemons to crash with SIGSEGV. This patch attempts to fix that as follows: * We no longer embed a struct mem_acct directly in a struct xlator, but instead allocate it separately. * Allocated memory blocks now contain a pointer to the mem_acct instead of the xlator. * The mem_acct structure contains a reference count, manipulated in both the normal and translator allocate/free code using atomic increments and decrements. * Because it's now a separate structure, we can defer freeing the mem_acct until its reference count reaches zero (either way). * Some unit tests were disabled, because they embedded their own copies of the implementation for what they were supposedly testing. Life's too short to spend time fixing tests that seem designed to impede progress by requiring a certain implementation as well as behavior. Change-Id: Id929b11387927136f78626901729296b6c0d0fd7 BUG: 1211749 Signed-off-by: Jeff Darcy <jdarcy> Reviewed-on: http://review.gluster.org/10417 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Krishnan Parthasarathi <kparthas> Reviewed-by: Niels de Vos <ndevos> Reviewed-by: Pranith Kumar Karampuri <pkarampu> REVIEW: http://review.gluster.org/10723 (core: use reference counting for mem_acct structures) posted (#1) for review on release-3.7 by Vijay Bellur (vbellur) *** Bug 1211473 has been marked as a duplicate of this bug. *** This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report. glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/ [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user |
While working on server-side AFR/NSR support, I added the following test to tests/basic/afr/read-subvol-data.t: TEST ! set_read_subvol $V0 no-such-xlator The result was a glusterd crash, like this: (gdb) bt #0 0x00007f0e5d5920b0 in pthread_spin_lock () from /lib64/libpthread.so.0 #1 0x00007f0e5e46dd69 in __gf_free (...) at mem-pool.c:303 #2 0x00007f0e59e251b4 in gd_sync_task_begin (...) at glusterd-syncop.c:1767 #3 0x00007f0e59e25260 in glusterd_op_begin_synctask (...) at glusterd-syncop.c:1787 #4 0x00007f0e59d770b2 in __glusterd_handle_set_volume (...) at glusterd-handler.c:1871 The __gf_free in question is for op_errstr, explaining the nature of the validation error. Here's a relevant comment from the patch I'll be posting momentarily, at the point where op_errstr is set, describing the real problem and a fix/workaround. * In the validation-error code path, the graph is freed * before op_errstr is. Therefore, if the memory block for * op_errstr still contains a reference to a translator within * that graph, we'll crash. Make sure the reference is to a * translator that's not going away instead.