+++ This bug was initially created as a clone of Bug #1138503 +++ Description of problem: glusterd' runtime state is not directly observable. During debugging users/developers need to look at the logs, gdb the process and make (complex) inferences from them. This has been the practice due to lack of observability. This bug tracks development efforts towards make glusterd' runtime more observable. Version-Release number of selected component (if applicable): How reproducible: N/A Steps to Reproduce: 1.N/A 2. 3. Actual results: Expected results: Additional info: --- Additional comment from Anand Avati on 2014-09-04 23:14:11 EDT --- REVIEW: http://review.gluster.org/8584 (glusterd: Add last successful glusterd lock backtrace) posted (#3) for review on master by Krishnan Parthasarathi (kparthas) --- Additional comment from Anand Avati on 2014-09-05 03:17:06 EDT --- REVIEW: http://review.gluster.org/8584 (glusterd: Add last successful glusterd lock backtrace) posted (#4) for review on master by Krishnan Parthasarathi (kparthas) --- Additional comment from Anand Avati on 2014-09-05 07:28:13 EDT --- REVIEW: http://review.gluster.org/8584 (glusterd: Add last successful glusterd lock backtrace) posted (#5) for review on master by Krishnan Parthasarathi (kparthas) --- Additional comment from Anand Avati on 2014-09-05 07:28:21 EDT --- REVIEW: http://review.gluster.org/8622 (synctask: add backtrace per waiting task) posted (#1) for review on master by Krishnan Parthasarathi (kparthas) --- Additional comment from Anand Avati on 2014-09-11 05:46:04 EDT --- REVIEW: http://review.gluster.org/8584 (glusterd: Add last successful glusterd lock backtrace) posted (#6) for review on master by Krishnan Parthasarathi (kparthas) --- Additional comment from Anand Avati on 2014-09-11 05:46:12 EDT --- REVIEW: http://review.gluster.org/8622 (synctask: add backtrace per waiting task) posted (#2) for review on master by Krishnan Parthasarathi (kparthas) --- Additional comment from Anand Avati on 2014-09-12 01:18:25 EDT --- REVIEW: http://review.gluster.org/8584 (glusterd: Add last successful glusterd lock backtrace) posted (#7) for review on master by Krishnan Parthasarathi (kparthas) --- Additional comment from Anand Avati on 2014-09-12 01:18:28 EDT --- REVIEW: http://review.gluster.org/8622 (synctask: add backtrace per waiting task) posted (#3) for review on master by Krishnan Parthasarathi (kparthas) --- Additional comment from Anand Avati on 2014-09-22 07:09:02 EDT --- COMMIT: http://review.gluster.org/8584 committed in master by Krishnan Parthasarathi (kparthas) ------ commit 1b53756e50cb9ad5422292d7f4e1e6fe23357222 Author: Krishnan Parthasarathi <kparthas> Date: Wed Sep 3 11:03:06 2014 +0530 glusterd: Add last successful glusterd lock backtrace Also, moved the backtrace fetching logic to a separate function. Modified the backtrace fetching logic able to work under memory pressure conditions. Change-Id: Ie38bea425a085770f41831314aeda95595177ece BUG: 1138503 Signed-off-by: Krishnan Parthasarathi <kparthas> Reviewed-on: http://review.gluster.org/8584 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Jeff Darcy <jdarcy> Reviewed-by: Atin Mukherjee <amukherj>
REVIEW: http://review.gluster.org/8795 (synctask: add backtrace per waiting task) posted (#2) for review on release-3.6 by Krishnan Parthasarathi (kparthas)
REVIEW: http://review.gluster.org/8842 (glusterd: Improve debugging experience for glusterd locks) posted (#1) for review on release-3.6 by Krishnan Parthasarathi (kparthas)
A beta release for GlusterFS 3.6.0 has been released. Please verify if the release solves this bug report for you. In case the glusterfs-3.6.0beta2 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED. Packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update (possibly an "updates-testing" repository) infrastructure for your distribution. [1] http://supercolony.gluster.org/pipermail/gluster-users/2014-September/018883.html [2] http://supercolony.gluster.org/pipermail/gluster-users/
COMMIT: http://review.gluster.org/8795 committed in release-3.6 by Vijay Bellur (vbellur) ------ commit 04edacc5a0cc0a4c8c6bf5188a5ca4afd2fa4965 Author: Krishnan Parthasarathi <kparthas> Date: Fri Sep 5 14:55:01 2014 +0530 synctask: add backtrace per waiting task The backtrace is 'saved' in a per-task buffer. This would come handy while debugging code using synctasks. Change-Id: I732b275f6d15b31f31361f5ecf2ba47cacde9b54 BUG: 1145093 Signed-off-by: Krishnan Parthasarathi <kparthas> Reviewed-on: http://review.gluster.org/8795 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Niels de Vos <ndevos> Reviewed-by: Vijay Bellur <vbellur>
COMMIT: http://review.gluster.org/8842 committed in release-3.6 by Vijay Bellur (vbellur) ------ commit ce0044ac245002da461c40e5f2014d76e31afb2a Author: Krishnan Parthasarathi <kparthas> Date: Wed Sep 3 10:29:07 2014 +0530 glusterd: Improve debugging experience for glusterd locks Today, when glusterd's internal locking mechanism fails with invalid type or when another competing lock is being held, the log message doesn't provide enough information directly as to which command saw this (first). Following is a snippet of how a failure would look in the log file. This would greatly assist in debugging. [2014-09-03 04:57:58.549418] E [glusterd-locks.c:520:glusterd_mgmt_v3_lock] (-->/usr/local/lib/glusterfs/3.7dev/xlator/mgmt/glusterd.so(__glusterd_handle_create_volume+0x801) [0x7f30b071e651] (-->/usr/local/lib/glusterfs/3.7dev/xlator/mgmt/glusterd.so(glusterd_op_begin_synctask+0x2c) [0x7f30b072e19c] (-->/usr/local/lib/glusterfs/3.7dev/xlator/mgmt/glusterd.so(gd_sync_task_begin+0x55d) [0x7f30b072de6d]))) 0-management: Invalid entity. Cannot perform locking operation on vol types Change-Id: I0595f49d60e620e8b065f3506bdb147ccee383a7 BUG: 1145093 Signed-off-by: Krishnan Parthasarathi <kparthas> Reviewed-on: http://review.gluster.org/8842 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.6.1, please reopen this bug report. glusterfs-3.6.1 has been announced [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019410.html [2] http://supercolony.gluster.org/mailman/listinfo/gluster-users