Bug 1491059
| Summary: | PID File handling: brick pid file leaves stale pid and brick fails to start when glusterd is started | ||
|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Ben Werthmann <ben> |
| Component: | glusterd | Assignee: | Mohit Agrawal <moagrawa> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | |
| Severity: | urgent | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.10 | CC: | amukherj, ben, bugs, joe, moagrawa |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | glusterfs-3.10.7 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-11-01 12:58:54 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1258561, 1464072 | ||
| Bug Blocks: | |||
|
Description
Ben Werthmann
2017-09-12 23:31:57 UTC
Looks like there may be a fix for this already: https://review.gluster.org/#/c/13580/ https://review.gluster.org/#/c/17601 May also lead to situations like this: $ gluster vol heal $vol statistics Gathering crawl statistics on volume $vol has been unsuccessful on bricks that are down. Please check if all brick processes are running. or gluster v heal testvol statistics Gathering crawl statistics on volume testvol has been unsuccessful: Staging failed on vm1. Error: Self-heal daemon is not running. Check self-heal daemon log file./ Also occurs with 3.10.5 from ppa:gluster/glusterfs-3.10 Upgrading to urgent as this affects stability of gluster in general. commit 220d406ad13d840e950eef001a2b36f87570058d
Author: Gaurav Kumar Garg <garg.gaurav52>
Date: Wed Mar 2 17:42:07 2016 +0530
glusterd: Gluster should keep PID file in correct location
Currently Gluster keeps process pid information of all the daemons
and brick processes in Gluster configuration file directory
(ie., /var/lib/glusterd/*).
These pid files should be seperate from configuration files.
Deletion of the configuration file directory might result into serious problems.
Also, /var/run/gluster is the default placeholder directory for pid files.
So, with this fix Gluster will keep all process pid information of all
processes in /var/run/gluster/* directory.
Change-Id: Idb09e3fccb6a7355fbac1df31082637c8d7ab5b4
BUG: 1258561
Signed-off-by: Gaurav Kumar Garg <ggarg>
Signed-off-by: Saravanakumar Arumugam <sarumuga>
Reviewed-on: https://review.gluster.org/13580
Tested-by: MOHIT AGRAWAL <moagrawa>
Smoke: Gluster Build System <jenkins.org>
CentOS-regression: Gluster Build System <jenkins.org>
Reviewed-by: Atin Mukherjee <amukherj>
The above commit takes care of this issue. Please note this fix is available in release-3.12 branch. Since this is a major change in the way pidfiles are placed, I don't have a plan to cherry pick this into release-3.10 branch.
Ben - Do you mind if I close this issue now? As I mentioned in the earlier comment, a stable release branch may not accept this change in the behaviour. So if you're fine with the workaround, you can choose to stick to release-3.10 branch otherwise please upgrade to release-3.12? I think there should be a minimal fix for 3.10. The minimal fix in this context is: - glusterd should start the brick when the process in the pid file is not a glusterfd process I will also run my tests with 3.12 and report results. Mohit - can you please backport https://review.gluster.org/13580 to release-3.10 branch? REVIEW: https://review.gluster.org/18484 (glusterd: Gluster should keep PID file in correct location) posted (#1) for review on release-3.10 by MOHIT AGRAWAL (moagrawa) COMMIT: https://review.gluster.org/18484 committed in release-3.10 by Shyamsundar Ranganathan (srangana) ------ commit 411a401f7e4f81f6a77eea1438a3a43c73e06104 Author: Gaurav Kumar Garg <garg.gaurav52> Date: Wed Mar 2 17:42:07 2016 +0530 glusterd: Gluster should keep PID file in correct location Currently Gluster keeps process pid information of all the daemons and brick processes in Gluster configuration file directory (ie., /var/lib/glusterd/*). These pid files should be seperate from configuration files. Deletion of the configuration file directory might result into serious problems. Also, /var/run/gluster is the default placeholder directory for pid files. So, with this fix Gluster will keep all process pid information of all processes in /var/run/gluster/* directory. > Change-Id: Idb09e3fccb6a7355fbac1df31082637c8d7ab5b4 > BUG: 1258561 > Signed-off-by: Gaurav Kumar Garg <ggarg> > Signed-off-by: Saravanakumar Arumugam <sarumuga> > Reviewed-on: https://review.gluster.org/13580 > Tested-by: MOHIT AGRAWAL <moagrawa> > Smoke: Gluster Build System <jenkins.org> > CentOS-regression: Gluster Build System <jenkins.org> > Reviewed-by: Atin Mukherjee <amukherj> > (Cherry pick from commit 220d406ad13d840e950eef001a2b36f87570058d) BUG: 1491059 Change-Id: Idb09e3fccb6a7355fbac1df31082637c8d7ab5b4 Signed-off-by: Mohit Agrawal <moagrawa> This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.7, please open a new bug report. glusterfs-3.10.7 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/announce/2017-November/000085.html [2] https://www.gluster.org/pipermail/gluster-users/ |