Bug 1153610
| Summary: | libgfapi crashes in glfs_fini for RDMA type volumes | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Anoop C S <anoopcs> | ||||
| Component: | libgfapi | Assignee: | bugs <bugs> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | storage-qa-internal <storage-qa-internal> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | mainline | CC: | anoopcs, bugs, jthottan, rgowdapp, rkavunga, rtalur, skoduri | ||||
| Target Milestone: | --- | Keywords: | Triaged | ||||
| Target Release: | --- | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | glusterfs-3.7.0 | Doc Type: | Bug Fix | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | |||||||
| : | 1171662 (view as bug list) | Environment: | |||||
| Last Closed: | 2015-05-14 17:28:00 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1171662 | ||||||
| Attachments: |
|
||||||
Root cause of the crash was identified as follows: When main() returns from a C program, global prioritized destructor functions are called in priority order before the process terminates. According to rdmacm standard library, rdma_cma_fini() is defined as a destructor function. The same function is also invoked as part of rdma_disconnect() initiated through xlator_notify() inside glfs_fini(). Due to the increased delay in rdma_disconnect() and associated cleanup, there will be a race between the main thread and rdma thread to execute rdma_cm_fini(), which will result in a segmentation fault. REVIEW: http://review.gluster.org/9060 (libgfapi: Wait for GF_EVENT_CHILD_DOWN in glfs_fini()) posted (#1) for review on master by Anoop C S (achiraya) REVIEW: http://review.gluster.org/9060 (libgfapi: Wait for GF_EVENT_CHILD_DOWN in glfs_fini()) posted (#2) for review on master by Anoop C S (achiraya) REVIEW: http://review.gluster.org/9060 (libgfapi: Wait for GF_EVENT_CHILD_DOWN in glfs_fini()) posted (#3) for review on master by Anoop C S (achiraya) REVIEW: http://review.gluster.org/9060 (libgfapi: Wait for GF_EVENT_CHILD_DOWN in glfs_fini()) posted (#4) for review on master by Anoop C S (achiraya) REVIEW: http://review.gluster.org/9060 (libgfapi: Wait for GF_EVENT_CHILD_DOWN in glfs_fini()) posted (#5) for review on master by Anoop C S (achiraya) REVIEW: http://review.gluster.org/9060 (libgfapi: Wait for GF_EVENT_CHILD_DOWN in glfs_fini()) posted (#6) for review on master by Anoop C S (achiraya) REVIEW: http://review.gluster.org/9060 (libgfapi: Wait for GF_EVENT_CHILD_DOWN in glfs_fini()) posted (#7) for review on master by Poornima G (pgurusid) REVIEW: http://review.gluster.org/9060 (libgfapi: Wait for GF_EVENT_CHILD_DOWN in glfs_fini()) posted (#8) for review on master by Anoop C S (achiraya) REVIEW: http://review.gluster.org/9060 (libgfapi: Wait for GF_EVENT_CHILD_DOWN in glfs_fini()) posted (#9) for review on master by Anoop C S (achiraya) REVIEW: http://review.gluster.org/9060 (libgfapi: Wait for GF_EVENT_CHILD_DOWN in glfs_fini()) posted (#10) for review on master by Anoop C S (achiraya) REVIEW: http://review.gluster.org/9060 (libgfapi: Wait for GF_EVENT_CHILD_DOWN in glfs_fini()) posted (#11) for review on master by Anoop C S (achiraya) REVIEW: http://review.gluster.org/9060 (libgfapi: Wait for GF_EVENT_CHILD_DOWN in glfs_fini()) posted (#12) for review on master by Anoop C S (achiraya) REVIEW: http://review.gluster.org/9060 (libgfapi: Wait for GF_EVENT_CHILD_DOWN in glfs_fini()) posted (#13) for review on master by Anoop C S (achiraya) REVIEW: http://review.gluster.org/9060 (libgfapi: Wait for GF_EVENT_CHILD_DOWN in glfs_fini()) posted (#14) for review on master by Anoop C S (achiraya) REVIEW: http://review.gluster.org/9060 (libgfapi: Wait for GF_EVENT_CHILD_DOWN in glfs_fini()) posted (#15) for review on master by Anoop C S (achiraya) REVIEW: http://review.gluster.org/9060 (libgfapi: Wait for GF_EVENT_CHILD_DOWN in glfs_fini()) posted (#16) for review on master by Anoop C S (achiraya) REVIEW: http://review.gluster.org/9060 (libgfapi: Wait for GF_EVENT_CHILD_DOWN in glfs_fini()) posted (#17) for review on master by Anoop C S (achiraya) REVIEW: http://review.gluster.org/9060 (libgfapi: Wait for GF_EVENT_CHILD_DOWN in glfs_fini()) posted (#18) for review on master by Anoop C S (achiraya) COMMIT: http://review.gluster.org/9060 committed in master by Niels de Vos (ndevos) ------ commit cd6ffa93dc2a3cb1fcc5438086aebc54f368c2e9 Author: Anoop C S <achiraya> Date: Wed Oct 29 09:12:46 2014 -0400 libgfapi: Wait for GF_EVENT_CHILD_DOWN in glfs_fini() Whenever glfs_fini() is being called, currently no check is made inside the function to determine whether the child is already down or not. This patch will wait for GF_EVENT_CHILD_DOWN for the active subvol and then exits. TBD: Apart from the active subvol, wait for other CHILD_DOWN events generated through operations like volume set in future. Change-Id: I81c64ac07b463bfed48bf306f9e8f46ba0f0a76f BUG: 1153610 Signed-off-by: Anoop C S <achiraya> Reviewed-on: http://review.gluster.org/9060 Reviewed-by: Shyamsundar Ranganathan <srangana> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Raghavendra G <rgowdapp> Reviewed-by: Niels de Vos <ndevos> This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report. glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report. glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report. glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report. glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user |
Created attachment 947560 [details] libgfapi test program Description of problem: C program which uses libgfapi for RDMA volume crashes in glfs_fini(). Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Create a 1-brick RDMA volume. 2. Compile the attached C program that uses libgfapi gcc -pthreads -g -O0 -Wall --pedantic -o gfapi_perf_test -I /usr/include/glusterfs/api gfapi_perf_test.c -lgfapi -lrt 3. Do the following exports export LD_LIBRARY_PATH=/usr/local/lib export GFAPI_HOSTNAME=<server ip> export GFAPI_TRANSPORT=rdma export GFAPI_VOLNAME=<volume name> 4.Run the compiled output as GFAPI_FILES=1 GFAPI_RECSZ=1024 GFAPI_FSZ=1048576 ./gfapi_perf_test Actual results: Segmentation fault (core dumped) Expected results: Program executes successfully. Additional info: