Bug 1577574
| Summary: | brick crash seen while creating and deleting two volumes in loop | ||
|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Mohit Agrawal <moagrawa> |
| Component: | rpc | Assignee: | Mohit Agrawal <moagrawa> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | mainline | CC: | bmekala, bugs, moagrawa, rhinduja, rhs-bugs, sankarshan, vbellur |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | glusterfs-5.0 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 1575840 | Environment: | |
| Last Closed: | 2018-10-23 15:08:41 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1575840 | ||
| Bug Blocks: | |||
|
Description
Mohit Agrawal
2018-05-13 06:10:19 UTC
RCA: The brick process was getting crash because in there was no ctx available at inode and inode was already free in xlator_mem_cleanup, It call's by at the time of destroying transport in free_state before call fd_unref.
To resolve the same move the code in free_state destroying transport after free all resources.
>>>>>>>>>>>>>>>>>>>>
#0 0x00007fa5a3c08dc7 in __inode_get_xl_index (xlator=0x7fa590029ff0, inode=0x7fa53c002260) at inode.c:455
#1 __inode_unref (inode=inode@entry=0x7fa53c002260) at inode.c:489
#2 0x00007fa5a3c09621 in inode_unref (inode=0x7fa53c002260) at inode.c:559
#3 0x00007fa5a3c1f502 in fd_destroy (bound=_gf_true, fd=0x7fa538004d20) at fd.c:532
#4 fd_unref (fd=0x7fa538004d20) at fd.c:569
#5 0x00007fa58ed03169 in free_state (state=0x7fa5380013b0) at server-helpers.c:185
#6 0x00007fa58ecfe64a in server_submit_reply (frame=frame@entry=0x7fa538002910, req=0x7fa50c29ade0,
arg=arg@entry=0x7fa58e8ec910, payload=payload@entry=0x0, payloadcount=payloadcount@entry=0, iobref=0x7fa538004e50,
iobref@entry=0x0, xdrproc=0x7fa5a37a36b0 <xdr_gfs3_opendir_rsp>) at server.c:212
#7 0x00007fa58ed12de4 in server_opendir_cbk (frame=frame@entry=0x7fa538002910, cookie=<optimized out>, this=0x7fa590029ff0,
op_ret=op_ret@entry=0, op_errno=op_errno@entry=0, fd=fd@entry=0x7fa538004d20, xdata=xdata@entry=0x0)
at server-rpc-fops.c:710
#8 0x00007fa58f173111 in io_stats_opendir_cbk (frame=0x7fa538006f10, cookie=<optimized out>, this=<optimized out>,
op_ret=0, op_errno=0, fd=0x7fa538004d20, xdata=0x0) at io-stats.c:2315
#9 0x00007fa58f5b419d in index_opendir (frame=frame@entry=0x7fa538002480, this=this@entry=0x7fa5640138a0,
loc=loc@entry=0x7fa5380013c8, fd=fd@entry=0x7fa538004d20, xdata=xdata@entry=0x0) at index.c:2113
#10 0x00007fa5a3c7a27b in default_opendir (frame=0x7fa538002480, this=<optimized out>, loc=0x7fa5380013c8,
fd=0x7fa538004d20, xdata=0x0) at defaults.c:2956
#11 0x00007fa58f1621bb in io_stats_opendir (frame=frame@entry=0x7fa538006f10, this=this@entry=0x7fa564016110,
loc=loc@entry=0x7fa5380013c8, fd=fd@entry=0x7fa538004d20, xdata=xdata@entry=0x0) at io-stats.c:3311
#12 0x00007fa5a3c7a27b in default_opendir (frame=0x7fa538006f10, this=<optimized out>, loc=0x7fa5380013c8,
fd=0x7fa538004d20, xdata=0x0) at defaults.c:2956
#13 0x00007fa58ed1b082 in server_opendir_resume (frame=0x7fa538002910, bound_xl=0x7fa564017720) at server-rpc-fops.c:2672
#14 0x00007fa58ed01d29 in server_resolve_done (frame=0x7fa538002910) at server-resolve.c:587
#15 0x00007fa58ed01dcd in server_resolve_all (frame=frame@entry=0x7fa538002910) at server-resolve.c:622
#16 0x00007fa58ed027e5 in server_resolve (frame=0x7fa538002910) at server-resolve.c:571
#17 0x00007fa58ed01e0e in server_resolve_all (frame=frame@entry=0x7fa538002910) at server-resolve.c:618
#18 0x00007fa58ed0257b in server_resolve_inode (frame=frame@entry=0x7fa538002910) at server-resolve.c:425
#19 0x00007fa58ed02810 in server_resolve (frame=0x7fa538002910) at server-resolve.c:559
#20 0x00007fa58ed01dee in server_resolve_all (frame=frame@entry=0x7fa538002910) at server-resolve.c:611
#21 0x00007fa58ed028a4 in resolve_and_resume (frame=frame@entry=0x7fa538002910,
fn=fn@entry=0x7fa58ed1ae90 <server_opendir_resume>) at server-resolve.c:642
---Type <return> to continue, or q <return> to quit---
#22 0x00007fa58ed1c851 in server3_3_opendir (req=<optimized out>) at server-rpc-fops.c:4938
#23 0x00007fa5a39ba66e in rpcsvc_request_handler (arg=0x7fa59003f9b0) at rpcsvc.c:1915
#24 0x00007fa5a2a57dd5 in start_thread () from /lib64/libpthread.so.0
#25 0x00007fa5a2320b3d in clone () from /lib64/libc.so.6
Regards
Mohit Agrawal
REVIEW: https://review.gluster.org/20014 (glusterfs: Resolve brick crashes at the time of inode_unref) posted (#1) for review on master by MOHIT AGRAWAL COMMIT: https://review.gluster.org/20014 committed in master by "Raghavendra G" <rgowdapp> with a commit message- glusterfs: Resolve brick crashes at the time of inode_unref Problem: Sometimes brick process is getting crash at the time of calling inode_unref in fd_destroy Solution: Brick process is getting crash because inode is already free by xlator_mem_cleanup call by server_rpc_notify.To resolve the same move code specific to call transport_unref in last in free_state. BUG: 1577574 Change-Id: Ia517c230d68af4e929b6b753e4c374a26c39dc1a fixes: bz#1577574 Signed-off-by: Mohit Agrawal <moagrawa> This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-5.0, please open a new bug report. glusterfs-5.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2018-October/000115.html [2] https://www.gluster.org/pipermail/gluster-users/ |