Bug 1073023 - glusterfs mount crash after remove brick, detach peer and termination
Summary: glusterfs mount crash after remove brick, detach peer and termination
Keywords:
Status: CLOSED EOL
Alias: None
Product: GlusterFS
Classification: Community
Component: fuse
Version: 3.4.2
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
Assignee: bugs@gluster.org
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-03-05 15:54 UTC by Lukas Bezdicka
Modified: 2015-10-07 13:50 UTC (History)
3 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2015-10-07 13:49:43 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Lukas Bezdicka 2014-03-05 15:54:37 UTC
Description of problem:
[2014-03-05 13:36:18.850470] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host na1-dfs02.int.na.kacica.com
[2014-03-05 13:36:20.891348] W [defaults.c:1291:default_release] (-->/usr/lib64/glusterfs/3.4.2/xlator/protocol/client.so(client3_3_finodelk_cbk+0x157) [0x7fb3b62634a7] (-->/usr/lib64/glusterfs/3.4.2/xlator/protocol/client.so(client_local_wipe+0x28) [0x7fb3b6257df8] (-->/usr/lib64/libglusterfs.so.0(fd_unref+0x144) [0x31c2a391f4]))) 0-fuse: xlator does not implement release_cbk
pending frames:
frame : type(0) op(0)
frame : type(0) op(0)

patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 2014-03-05 13:36:21configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.4.2
[2014-03-05 13:36:21.857967] E [common-utils.c:211:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known)
[2014-03-05 13:36:21.857992] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host na1-dfs02.int.na.kacica.com
/lib64/libc.so.6[0x3297032920]
/lib64/libpthread.so.0(pthread_spin_lock+0x0)[0x329780c170]
/usr/lib64/libglusterfs.so.0(fd_ref+0x23)[0x31c2a38af3]
/usr/lib64/glusterfs/3.4.2/xlator/mount/fuse.so(+0xac05)[0x7fb3b90f1c05]
/usr/lib64/glusterfs/3.4.2/xlator/mount/fuse.so(+0x1f118)[0x7fb3b9106118]
/lib64/libpthread.so.0[0x3297807851]
/lib64/libc.so.6(clone+0x6d)[0x32970e890d]
---------

Version-Release number of selected component (if applicable):
3.4.2(with some patches from 3.4.3)

How reproducible:
Only on production

Steps to Reproduce:
1. Remove brick from cluster (gluster volume remove brick KAC start ... commit force)
2. detach the brick
3. kill the node associated with it and delete it's DNS record

Actual results:
Bunch of nodes complaining about:
[2014-03-05 13:36:18.850470] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host na1-dfs02.int.na.kacica.com

Expected results:
No logs about obsolete node and no crash

Additional info:

Comment 1 Lukas Bezdicka 2014-03-05 20:54:59 UTC
I think peer downtime and DNS resolving of dead peer aren't the root cause. Also the BT does not seem to help much :(

Comment 2 Niels de Vos 2015-05-17 21:57:55 UTC
GlusterFS 3.7.0 has been released (http://www.gluster.org/pipermail/gluster-users/2015-May/021901.html), and the Gluster project maintains N-2 supported releases. The last two releases before 3.7 are still maintained, at the moment these are 3.6 and 3.5.

This bug has been filed against the 3,4 release, and will not get fixed in a 3.4 version any more. Please verify if newer versions are affected with the reported problem. If that is the case, update the bug with a note, and update the version if you can. In case updating the version is not possible, leave a comment in this bug report with the version you tested, and set the "Need additional information the selected bugs from" below the comment box to "bugs".

If there is no response by the end of the month, this bug will get automatically closed.

Comment 3 Kaleb KEITHLEY 2015-10-07 13:49:43 UTC
GlusterFS 3.4.x has reached end-of-life.

If this bug still exists in a later release please reopen this and change the version or open a new bug.

Comment 4 Kaleb KEITHLEY 2015-10-07 13:50:53 UTC
GlusterFS 3.4.x has reached end-of-life.\                                                   \                                                                               If this bug still exists in a later release please reopen this and change the version or open a new bug.


Note You need to log in before you can comment on or make changes to this bug.