Bug 1073023

Summary:	glusterfs mount crash after remove brick, detach peer and termination
Product:	[Community] GlusterFS	Reporter:	Lukas Bezdicka <social>
Component:	fuse	Assignee:	bugs <bugs>
Status:	CLOSED EOL	QA Contact:
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	3.4.2	CC:	bugs, gluster-bugs, johan.huysmans
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-10-07 13:49:43 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Lukas Bezdicka 2014-03-05 15:54:37 UTC

Description of problem:
[2014-03-05 13:36:18.850470] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host na1-dfs02.int.na.kacica.com
[2014-03-05 13:36:20.891348] W [defaults.c:1291:default_release] (-->/usr/lib64/glusterfs/3.4.2/xlator/protocol/client.so(client3_3_finodelk_cbk+0x157) [0x7fb3b62634a7] (-->/usr/lib64/glusterfs/3.4.2/xlator/protocol/client.so(client_local_wipe+0x28) [0x7fb3b6257df8] (-->/usr/lib64/libglusterfs.so.0(fd_unref+0x144) [0x31c2a391f4]))) 0-fuse: xlator does not implement release_cbk
pending frames:
frame : type(0) op(0)
frame : type(0) op(0)

patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 2014-03-05 13:36:21configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.4.2
[2014-03-05 13:36:21.857967] E [common-utils.c:211:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known)
[2014-03-05 13:36:21.857992] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host na1-dfs02.int.na.kacica.com
/lib64/libc.so.6[0x3297032920]
/lib64/libpthread.so.0(pthread_spin_lock+0x0)[0x329780c170]
/usr/lib64/libglusterfs.so.0(fd_ref+0x23)[0x31c2a38af3]
/usr/lib64/glusterfs/3.4.2/xlator/mount/fuse.so(+0xac05)[0x7fb3b90f1c05]
/usr/lib64/glusterfs/3.4.2/xlator/mount/fuse.so(+0x1f118)[0x7fb3b9106118]
/lib64/libpthread.so.0[0x3297807851]
/lib64/libc.so.6(clone+0x6d)[0x32970e890d]
---------

Version-Release number of selected component (if applicable):
3.4.2(with some patches from 3.4.3)

How reproducible:
Only on production

Steps to Reproduce:
1. Remove brick from cluster (gluster volume remove brick KAC start ... commit force)
2. detach the brick
3. kill the node associated with it and delete it's DNS record

Actual results:
Bunch of nodes complaining about:
[2014-03-05 13:36:18.850470] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host na1-dfs02.int.na.kacica.com

Expected results:
No logs about obsolete node and no crash

Additional info:

Comment 1 Lukas Bezdicka 2014-03-05 20:54:59 UTC

I think peer downtime and DNS resolving of dead peer aren't the root cause. Also the BT does not seem to help much :(

Comment 2 Niels de Vos 2015-05-17 21:57:55 UTC

GlusterFS 3.7.0 has been released (http://www.gluster.org/pipermail/gluster-users/2015-May/021901.html), and the Gluster project maintains N-2 supported releases. The last two releases before 3.7 are still maintained, at the moment these are 3.6 and 3.5.

This bug has been filed against the 3,4 release, and will not get fixed in a 3.4 version any more. Please verify if newer versions are affected with the reported problem. If that is the case, update the bug with a note, and update the version if you can. In case updating the version is not possible, leave a comment in this bug report with the version you tested, and set the "Need additional information the selected bugs from" below the comment box to "bugs".

If there is no response by the end of the month, this bug will get automatically closed.

Comment 3 Kaleb KEITHLEY 2015-10-07 13:49:43 UTC

GlusterFS 3.4.x has reached end-of-life.

If this bug still exists in a later release please reopen this and change the version or open a new bug.

Comment 4 Kaleb KEITHLEY 2015-10-07 13:50:53 UTC

GlusterFS 3.4.x has reached end-of-life.\                                                   \                                                                               If this bug still exists in a later release please reopen this and change the version or open a new bug.