Bug 1459467

Summary:	Brick Multiplexing: brick process crashed while stopping volumes
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Nag Pavan Chilakam <nchilaka>
Component:	core	Assignee:	Vijay Bellur <vbellur>
Status:	CLOSED WORKSFORME	QA Contact:	Nag Pavan Chilakam <nchilaka>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.3	CC:	amukherj, nchilaka, rhs-bugs, storage-qa-internal
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-06-09 07:33:49 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Nag Pavan Chilakam 2017-06-07 08:57:02 UTC

Description of problem:
=====================
I had about 46 volume in my brick mux setup
While stopping these volumes, the glusterfsd process crashed on one of the nodes
[root@dhcp35-45 ~]# gluster v list
cross3-1
cross3-10
cross3-11
cross3-12
cross3-13
cross3-14
cross3-15
cross3-16
cross3-17
cross3-18
cross3-19
cross3-2
cross3-20
cross3-21
cross3-22
cross3-23
cross3-24
cross3-25
cross3-27
cross3-29
cross3-3
cross3-30
cross3-31
cross3-32
cross3-33
cross3-34
cross3-35
cross3-36
cross3-37
cross3-38
cross3-39
cross3-4
cross3-40
cross3-41
cross3-43
cross3-45
cross3-46
cross3-47
cross3-48
cross3-49
cross3-5
cross3-50
cross3-6
cross3-7
cross3-8
cross3-9
[root@dhcp35-45 ~]# rpm -qa|grep gluster
glusterfs-server-3.8.4-25.el7rhgs.x86_64
glusterfs-3.8.4-25.el7rhgs.x86_64
glusterfs-events-3.8.4-25.el7rhgs.x86_64
glusterfs-fuse-3.8.4-25.el7rhgs.x86_64
glusterfs-rdma-3.8.4-25.el7rhgs.x86_64
gluster-nagios-addons-0.2.8-1.el7rhgs.x86_64
glusterfs-api-3.8.4-25.el7rhgs.x86_64
glusterfs-devel-3.8.4-25.el7rhgs.x86_64
glusterfs-geo-replication-3.8.4-25.el7rhgs.x86_64
gluster-nagios-common-0.2.4-1.el7rhgs.noarch
glusterfs-client-xlators-3.8.4-25.el7rhgs.x86_64
glusterfs-api-devel-3.8.4-25.el7rhgs.x86_64
glusterfs-cli-3.8.4-25.el7rhgs.x86_64
glusterfs-debuginfo-3.8.4-25.el7rhgs.x86_64
glusterfs-libs-3.8.4-25.el7rhgs.x86_64
python-gluster-3.8.4-25.el7rhgs.noarch
[root@dhcp35-45 ~]# for i in $(gluster v list);do gluster v stop $i;done
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-1: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-10: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-11: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-12: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-13: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-14: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-15: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-16: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-17: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-18: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-19: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-2: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-20: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-21: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
y
Error : Request timed out
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) volume stop: cross3-23: failed: Commit failed on 10.70.35.122. Error: error
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-24: failed: Commit failed on 10.70.35.122. Error: error
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) yy
Invalid input, please enter y/n
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-27: failed: Commit failed on 10.70.35.122. Error: error
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) n
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) ^C
[root@dhcp35-45 ~]# gluster v status cross3-23:
Volume cross3-23: does not exist
[root@dhcp35-45 ~]# gluster v status cross3-23
Volume cross3-23 is not started
[root@dhcp35-45 ~]# gluster v status
Volume cross3-1 is not started
 
Volume cross3-10 is not started
 
Volume cross3-11 is not started
 
Volume cross3-12 is not started
 
Volume cross3-13 is not started
 
Volume cross3-14 is not started
 
Volume cross3-15 is not started
 
Volume cross3-16 is not started
 
Volume cross3-17 is not started
 
Volume cross3-18 is not started
 
Volume cross3-19 is not started
 
Volume cross3-2 is not started
 
Volume cross3-20 is not started
 
Volume cross3-21 is not started
 
Volume cross3-22 is not started
 
Volume cross3-23 is not started
 
Volume cross3-24 is not started
 
Status of volume: cross3-25
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick25/cross3-25    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick25/cross3-25   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick25/cross3-25   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-25
------------------------------------------------------------------------------
There are no active volume tasks
 
Volume cross3-27 is not started
 
Status of volume: cross3-29
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick29/cross3-29    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick29/cross3-29   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick29/cross3-29   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-29
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-3
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick3/cross3-3      49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick3/cross3-3     49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick3/cross3-3     N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-3
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-30
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick30/cross3-30    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick30/cross3-30   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick30/cross3-30   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-30
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-31
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick31/cross3-31    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick31/cross3-31   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick31/cross3-31   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-31
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-32
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick32/cross3-32    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick32/cross3-32   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick32/cross3-32   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-32
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-33
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick33/cross3-33    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick33/cross3-33   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick33/cross3-33   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-33
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-34
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick34/cross3-34    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick34/cross3-34   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick34/cross3-34   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-34
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-35
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick35/cross3-35    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick35/cross3-35   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick35/cross3-35   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
 
Task Status of Volume cross3-35
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-36
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick36/cross3-36    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick36/cross3-36   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick36/cross3-36   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
 
Task Status of Volume cross3-36
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-37
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick37/cross3-37    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick37/cross3-37   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick37/cross3-37   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
 
Task Status of Volume cross3-37
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-38
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick38/cross3-38    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick38/cross3-38   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick38/cross3-38   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
 
Task Status of Volume cross3-38
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-39
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick39/cross3-39    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick39/cross3-39   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick39/cross3-39   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-39
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-4
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick4/cross3-4      49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick4/cross3-4     49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick4/cross3-4     N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-4
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-40
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick40/cross3-40    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick40/cross3-40   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick40/cross3-40   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-40
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-41
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick41/cross3-41    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick41/cross3-41   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick41/cross3-41   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-41
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-43
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick43/cross3-43    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick43/cross3-43   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick43/cross3-43   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-43
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-45
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick45/cross3-45    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick45/cross3-45   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick45/cross3-45   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-45
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-46
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick46/cross3-46    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick46/cross3-46   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick46/cross3-46   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-46
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-47
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick47/cross3-47    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick47/cross3-47   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick47/cross3-47   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-47
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-48
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick48/cross3-48    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick48/cross3-48   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick48/cross3-48   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-48
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-49
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick49/cross3-49    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick49/cross3-49   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick49/cross3-49   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
 
Task Status of Volume cross3-49
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-5
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick5/cross3-5      49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick5/cross3-5     49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick5/cross3-5     N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-5
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-50
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick50/cross3-50    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick50/cross3-50   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick50/cross3-50   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-50
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-6
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick6/cross3-6      49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick6/cross3-6     49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick6/cross3-6     N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-6
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-7
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick7/cross3-7      49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick7/cross3-7     49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick7/cross3-7     N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-7
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-8
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick8/cross3-8      49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick8/cross3-8     49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick8/cross3-8     N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-8
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-9
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick9/cross3-9      49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick9/cross3-9     49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick9/cross3-9     N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-9
------------------------------------------------------------------------------
There are no active volume tasks
 



############## on the node where glusterfsd crashed#########
root@dhcp35-122 ~]# gluster v info cross3-23
 
Volume Name: cross3-23
Type: Replicate
Volume ID: cda18b59-c670-43da-a5a5-7eb15fe4d1c0
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.70.35.45:/rhs/brick23/cross3-23
Brick2: 10.70.35.130:/rhs/brick23/cross3-23
Brick3: 10.70.35.122:/rhs/brick23/cross3-23
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
cluster.brick-multiplex: on
[root@dhcp35-122 ~]# gluster v status cross3-23:
Volume cross3-23: does not exist
[root@dhcp35-122 ~]# gluster v status cross3-23
Staging failed on 10.70.35.23. Error: Volume cross3-23 is not started
Staging failed on 10.70.35.130. Error: Volume cross3-23 is not started
Staging failed on dhcp35-45.lab.eng.blr.redhat.com. Error: Volume cross3-23 is not started
[root@dhcp35-122 ~]# 


backtrace

New LWP 22060]
Reading symbols from /usr/sbin/glusterfsd...Reading symbols from /usr/lib/debug/usr/sbin/glusterfsd.debug...done.
done.
Missing separate debuginfo for 
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/50/d1ccf000162361f74a9d7c2ecea856f7881f07
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/glusterfsd -s 10.70.35.122 --volfile-id cross3-1.10.70.35.122.rhs-bri'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f3315265d37 in glusterfs_handle_terminate (req=0x7f3302f3dd90) at glusterfsd-mgmt.c:254
254	                *trav_p = (*trav_p)->next;
Missing separate debuginfos, use: debuginfo-install glibc-2.17-157.el7_3.1.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-27.el7_3.x86_64 libacl-2.2.51-12.el7.x86_64 libaio-0.3.109-13.el7.x86_64 libattr-2.4.46-12.el7.x86_64 libcom_err-1.42.9-9.el7.x86_64 libgcc-4.8.5-11.el7.x86_64 libselinux-2.5-6.el7.x86_64 libuuid-2.23.2-33.el7.x86_64 openssl-libs-1.0.1e-60.el7_3.1.x86_64 pcre-8.32-15.el7_2.1.x86_64 sqlite-3.7.17-8.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0  0x00007f3315265d37 in glusterfs_handle_terminate (req=0x7f3302f3dd90) at glusterfsd-mgmt.c:254
#1  0x00007f3314d9f1f2 in synctask_wrap (old_task=<optimized out>) at syncop.c:375
#2  0x00007f331345ccf0 in ?? () from /lib64/libc.so.6
#3  0x0000000000000000 in ?? ()
(gdb) 
#0  0x00007f3315265d37 in glusterfs_handle_terminate (req=0x7f3302f3dd90) at glusterfsd-mgmt.c:254
#1  0x00007f3314d9f1f2 in synctask_wrap (old_task=<optimized out>) at syncop.c:375
#2  0x00007f331345ccf0 in ?? () from /lib64/libc.so.6
#3  0x0000000000000000 in ?? ()
(gdb) t a a bt

Thread 584 (Thread 0x7f32d81f4700 (LWP 22060)):
#0  0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f3306197ccb in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:369
#2  0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f331350d73d in clone () from /lib64/libc.so.6

Thread 583 (Thread 0x7f31dd982700 (LWP 22348)):
#0  0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f33065d9cf3 in changelog_ev_connector (data=0x7f32ba8b4228) at changelog-ev-handle.c:202
#2  0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f331350d73d in clone () from /lib64/libc.so.6

Thread 582 (Thread 0x7f325c1f2700 (LWP 22347)):
#0  0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f3306197ccb in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:369
#2  0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f331350d73d in clone () from /lib64/libc.so.6

Thread 581 (Thread 0x7f32edaf9700 (LWP 21923)):
#0  0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f3306197ccb in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:369
#2  0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f331350d73d in clone () from /lib64/libc.so.6

Thread 580 (Thread 0x7f32806f9700 (LWP 22218)):
#0  0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f3306197ccb in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:369
#2  0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f331350d73d in clone () from /lib64/libc.so.6

Thread 579 (Thread 0x7f321c6f1700 (LWP 22067)):
#0  0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f330771caab in posix_fsyncer_pick (this=this@entry=0x7f32e0964ec0, head=head@entry=0x7f321c6f0e80) at posix-helpers.c:1978
#2  0x00007f330771cd35 in posix_fsyncer (d=0x7f32e0964ec0) at posix-helpers.c:2077
#3  0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f331350d73d in clone () from /lib64/libc.so.6

Thread 578 (Thread 0x7f321f6f7700 (LWP 22061)):
#0  0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f33065d9cf3 in changelog_ev_connector (data=0x7f32e09c5288) at changelog-ev-handle.c:202
#2  0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f331350d73d in clone () from /lib64/libc.so.6

Thread 577 (Thread 0x7f32ec5f2700 (LWP 21949)):
#0  0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f3306197ccb in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:369
#2  0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f331350d73d in clone () from /lib64/libc.so.6

Thread 576 (Thread 0x7f31a8397700 (LWP 22610)):
#0  0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f3306197ccb in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:369
---Type <return> to continue, or q <return> to quit--- 


version:
3.8.4-25

Comment 2 Nag Pavan Chilakam 2017-06-07 08:58:56 UTC

brick log where it crashed


[2017-06-07 08:43:42.321599] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-130.lab.eng.blr.redhat.com-14070-2017/06/07-08:43:32:148610-cross3-50-client-2-0-0
[2017-06-07 08:43:42.321763] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-130.lab.eng.blr.redhat.com-14070-2017/06/07-08:43:32:148610-cross3-50-client-2-0-0
[2017-06-07 08:43:47.050575] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick12/cross3-12
[2017-06-07 08:43:47.050643] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:940
[2017-06-07 08:43:47.051202] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-28703-2017/05/17-14:41:54:841551-cross3-12-client-2-0-0
[2017-06-07 08:43:47.051632] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-28703-2017/05/17-14:41:54:841551-cross3-12-client-2-0-0
[2017-06-07 08:43:47.051963] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick12/cross3-12 - not found
[2017-06-07 08:43:50.715257] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick13/cross3-13
[2017-06-07 08:43:50.715318] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:933
[2017-06-07 08:43:50.715787] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-28758-2017/05/17-14:41:54:920251-cross3-13-client-2-0-0
[2017-06-07 08:43:50.716787] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-28758-2017/05/17-14:41:54:920251-cross3-13-client-2-0-0
[2017-06-07 08:43:50.717204] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick13/cross3-13 - not found
[2017-06-07 08:43:54.582866] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick14/cross3-14
[2017-06-07 08:43:54.582900] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:926
[2017-06-07 08:43:54.583330] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-28813-2017/05/17-14:41:55:558-cross3-14-client-2-0-0
[2017-06-07 08:43:54.588229] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-28813-2017/05/17-14:41:55:558-cross3-14-client-2-0-0
[2017-06-07 08:43:54.589010] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick14/cross3-14
[2017-06-07 08:43:59.304553] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick15/cross3-15 - not found
[2017-06-07 08:43:59.305369] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick15/cross3-15 - not found
[2017-06-07 08:44:02.097158] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick16/cross3-16
[2017-06-07 08:44:02.097214] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:912
[2017-06-07 08:44:02.097379] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-28923-2017/05/17-14:41:55:152920-cross3-16-client-2-0-0
[2017-06-07 08:44:02.097925] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-28923-2017/05/17-14:41:55:152920-cross3-16-client-2-0-0
[2017-06-07 08:44:02.098338] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick16/cross3-16 - not found
[2017-06-07 08:44:05.831594] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick17/cross3-17
[2017-06-07 08:44:05.831639] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:905
[2017-06-07 08:44:05.831745] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-28978-2017/05/17-14:41:55:227086-cross3-17-client-2-0-0
[2017-06-07 08:44:05.832220] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-28978-2017/05/17-14:41:55:227086-cross3-17-client-2-0-0
[2017-06-07 08:44:05.832614] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick17/cross3-17 - not found
[2017-06-07 08:44:09.639722] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick18/cross3-18
[2017-06-07 08:44:09.639765] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:898
[2017-06-07 08:44:09.639903] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-29033-2017/05/17-14:41:55:295359-cross3-18-client-2-0-0
[2017-06-07 08:44:09.640336] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-29033-2017/05/17-14:41:55:295359-cross3-18-client-2-0-0
[2017-06-07 08:44:09.640678] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick18/cross3-18 - not found
[2017-06-07 08:44:13.453454] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick19/cross3-19
[2017-06-07 08:44:13.453504] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:891
[2017-06-07 08:44:13.453598] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.37.43:1017
[2017-06-07 08:44:13.453922] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-29088-2017/05/17-14:41:55:371105-cross3-19-client-2-0-0
[2017-06-07 08:44:13.454613] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-29088-2017/05/17-14:41:55:371105-cross3-19-client-2-0-0
[2017-06-07 08:44:13.454787] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp37-43.lab.eng.blr.redhat.com-11004-2017/05/20-08:02:20:362257-cross3-19-client-2-0-0
[2017-06-07 08:44:13.455144] I [MSGID: 115013] [server-helpers.c:296:do_fd_cleanup] 0-cross3-1-server: fd cleanup on <gfid:acff7539-45c1-4b43-9478-c93390cf2f1f>
[2017-06-07 08:44:13.455464] I [MSGID: 115013] [server-helpers.c:296:do_fd_cleanup] 0-cross3-1-server: fd cleanup on <gfid:c4b8ef94-52b7-4b92-a3c0-ad80f9634057>
[2017-06-07 08:44:13.463460] I [MSGID: 115013] [server-helpers.c:296:do_fd_cleanup] 0-cross3-1-server: fd cleanup on <gfid:f553b864-2b4d-4a34-9066-ea05e34d29cc>
[2017-06-07 08:44:13.463717] I [MSGID: 115013] [server-helpers.c:296:do_fd_cleanup] 0-cross3-1-server: fd cleanup on <gfid:01a65aea-a651-4ed9-960f-f7bb4613a748>
[2017-06-07 08:44:13.469857] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp37-43.lab.eng.blr.redhat.com-11004-2017/05/20-08:02:20:362257-cross3-19-client-2-0-0
[2017-06-07 08:44:13.512892] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick19/cross3-19 - not found
[2017-06-07 08:44:18.191049] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick2/cross3-2
[2017-06-07 08:44:18.191129] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:1010
[2017-06-07 08:44:18.192018] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-28152-2017/05/17-14:41:24:443886-cross3-2-client-2-0-0
[2017-06-07 08:44:18.192611] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-28152-2017/05/17-14:41:24:443886-cross3-2-client-2-0-0
[2017-06-07 08:44:18.193007] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick2/cross3-2 - not found
[2017-06-07 08:44:21.062841] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick20/cross3-20
[2017-06-07 08:44:21.062871] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:884
[2017-06-07 08:44:21.063320] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-29143-2017/05/17-14:41:55:444481-cross3-20-client-2-0-0
[2017-06-07 08:44:21.063725] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-29143-2017/05/17-14:41:55:444481-cross3-20-client-2-0-0
[2017-06-07 08:44:21.064118] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick20/cross3-20 - not found
[2017-06-07 08:44:25.022927] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick21/cross3-21
[2017-06-07 08:44:25.023051] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:886
[2017-06-07 08:44:25.023212] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-29198-2017/05/17-14:42:22:178377-cross3-21-client-2-0-1
[2017-06-07 08:44:25.018916] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick21/cross3-21
[2017-06-07 08:44:25.030705] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-29198-2017/05/17-14:42:22:178377-cross3-21-client-2-0-1
pending frames:
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 
2017-06-07 08:44:25
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.8.4
/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xc2)[0x7f3314d691b2]
/lib64/libglusterfs.so.0(gf_print_trace+0x324)[0x7f3314d72bd4]
/lib64/libc.so.6(+0x35250)[0x7f331344b250]
/usr/sbin/glusterfsd(glusterfs_handle_terminate+0x147)[0x7f3315265d37]
/lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x7f3314d9f1f2]
/lib64/libc.so.6(+0x46cf0)[0x7f331345ccf0]
---------

Comment 4 Nag Pavan Chilakam 2017-06-07 09:22:54 UTC


sosreports@http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/bug.1459467/

Comment 7 Atin Mukherjee 2017-06-09 07:33:49 UTC

As per our agreement now, I'm closing this bug. If you happen to hit it again please reopen.