Bug 1459467 - Brick Multiplexing: brick process crashed while stopping volumes
Brick Multiplexing: brick process crashed while stopping volumes
Status: CLOSED WORKSFORME
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: core (Show other bugs)
3.3
Unspecified Unspecified
unspecified Severity urgent
: ---
: ---
Assigned To: Vijay Bellur
nchilaka
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-07 04:57 EDT by nchilaka
Modified: 2017-06-09 03:33 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-06-09 03:33:49 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description nchilaka 2017-06-07 04:57:02 EDT
Description of problem:
=====================
I had about 46 volume in my brick mux setup
While stopping these volumes, the glusterfsd process crashed on one of the nodes
[root@dhcp35-45 ~]# gluster v list
cross3-1
cross3-10
cross3-11
cross3-12
cross3-13
cross3-14
cross3-15
cross3-16
cross3-17
cross3-18
cross3-19
cross3-2
cross3-20
cross3-21
cross3-22
cross3-23
cross3-24
cross3-25
cross3-27
cross3-29
cross3-3
cross3-30
cross3-31
cross3-32
cross3-33
cross3-34
cross3-35
cross3-36
cross3-37
cross3-38
cross3-39
cross3-4
cross3-40
cross3-41
cross3-43
cross3-45
cross3-46
cross3-47
cross3-48
cross3-49
cross3-5
cross3-50
cross3-6
cross3-7
cross3-8
cross3-9
[root@dhcp35-45 ~]# rpm -qa|grep gluster
glusterfs-server-3.8.4-25.el7rhgs.x86_64
glusterfs-3.8.4-25.el7rhgs.x86_64
glusterfs-events-3.8.4-25.el7rhgs.x86_64
glusterfs-fuse-3.8.4-25.el7rhgs.x86_64
glusterfs-rdma-3.8.4-25.el7rhgs.x86_64
gluster-nagios-addons-0.2.8-1.el7rhgs.x86_64
glusterfs-api-3.8.4-25.el7rhgs.x86_64
glusterfs-devel-3.8.4-25.el7rhgs.x86_64
glusterfs-geo-replication-3.8.4-25.el7rhgs.x86_64
gluster-nagios-common-0.2.4-1.el7rhgs.noarch
glusterfs-client-xlators-3.8.4-25.el7rhgs.x86_64
glusterfs-api-devel-3.8.4-25.el7rhgs.x86_64
glusterfs-cli-3.8.4-25.el7rhgs.x86_64
glusterfs-debuginfo-3.8.4-25.el7rhgs.x86_64
glusterfs-libs-3.8.4-25.el7rhgs.x86_64
python-gluster-3.8.4-25.el7rhgs.noarch
[root@dhcp35-45 ~]# for i in $(gluster v list);do gluster v stop $i;done
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-1: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-10: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-11: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-12: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-13: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-14: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-15: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-16: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-17: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-18: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-19: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-2: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-20: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-21: success
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
y
Error : Request timed out
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) volume stop: cross3-23: failed: Commit failed on 10.70.35.122. Error: error
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-24: failed: Commit failed on 10.70.35.122. Error: error
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) yy
Invalid input, please enter y/n
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: cross3-27: failed: Commit failed on 10.70.35.122. Error: error
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) n
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) ^C
[root@dhcp35-45 ~]# gluster v status cross3-23:
Volume cross3-23: does not exist
[root@dhcp35-45 ~]# gluster v status cross3-23
Volume cross3-23 is not started
[root@dhcp35-45 ~]# gluster v status
Volume cross3-1 is not started
 
Volume cross3-10 is not started
 
Volume cross3-11 is not started
 
Volume cross3-12 is not started
 
Volume cross3-13 is not started
 
Volume cross3-14 is not started
 
Volume cross3-15 is not started
 
Volume cross3-16 is not started
 
Volume cross3-17 is not started
 
Volume cross3-18 is not started
 
Volume cross3-19 is not started
 
Volume cross3-2 is not started
 
Volume cross3-20 is not started
 
Volume cross3-21 is not started
 
Volume cross3-22 is not started
 
Volume cross3-23 is not started
 
Volume cross3-24 is not started
 
Status of volume: cross3-25
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick25/cross3-25    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick25/cross3-25   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick25/cross3-25   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-25
------------------------------------------------------------------------------
There are no active volume tasks
 
Volume cross3-27 is not started
 
Status of volume: cross3-29
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick29/cross3-29    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick29/cross3-29   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick29/cross3-29   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-29
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-3
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick3/cross3-3      49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick3/cross3-3     49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick3/cross3-3     N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-3
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-30
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick30/cross3-30    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick30/cross3-30   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick30/cross3-30   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-30
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-31
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick31/cross3-31    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick31/cross3-31   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick31/cross3-31   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-31
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-32
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick32/cross3-32    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick32/cross3-32   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick32/cross3-32   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-32
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-33
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick33/cross3-33    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick33/cross3-33   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick33/cross3-33   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-33
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-34
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick34/cross3-34    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick34/cross3-34   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick34/cross3-34   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-34
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-35
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick35/cross3-35    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick35/cross3-35   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick35/cross3-35   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
 
Task Status of Volume cross3-35
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-36
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick36/cross3-36    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick36/cross3-36   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick36/cross3-36   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
 
Task Status of Volume cross3-36
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-37
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick37/cross3-37    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick37/cross3-37   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick37/cross3-37   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
 
Task Status of Volume cross3-37
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-38
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick38/cross3-38    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick38/cross3-38   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick38/cross3-38   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
 
Task Status of Volume cross3-38
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-39
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick39/cross3-39    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick39/cross3-39   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick39/cross3-39   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-39
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-4
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick4/cross3-4      49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick4/cross3-4     49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick4/cross3-4     N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-4
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-40
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick40/cross3-40    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick40/cross3-40   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick40/cross3-40   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-40
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-41
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick41/cross3-41    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick41/cross3-41   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick41/cross3-41   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-41
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-43
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick43/cross3-43    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick43/cross3-43   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick43/cross3-43   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-43
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-45
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick45/cross3-45    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick45/cross3-45   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick45/cross3-45   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-45
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-46
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick46/cross3-46    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick46/cross3-46   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick46/cross3-46   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-46
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-47
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick47/cross3-47    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick47/cross3-47   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick47/cross3-47   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-47
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-48
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick48/cross3-48    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick48/cross3-48   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick48/cross3-48   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-48
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-49
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick49/cross3-49    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick49/cross3-49   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick49/cross3-49   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
 
Task Status of Volume cross3-49
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-5
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick5/cross3-5      49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick5/cross3-5     49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick5/cross3-5     N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-5
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-50
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick50/cross3-50    49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick50/cross3-50   49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick50/cross3-50   N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-50
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-6
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick6/cross3-6      49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick6/cross3-6     49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick6/cross3-6     N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-6
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-7
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick7/cross3-7      49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick7/cross3-7     49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick7/cross3-7     N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-7
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-8
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick8/cross3-8      49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick8/cross3-8     49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick8/cross3-8     N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-8
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: cross3-9
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick9/cross3-9      49152     0          Y       6094 
Brick 10.70.35.130:/rhs/brick9/cross3-9     49152     0          Y       22705
Brick 10.70.35.122:/rhs/brick9/cross3-9     N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       15145
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       26724
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       14822
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       29706
 
Task Status of Volume cross3-9
------------------------------------------------------------------------------
There are no active volume tasks
 



############## on the node where glusterfsd crashed#########
root@dhcp35-122 ~]# gluster v info cross3-23
 
Volume Name: cross3-23
Type: Replicate
Volume ID: cda18b59-c670-43da-a5a5-7eb15fe4d1c0
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.70.35.45:/rhs/brick23/cross3-23
Brick2: 10.70.35.130:/rhs/brick23/cross3-23
Brick3: 10.70.35.122:/rhs/brick23/cross3-23
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
cluster.brick-multiplex: on
[root@dhcp35-122 ~]# gluster v status cross3-23:
Volume cross3-23: does not exist
[root@dhcp35-122 ~]# gluster v status cross3-23
Staging failed on 10.70.35.23. Error: Volume cross3-23 is not started
Staging failed on 10.70.35.130. Error: Volume cross3-23 is not started
Staging failed on dhcp35-45.lab.eng.blr.redhat.com. Error: Volume cross3-23 is not started
[root@dhcp35-122 ~]# 


backtrace

New LWP 22060]
Reading symbols from /usr/sbin/glusterfsd...Reading symbols from /usr/lib/debug/usr/sbin/glusterfsd.debug...done.
done.
Missing separate debuginfo for 
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/50/d1ccf000162361f74a9d7c2ecea856f7881f07
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/glusterfsd -s 10.70.35.122 --volfile-id cross3-1.10.70.35.122.rhs-bri'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f3315265d37 in glusterfs_handle_terminate (req=0x7f3302f3dd90) at glusterfsd-mgmt.c:254
254	                *trav_p = (*trav_p)->next;
Missing separate debuginfos, use: debuginfo-install glibc-2.17-157.el7_3.1.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-27.el7_3.x86_64 libacl-2.2.51-12.el7.x86_64 libaio-0.3.109-13.el7.x86_64 libattr-2.4.46-12.el7.x86_64 libcom_err-1.42.9-9.el7.x86_64 libgcc-4.8.5-11.el7.x86_64 libselinux-2.5-6.el7.x86_64 libuuid-2.23.2-33.el7.x86_64 openssl-libs-1.0.1e-60.el7_3.1.x86_64 pcre-8.32-15.el7_2.1.x86_64 sqlite-3.7.17-8.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0  0x00007f3315265d37 in glusterfs_handle_terminate (req=0x7f3302f3dd90) at glusterfsd-mgmt.c:254
#1  0x00007f3314d9f1f2 in synctask_wrap (old_task=<optimized out>) at syncop.c:375
#2  0x00007f331345ccf0 in ?? () from /lib64/libc.so.6
#3  0x0000000000000000 in ?? ()
(gdb) 
#0  0x00007f3315265d37 in glusterfs_handle_terminate (req=0x7f3302f3dd90) at glusterfsd-mgmt.c:254
#1  0x00007f3314d9f1f2 in synctask_wrap (old_task=<optimized out>) at syncop.c:375
#2  0x00007f331345ccf0 in ?? () from /lib64/libc.so.6
#3  0x0000000000000000 in ?? ()
(gdb) t a a bt

Thread 584 (Thread 0x7f32d81f4700 (LWP 22060)):
#0  0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f3306197ccb in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:369
#2  0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f331350d73d in clone () from /lib64/libc.so.6

Thread 583 (Thread 0x7f31dd982700 (LWP 22348)):
#0  0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f33065d9cf3 in changelog_ev_connector (data=0x7f32ba8b4228) at changelog-ev-handle.c:202
#2  0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f331350d73d in clone () from /lib64/libc.so.6

Thread 582 (Thread 0x7f325c1f2700 (LWP 22347)):
#0  0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f3306197ccb in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:369
#2  0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f331350d73d in clone () from /lib64/libc.so.6

Thread 581 (Thread 0x7f32edaf9700 (LWP 21923)):
#0  0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f3306197ccb in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:369
#2  0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f331350d73d in clone () from /lib64/libc.so.6

Thread 580 (Thread 0x7f32806f9700 (LWP 22218)):
#0  0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f3306197ccb in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:369
#2  0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f331350d73d in clone () from /lib64/libc.so.6

Thread 579 (Thread 0x7f321c6f1700 (LWP 22067)):
#0  0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f330771caab in posix_fsyncer_pick (this=this@entry=0x7f32e0964ec0, head=head@entry=0x7f321c6f0e80) at posix-helpers.c:1978
#2  0x00007f330771cd35 in posix_fsyncer (d=0x7f32e0964ec0) at posix-helpers.c:2077
#3  0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f331350d73d in clone () from /lib64/libc.so.6

Thread 578 (Thread 0x7f321f6f7700 (LWP 22061)):
#0  0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f33065d9cf3 in changelog_ev_connector (data=0x7f32e09c5288) at changelog-ev-handle.c:202
#2  0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f331350d73d in clone () from /lib64/libc.so.6

Thread 577 (Thread 0x7f32ec5f2700 (LWP 21949)):
#0  0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f3306197ccb in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:369
#2  0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f331350d73d in clone () from /lib64/libc.so.6

Thread 576 (Thread 0x7f31a8397700 (LWP 22610)):
#0  0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f3306197ccb in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:369
---Type <return> to continue, or q <return> to quit--- 


version:
3.8.4-25
Comment 2 nchilaka 2017-06-07 04:58:56 EDT
brick log where it crashed


[2017-06-07 08:43:42.321599] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-130.lab.eng.blr.redhat.com-14070-2017/06/07-08:43:32:148610-cross3-50-client-2-0-0
[2017-06-07 08:43:42.321763] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-130.lab.eng.blr.redhat.com-14070-2017/06/07-08:43:32:148610-cross3-50-client-2-0-0
[2017-06-07 08:43:47.050575] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick12/cross3-12
[2017-06-07 08:43:47.050643] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:940
[2017-06-07 08:43:47.051202] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-28703-2017/05/17-14:41:54:841551-cross3-12-client-2-0-0
[2017-06-07 08:43:47.051632] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-28703-2017/05/17-14:41:54:841551-cross3-12-client-2-0-0
[2017-06-07 08:43:47.051963] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick12/cross3-12 - not found
[2017-06-07 08:43:50.715257] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick13/cross3-13
[2017-06-07 08:43:50.715318] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:933
[2017-06-07 08:43:50.715787] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-28758-2017/05/17-14:41:54:920251-cross3-13-client-2-0-0
[2017-06-07 08:43:50.716787] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-28758-2017/05/17-14:41:54:920251-cross3-13-client-2-0-0
[2017-06-07 08:43:50.717204] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick13/cross3-13 - not found
[2017-06-07 08:43:54.582866] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick14/cross3-14
[2017-06-07 08:43:54.582900] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:926
[2017-06-07 08:43:54.583330] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-28813-2017/05/17-14:41:55:558-cross3-14-client-2-0-0
[2017-06-07 08:43:54.588229] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-28813-2017/05/17-14:41:55:558-cross3-14-client-2-0-0
[2017-06-07 08:43:54.589010] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick14/cross3-14
[2017-06-07 08:43:59.304553] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick15/cross3-15 - not found
[2017-06-07 08:43:59.305369] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick15/cross3-15 - not found
[2017-06-07 08:44:02.097158] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick16/cross3-16
[2017-06-07 08:44:02.097214] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:912
[2017-06-07 08:44:02.097379] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-28923-2017/05/17-14:41:55:152920-cross3-16-client-2-0-0
[2017-06-07 08:44:02.097925] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-28923-2017/05/17-14:41:55:152920-cross3-16-client-2-0-0
[2017-06-07 08:44:02.098338] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick16/cross3-16 - not found
[2017-06-07 08:44:05.831594] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick17/cross3-17
[2017-06-07 08:44:05.831639] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:905
[2017-06-07 08:44:05.831745] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-28978-2017/05/17-14:41:55:227086-cross3-17-client-2-0-0
[2017-06-07 08:44:05.832220] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-28978-2017/05/17-14:41:55:227086-cross3-17-client-2-0-0
[2017-06-07 08:44:05.832614] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick17/cross3-17 - not found
[2017-06-07 08:44:09.639722] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick18/cross3-18
[2017-06-07 08:44:09.639765] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:898
[2017-06-07 08:44:09.639903] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-29033-2017/05/17-14:41:55:295359-cross3-18-client-2-0-0
[2017-06-07 08:44:09.640336] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-29033-2017/05/17-14:41:55:295359-cross3-18-client-2-0-0
[2017-06-07 08:44:09.640678] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick18/cross3-18 - not found
[2017-06-07 08:44:13.453454] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick19/cross3-19
[2017-06-07 08:44:13.453504] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:891
[2017-06-07 08:44:13.453598] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.37.43:1017
[2017-06-07 08:44:13.453922] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-29088-2017/05/17-14:41:55:371105-cross3-19-client-2-0-0
[2017-06-07 08:44:13.454613] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-29088-2017/05/17-14:41:55:371105-cross3-19-client-2-0-0
[2017-06-07 08:44:13.454787] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp37-43.lab.eng.blr.redhat.com-11004-2017/05/20-08:02:20:362257-cross3-19-client-2-0-0
[2017-06-07 08:44:13.455144] I [MSGID: 115013] [server-helpers.c:296:do_fd_cleanup] 0-cross3-1-server: fd cleanup on <gfid:acff7539-45c1-4b43-9478-c93390cf2f1f>
[2017-06-07 08:44:13.455464] I [MSGID: 115013] [server-helpers.c:296:do_fd_cleanup] 0-cross3-1-server: fd cleanup on <gfid:c4b8ef94-52b7-4b92-a3c0-ad80f9634057>
[2017-06-07 08:44:13.463460] I [MSGID: 115013] [server-helpers.c:296:do_fd_cleanup] 0-cross3-1-server: fd cleanup on <gfid:f553b864-2b4d-4a34-9066-ea05e34d29cc>
[2017-06-07 08:44:13.463717] I [MSGID: 115013] [server-helpers.c:296:do_fd_cleanup] 0-cross3-1-server: fd cleanup on <gfid:01a65aea-a651-4ed9-960f-f7bb4613a748>
[2017-06-07 08:44:13.469857] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp37-43.lab.eng.blr.redhat.com-11004-2017/05/20-08:02:20:362257-cross3-19-client-2-0-0
[2017-06-07 08:44:13.512892] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick19/cross3-19 - not found
[2017-06-07 08:44:18.191049] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick2/cross3-2
[2017-06-07 08:44:18.191129] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:1010
[2017-06-07 08:44:18.192018] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-28152-2017/05/17-14:41:24:443886-cross3-2-client-2-0-0
[2017-06-07 08:44:18.192611] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-28152-2017/05/17-14:41:24:443886-cross3-2-client-2-0-0
[2017-06-07 08:44:18.193007] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick2/cross3-2 - not found
[2017-06-07 08:44:21.062841] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick20/cross3-20
[2017-06-07 08:44:21.062871] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:884
[2017-06-07 08:44:21.063320] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-29143-2017/05/17-14:41:55:444481-cross3-20-client-2-0-0
[2017-06-07 08:44:21.063725] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-29143-2017/05/17-14:41:55:444481-cross3-20-client-2-0-0
[2017-06-07 08:44:21.064118] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick20/cross3-20 - not found
[2017-06-07 08:44:25.022927] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick21/cross3-21
[2017-06-07 08:44:25.023051] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:886
[2017-06-07 08:44:25.023212] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-29198-2017/05/17-14:42:22:178377-cross3-21-client-2-0-1
[2017-06-07 08:44:25.018916] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick21/cross3-21
[2017-06-07 08:44:25.030705] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-29198-2017/05/17-14:42:22:178377-cross3-21-client-2-0-1
pending frames:
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 
2017-06-07 08:44:25
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.8.4
/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xc2)[0x7f3314d691b2]
/lib64/libglusterfs.so.0(gf_print_trace+0x324)[0x7f3314d72bd4]
/lib64/libc.so.6(+0x35250)[0x7f331344b250]
/usr/sbin/glusterfsd(glusterfs_handle_terminate+0x147)[0x7f3315265d37]
/lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x7f3314d9f1f2]
/lib64/libc.so.6(+0x46cf0)[0x7f331345ccf0]
---------
Comment 7 Atin Mukherjee 2017-06-09 03:33:49 EDT
As per our agreement now, I'm closing this bug. If you happen to hit it again please reopen.

Note You need to log in before you can comment on or make changes to this bug.