Description of problem: ===================== I had about 46 volume in my brick mux setup While stopping these volumes, the glusterfsd process crashed on one of the nodes [root@dhcp35-45 ~]# gluster v list cross3-1 cross3-10 cross3-11 cross3-12 cross3-13 cross3-14 cross3-15 cross3-16 cross3-17 cross3-18 cross3-19 cross3-2 cross3-20 cross3-21 cross3-22 cross3-23 cross3-24 cross3-25 cross3-27 cross3-29 cross3-3 cross3-30 cross3-31 cross3-32 cross3-33 cross3-34 cross3-35 cross3-36 cross3-37 cross3-38 cross3-39 cross3-4 cross3-40 cross3-41 cross3-43 cross3-45 cross3-46 cross3-47 cross3-48 cross3-49 cross3-5 cross3-50 cross3-6 cross3-7 cross3-8 cross3-9 [root@dhcp35-45 ~]# rpm -qa|grep gluster glusterfs-server-3.8.4-25.el7rhgs.x86_64 glusterfs-3.8.4-25.el7rhgs.x86_64 glusterfs-events-3.8.4-25.el7rhgs.x86_64 glusterfs-fuse-3.8.4-25.el7rhgs.x86_64 glusterfs-rdma-3.8.4-25.el7rhgs.x86_64 gluster-nagios-addons-0.2.8-1.el7rhgs.x86_64 glusterfs-api-3.8.4-25.el7rhgs.x86_64 glusterfs-devel-3.8.4-25.el7rhgs.x86_64 glusterfs-geo-replication-3.8.4-25.el7rhgs.x86_64 gluster-nagios-common-0.2.4-1.el7rhgs.noarch glusterfs-client-xlators-3.8.4-25.el7rhgs.x86_64 glusterfs-api-devel-3.8.4-25.el7rhgs.x86_64 glusterfs-cli-3.8.4-25.el7rhgs.x86_64 glusterfs-debuginfo-3.8.4-25.el7rhgs.x86_64 glusterfs-libs-3.8.4-25.el7rhgs.x86_64 python-gluster-3.8.4-25.el7rhgs.noarch [root@dhcp35-45 ~]# for i in $(gluster v list);do gluster v stop $i;done Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y volume stop: cross3-1: success Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y volume stop: cross3-10: success Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y volume stop: cross3-11: success Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y volume stop: cross3-12: success Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y volume stop: cross3-13: success Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y volume stop: cross3-14: success Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y volume stop: cross3-15: success Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y volume stop: cross3-16: success Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y volume stop: cross3-17: success Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y volume stop: cross3-18: success Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y volume stop: cross3-19: success Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y volume stop: cross3-2: success Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y volume stop: cross3-20: success Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y volume stop: cross3-21: success Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y y Error : Request timed out Stopping volume will make its data inaccessible. Do you want to continue? (y/n) volume stop: cross3-23: failed: Commit failed on 10.70.35.122. Error: error Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y volume stop: cross3-24: failed: Commit failed on 10.70.35.122. Error: error Stopping volume will make its data inaccessible. Do you want to continue? (y/n) yy Invalid input, please enter y/n Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y volume stop: cross3-27: failed: Commit failed on 10.70.35.122. Error: error Stopping volume will make its data inaccessible. Do you want to continue? (y/n) n Stopping volume will make its data inaccessible. Do you want to continue? (y/n) ^C [root@dhcp35-45 ~]# gluster v status cross3-23: Volume cross3-23: does not exist [root@dhcp35-45 ~]# gluster v status cross3-23 Volume cross3-23 is not started [root@dhcp35-45 ~]# gluster v status Volume cross3-1 is not started Volume cross3-10 is not started Volume cross3-11 is not started Volume cross3-12 is not started Volume cross3-13 is not started Volume cross3-14 is not started Volume cross3-15 is not started Volume cross3-16 is not started Volume cross3-17 is not started Volume cross3-18 is not started Volume cross3-19 is not started Volume cross3-2 is not started Volume cross3-20 is not started Volume cross3-21 is not started Volume cross3-22 is not started Volume cross3-23 is not started Volume cross3-24 is not started Status of volume: cross3-25 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick25/cross3-25 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick25/cross3-25 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick25/cross3-25 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Task Status of Volume cross3-25 ------------------------------------------------------------------------------ There are no active volume tasks Volume cross3-27 is not started Status of volume: cross3-29 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick29/cross3-29 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick29/cross3-29 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick29/cross3-29 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Task Status of Volume cross3-29 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-3 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick3/cross3-3 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick3/cross3-3 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick3/cross3-3 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Task Status of Volume cross3-3 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-30 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick30/cross3-30 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick30/cross3-30 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick30/cross3-30 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Task Status of Volume cross3-30 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-31 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick31/cross3-31 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick31/cross3-31 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick31/cross3-31 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Task Status of Volume cross3-31 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-32 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick32/cross3-32 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick32/cross3-32 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick32/cross3-32 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Task Status of Volume cross3-32 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-33 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick33/cross3-33 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick33/cross3-33 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick33/cross3-33 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Task Status of Volume cross3-33 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-34 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick34/cross3-34 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick34/cross3-34 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick34/cross3-34 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Task Status of Volume cross3-34 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-35 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick35/cross3-35 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick35/cross3-35 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick35/cross3-35 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Task Status of Volume cross3-35 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-36 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick36/cross3-36 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick36/cross3-36 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick36/cross3-36 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Task Status of Volume cross3-36 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-37 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick37/cross3-37 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick37/cross3-37 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick37/cross3-37 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Task Status of Volume cross3-37 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-38 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick38/cross3-38 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick38/cross3-38 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick38/cross3-38 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Task Status of Volume cross3-38 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-39 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick39/cross3-39 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick39/cross3-39 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick39/cross3-39 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Task Status of Volume cross3-39 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-4 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick4/cross3-4 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick4/cross3-4 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick4/cross3-4 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Task Status of Volume cross3-4 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-40 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick40/cross3-40 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick40/cross3-40 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick40/cross3-40 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Task Status of Volume cross3-40 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-41 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick41/cross3-41 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick41/cross3-41 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick41/cross3-41 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Task Status of Volume cross3-41 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-43 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick43/cross3-43 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick43/cross3-43 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick43/cross3-43 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Task Status of Volume cross3-43 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-45 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick45/cross3-45 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick45/cross3-45 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick45/cross3-45 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Task Status of Volume cross3-45 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-46 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick46/cross3-46 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick46/cross3-46 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick46/cross3-46 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Task Status of Volume cross3-46 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-47 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick47/cross3-47 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick47/cross3-47 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick47/cross3-47 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Task Status of Volume cross3-47 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-48 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick48/cross3-48 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick48/cross3-48 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick48/cross3-48 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Task Status of Volume cross3-48 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-49 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick49/cross3-49 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick49/cross3-49 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick49/cross3-49 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Task Status of Volume cross3-49 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-5 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick5/cross3-5 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick5/cross3-5 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick5/cross3-5 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Task Status of Volume cross3-5 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-50 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick50/cross3-50 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick50/cross3-50 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick50/cross3-50 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Task Status of Volume cross3-50 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-6 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick6/cross3-6 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick6/cross3-6 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick6/cross3-6 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Task Status of Volume cross3-6 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-7 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick7/cross3-7 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick7/cross3-7 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick7/cross3-7 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Task Status of Volume cross3-7 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-8 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick8/cross3-8 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick8/cross3-8 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick8/cross3-8 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Task Status of Volume cross3-8 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cross3-9 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick9/cross3-9 49152 0 Y 6094 Brick 10.70.35.130:/rhs/brick9/cross3-9 49152 0 Y 22705 Brick 10.70.35.122:/rhs/brick9/cross3-9 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 15145 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26724 Self-heal Daemon on 10.70.35.130 N/A N/A Y 14822 Self-heal Daemon on 10.70.35.122 N/A N/A Y 29706 Task Status of Volume cross3-9 ------------------------------------------------------------------------------ There are no active volume tasks ############## on the node where glusterfsd crashed######### root@dhcp35-122 ~]# gluster v info cross3-23 Volume Name: cross3-23 Type: Replicate Volume ID: cda18b59-c670-43da-a5a5-7eb15fe4d1c0 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: 10.70.35.45:/rhs/brick23/cross3-23 Brick2: 10.70.35.130:/rhs/brick23/cross3-23 Brick3: 10.70.35.122:/rhs/brick23/cross3-23 Options Reconfigured: transport.address-family: inet nfs.disable: on cluster.brick-multiplex: on [root@dhcp35-122 ~]# gluster v status cross3-23: Volume cross3-23: does not exist [root@dhcp35-122 ~]# gluster v status cross3-23 Staging failed on 10.70.35.23. Error: Volume cross3-23 is not started Staging failed on 10.70.35.130. Error: Volume cross3-23 is not started Staging failed on dhcp35-45.lab.eng.blr.redhat.com. Error: Volume cross3-23 is not started [root@dhcp35-122 ~]# backtrace New LWP 22060] Reading symbols from /usr/sbin/glusterfsd...Reading symbols from /usr/lib/debug/usr/sbin/glusterfsd.debug...done. done. Missing separate debuginfo for Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/50/d1ccf000162361f74a9d7c2ecea856f7881f07 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/glusterfsd -s 10.70.35.122 --volfile-id cross3-1.10.70.35.122.rhs-bri'. Program terminated with signal 11, Segmentation fault. #0 0x00007f3315265d37 in glusterfs_handle_terminate (req=0x7f3302f3dd90) at glusterfsd-mgmt.c:254 254 *trav_p = (*trav_p)->next; Missing separate debuginfos, use: debuginfo-install glibc-2.17-157.el7_3.1.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-27.el7_3.x86_64 libacl-2.2.51-12.el7.x86_64 libaio-0.3.109-13.el7.x86_64 libattr-2.4.46-12.el7.x86_64 libcom_err-1.42.9-9.el7.x86_64 libgcc-4.8.5-11.el7.x86_64 libselinux-2.5-6.el7.x86_64 libuuid-2.23.2-33.el7.x86_64 openssl-libs-1.0.1e-60.el7_3.1.x86_64 pcre-8.32-15.el7_2.1.x86_64 sqlite-3.7.17-8.el7.x86_64 zlib-1.2.7-17.el7.x86_64 (gdb) bt #0 0x00007f3315265d37 in glusterfs_handle_terminate (req=0x7f3302f3dd90) at glusterfsd-mgmt.c:254 #1 0x00007f3314d9f1f2 in synctask_wrap (old_task=<optimized out>) at syncop.c:375 #2 0x00007f331345ccf0 in ?? () from /lib64/libc.so.6 #3 0x0000000000000000 in ?? () (gdb) #0 0x00007f3315265d37 in glusterfs_handle_terminate (req=0x7f3302f3dd90) at glusterfsd-mgmt.c:254 #1 0x00007f3314d9f1f2 in synctask_wrap (old_task=<optimized out>) at syncop.c:375 #2 0x00007f331345ccf0 in ?? () from /lib64/libc.so.6 #3 0x0000000000000000 in ?? () (gdb) t a a bt Thread 584 (Thread 0x7f32d81f4700 (LWP 22060)): #0 0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f3306197ccb in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:369 #2 0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f331350d73d in clone () from /lib64/libc.so.6 Thread 583 (Thread 0x7f31dd982700 (LWP 22348)): #0 0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f33065d9cf3 in changelog_ev_connector (data=0x7f32ba8b4228) at changelog-ev-handle.c:202 #2 0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f331350d73d in clone () from /lib64/libc.so.6 Thread 582 (Thread 0x7f325c1f2700 (LWP 22347)): #0 0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f3306197ccb in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:369 #2 0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f331350d73d in clone () from /lib64/libc.so.6 Thread 581 (Thread 0x7f32edaf9700 (LWP 21923)): #0 0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f3306197ccb in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:369 #2 0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f331350d73d in clone () from /lib64/libc.so.6 Thread 580 (Thread 0x7f32806f9700 (LWP 22218)): #0 0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f3306197ccb in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:369 #2 0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f331350d73d in clone () from /lib64/libc.so.6 Thread 579 (Thread 0x7f321c6f1700 (LWP 22067)): #0 0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f330771caab in posix_fsyncer_pick (this=this@entry=0x7f32e0964ec0, head=head@entry=0x7f321c6f0e80) at posix-helpers.c:1978 #2 0x00007f330771cd35 in posix_fsyncer (d=0x7f32e0964ec0) at posix-helpers.c:2077 #3 0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f331350d73d in clone () from /lib64/libc.so.6 Thread 578 (Thread 0x7f321f6f7700 (LWP 22061)): #0 0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f33065d9cf3 in changelog_ev_connector (data=0x7f32e09c5288) at changelog-ev-handle.c:202 #2 0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f331350d73d in clone () from /lib64/libc.so.6 Thread 577 (Thread 0x7f32ec5f2700 (LWP 21949)): #0 0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f3306197ccb in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:369 #2 0x00007f3313bc8dc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f331350d73d in clone () from /lib64/libc.so.6 Thread 576 (Thread 0x7f31a8397700 (LWP 22610)): #0 0x00007f3313bcc6d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f3306197ccb in br_stub_worker (data=<optimized out>) at bit-rot-stub-helpers.c:369 ---Type <return> to continue, or q <return> to quit--- version: 3.8.4-25
brick log where it crashed [2017-06-07 08:43:42.321599] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-130.lab.eng.blr.redhat.com-14070-2017/06/07-08:43:32:148610-cross3-50-client-2-0-0 [2017-06-07 08:43:42.321763] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-130.lab.eng.blr.redhat.com-14070-2017/06/07-08:43:32:148610-cross3-50-client-2-0-0 [2017-06-07 08:43:47.050575] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick12/cross3-12 [2017-06-07 08:43:47.050643] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:940 [2017-06-07 08:43:47.051202] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-28703-2017/05/17-14:41:54:841551-cross3-12-client-2-0-0 [2017-06-07 08:43:47.051632] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-28703-2017/05/17-14:41:54:841551-cross3-12-client-2-0-0 [2017-06-07 08:43:47.051963] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick12/cross3-12 - not found [2017-06-07 08:43:50.715257] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick13/cross3-13 [2017-06-07 08:43:50.715318] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:933 [2017-06-07 08:43:50.715787] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-28758-2017/05/17-14:41:54:920251-cross3-13-client-2-0-0 [2017-06-07 08:43:50.716787] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-28758-2017/05/17-14:41:54:920251-cross3-13-client-2-0-0 [2017-06-07 08:43:50.717204] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick13/cross3-13 - not found [2017-06-07 08:43:54.582866] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick14/cross3-14 [2017-06-07 08:43:54.582900] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:926 [2017-06-07 08:43:54.583330] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-28813-2017/05/17-14:41:55:558-cross3-14-client-2-0-0 [2017-06-07 08:43:54.588229] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-28813-2017/05/17-14:41:55:558-cross3-14-client-2-0-0 [2017-06-07 08:43:54.589010] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick14/cross3-14 [2017-06-07 08:43:59.304553] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick15/cross3-15 - not found [2017-06-07 08:43:59.305369] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick15/cross3-15 - not found [2017-06-07 08:44:02.097158] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick16/cross3-16 [2017-06-07 08:44:02.097214] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:912 [2017-06-07 08:44:02.097379] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-28923-2017/05/17-14:41:55:152920-cross3-16-client-2-0-0 [2017-06-07 08:44:02.097925] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-28923-2017/05/17-14:41:55:152920-cross3-16-client-2-0-0 [2017-06-07 08:44:02.098338] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick16/cross3-16 - not found [2017-06-07 08:44:05.831594] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick17/cross3-17 [2017-06-07 08:44:05.831639] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:905 [2017-06-07 08:44:05.831745] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-28978-2017/05/17-14:41:55:227086-cross3-17-client-2-0-0 [2017-06-07 08:44:05.832220] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-28978-2017/05/17-14:41:55:227086-cross3-17-client-2-0-0 [2017-06-07 08:44:05.832614] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick17/cross3-17 - not found [2017-06-07 08:44:09.639722] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick18/cross3-18 [2017-06-07 08:44:09.639765] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:898 [2017-06-07 08:44:09.639903] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-29033-2017/05/17-14:41:55:295359-cross3-18-client-2-0-0 [2017-06-07 08:44:09.640336] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-29033-2017/05/17-14:41:55:295359-cross3-18-client-2-0-0 [2017-06-07 08:44:09.640678] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick18/cross3-18 - not found [2017-06-07 08:44:13.453454] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick19/cross3-19 [2017-06-07 08:44:13.453504] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:891 [2017-06-07 08:44:13.453598] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.37.43:1017 [2017-06-07 08:44:13.453922] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-29088-2017/05/17-14:41:55:371105-cross3-19-client-2-0-0 [2017-06-07 08:44:13.454613] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-29088-2017/05/17-14:41:55:371105-cross3-19-client-2-0-0 [2017-06-07 08:44:13.454787] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp37-43.lab.eng.blr.redhat.com-11004-2017/05/20-08:02:20:362257-cross3-19-client-2-0-0 [2017-06-07 08:44:13.455144] I [MSGID: 115013] [server-helpers.c:296:do_fd_cleanup] 0-cross3-1-server: fd cleanup on <gfid:acff7539-45c1-4b43-9478-c93390cf2f1f> [2017-06-07 08:44:13.455464] I [MSGID: 115013] [server-helpers.c:296:do_fd_cleanup] 0-cross3-1-server: fd cleanup on <gfid:c4b8ef94-52b7-4b92-a3c0-ad80f9634057> [2017-06-07 08:44:13.463460] I [MSGID: 115013] [server-helpers.c:296:do_fd_cleanup] 0-cross3-1-server: fd cleanup on <gfid:f553b864-2b4d-4a34-9066-ea05e34d29cc> [2017-06-07 08:44:13.463717] I [MSGID: 115013] [server-helpers.c:296:do_fd_cleanup] 0-cross3-1-server: fd cleanup on <gfid:01a65aea-a651-4ed9-960f-f7bb4613a748> [2017-06-07 08:44:13.469857] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp37-43.lab.eng.blr.redhat.com-11004-2017/05/20-08:02:20:362257-cross3-19-client-2-0-0 [2017-06-07 08:44:13.512892] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick19/cross3-19 - not found [2017-06-07 08:44:18.191049] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick2/cross3-2 [2017-06-07 08:44:18.191129] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:1010 [2017-06-07 08:44:18.192018] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-28152-2017/05/17-14:41:24:443886-cross3-2-client-2-0-0 [2017-06-07 08:44:18.192611] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-28152-2017/05/17-14:41:24:443886-cross3-2-client-2-0-0 [2017-06-07 08:44:18.193007] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick2/cross3-2 - not found [2017-06-07 08:44:21.062841] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick20/cross3-20 [2017-06-07 08:44:21.062871] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:884 [2017-06-07 08:44:21.063320] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-29143-2017/05/17-14:41:55:444481-cross3-20-client-2-0-0 [2017-06-07 08:44:21.063725] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-29143-2017/05/17-14:41:55:444481-cross3-20-client-2-0-0 [2017-06-07 08:44:21.064118] E [glusterfsd-mgmt.c:225:glusterfs_handle_terminate] 0-glusterfs: can't terminate /rhs/brick20/cross3-20 - not found [2017-06-07 08:44:25.022927] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick21/cross3-21 [2017-06-07 08:44:25.023051] I [server.c:1508:notify] 0-cross3-1-server: disconnecting 10.70.35.103:886 [2017-06-07 08:44:25.023212] I [MSGID: 115036] [server.c:560:server_rpc_notify] 0-cross3-1-server: disconnecting connection from dhcp35-103.lab.eng.blr.redhat.com-29198-2017/05/17-14:42:22:178377-cross3-21-client-2-0-1 [2017-06-07 08:44:25.018916] I [glusterfsd-mgmt.c:250:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /rhs/brick21/cross3-21 [2017-06-07 08:44:25.030705] I [MSGID: 101055] [client_t.c:433:gf_client_unref] 0-cross3-1-server: Shutting down connection dhcp35-103.lab.eng.blr.redhat.com-29198-2017/05/17-14:42:22:178377-cross3-21-client-2-0-1 pending frames: frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2017-06-07 08:44:25 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.8.4 /lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xc2)[0x7f3314d691b2] /lib64/libglusterfs.so.0(gf_print_trace+0x324)[0x7f3314d72bd4] /lib64/libc.so.6(+0x35250)[0x7f331344b250] /usr/sbin/glusterfsd(glusterfs_handle_terminate+0x147)[0x7f3315265d37] /lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x7f3314d9f1f2] /lib64/libc.so.6(+0x46cf0)[0x7f331345ccf0] ---------
sosreports@http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/bug.1459467/
As per our agreement now, I'm closing this bug. If you happen to hit it again please reopen.