Description of problem: =============== with brick mux enabled, when we create say 10 volumes, and bring down one brick of a volume, all the remaining volumes show the root as heal pending Version-Release number of selected component (if applicable): ---------- 3.8.4-25 How reproducible: ======= always Steps to Reproduce: 1.have a 3 node setup, enable brick mux, create 10 1x3 volumes 2. bring down brick b1 of say vol9 using umount of LV 3. now perform some IOs on the vol9 4. Now check healinfo of vol9 --->as expected shows the new entries in heal pending 5. now check healinfo of all the other volumes, it will show the root as heal pending [root@dhcp35-45 ~]# #umount -l /rhs/brick9 [root@dhcp35-45 ~]# Brick 10.70.35.130:/rhs/brick9/test3_9 /dir1 / /dir1/f1 /dir1/f2 /dir1/f3 /dir1/f4 /dir1/f5 /dir1/f6 /dir1/f7 /dir1/f8 /dir1/f9 /dir1/f10 Status: Connected Number of entries: 12 Brick 10.70.35.122:/rhs/brick9/test3_9 /dir1 / /dir1/f1 /dir1/f2 /dir1/f3 /dir1/f4 /dir1/f5 /dir1/f6 /dir1/f7 /dir1/f8 /dir1/f9 /dir1/f10 Status: Connected Number of entries: 12 [root@dhcp35-45 ~]# gluster v heal test3_1 info Brick 10.70.35.45:/rhs/brick1/test3_1 Status: Connected Number of entries: 0 Brick 10.70.35.130:/rhs/brick1/test3_1 / Status: Connected Number of entries: 1 Brick 10.70.35.122:/rhs/brick1/test3_1 / Status: Connected Number of entries: 1
still seeing the issue on 3.8.4-27, hence fail_qa for i in $(gluster v list);do gluster v heal $i info;done Brick 10.70.35.45:/rhs/brick31/test3_31 Status: Transport endpoint is not connected Number of entries: - Brick 10.70.35.130:/rhs/brick31/test3_31 Status: Connected Number of entries: 0 Brick 10.70.35.122:/rhs/brick31/test3_31 Status: Connected Number of entries: 0 Brick 10.70.35.45:/rhs/brick32/test3_32 Status: Transport endpoint is not connected Number of entries: - Brick 10.70.35.130:/rhs/brick32/test3_32 Status: Connected Number of entries: 0 Brick 10.70.35.122:/rhs/brick32/test3_32 Status: Connected Number of entries: 0 Brick 10.70.35.45:/rhs/brick33/test3_33 Status: Transport endpoint is not connected Number of entries: - Brick 10.70.35.130:/rhs/brick33/test3_33 Status: Connected Number of entries: 0 Brick 10.70.35.122:/rhs/brick33/test3_33 Status: Connected Number of entries: 0 Brick 10.70.35.45:/rhs/brick34/test3_34 Status: Transport endpoint is not connected Number of entries: - Brick 10.70.35.130:/rhs/brick34/test3_34 / Status: Connected Number of entries: 1 Brick 10.70.35.122:/rhs/brick34/test3_34 / Status: Connected Number of entries: 1 Brick 10.70.35.45:/rhs/brick35/test3_35 Status: Transport endpoint is not connected Number of entries: - Brick 10.70.35.130:/rhs/brick35/test3_35 / Status: Connected Number of entries: 1 Brick 10.70.35.122:/rhs/brick35/test3_35 / Status: Connected Number of entries: 1 Brick 10.70.35.45:/rhs/brick36/test3_36 Status: Transport endpoint is not connected Number of entries: - Brick 10.70.35.130:/rhs/brick36/test3_36 / Status: Connected Number of entries: 1 Brick 10.70.35.122:/rhs/brick36/test3_36 / Status: Connected Number of entries: 1 Brick 10.70.35.45:/rhs/brick37/test3_37 Status: Transport endpoint is not connected Number of entries: - Brick 10.70.35.130:/rhs/brick37/test3_37 / Status: Connected Number of entries: 1 Brick 10.70.35.122:/rhs/brick37/test3_37 / Status: Connected Number of entries: 1 Brick 10.70.35.45:/rhs/brick38/test3_38 Status: Transport endpoint is not connected Number of entries: - Brick 10.70.35.130:/rhs/brick38/test3_38 / Status: Connected Number of entries: 1 Brick 10.70.35.122:/rhs/brick38/test3_38 / Status: Connected Number of entries: 1 Brick 10.70.35.45:/rhs/brick39/test3_39 Status: Transport endpoint is not connected Number of entries: - Brick 10.70.35.130:/rhs/brick39/test3_39 / Status: Connected Number of entries: 1 Brick 10.70.35.122:/rhs/brick39/test3_39 / Status: Connected Number of entries: 1 Brick 10.70.35.45:/rhs/brick40/test3_40 Status: Transport endpoint is not connected Number of entries: - Brick 10.70.35.130:/rhs/brick40/test3_40 Status: Connected Number of entries: 0 Brick 10.70.35.122:/rhs/brick40/test3_40 Status: Connected Number of entries: 0 Brick 10.70.35.45:/rhs/brick41/test3_41 Status: Transport endpoint is not connected Number of entries: - Brick 10.70.35.130:/rhs/brick41/test3_41 / Status: Connected Number of entries: 1 Brick 10.70.35.122:/rhs/brick41/test3_41 / Status: Connected Number of entries: 1 Brick 10.70.35.45:/rhs/brick42/test3_42 Status: Transport endpoint is not connected Number of entries: - Brick 10.70.35.130:/rhs/brick42/test3_42 / Status: Connected Number of entries: 1 Brick 10.70.35.122:/rhs/brick42/test3_42 / Status: Connected Number of entries: 1 Brick 10.70.35.45:/rhs/brick43/test3_43 Status: Transport endpoint is not connected Number of entries: - Brick 10.70.35.130:/rhs/brick43/test3_43 / Status: Connected Number of entries: 1 Brick 10.70.35.122:/rhs/brick43/test3_43 / Status: Connected Number of entries: 1 Brick 10.70.35.45:/rhs/brick44/test3_44 Status: Transport endpoint is not connected Number of entries: - Brick 10.70.35.130:/rhs/brick44/test3_44 / Status: Connected Number of entries: 1 Brick 10.70.35.122:/rhs/brick44/test3_44 / Status: Connected Number of entries: 1 Brick 10.70.35.45:/rhs/brick45/test3_45 Status: Transport endpoint is not connected Number of entries: - Brick 10.70.35.130:/rhs/brick45/test3_45 / Status: Connected Number of entries: 1 Brick 10.70.35.122:/rhs/brick45/test3_45 / Status: Connected Number of entries: 1 Brick 10.70.35.45:/rhs/brick46/test3_46 Status: Transport endpoint is not connected Number of entries: - Brick 10.70.35.130:/rhs/brick46/test3_46 / Status: Connected Number of entries: 1 Brick 10.70.35.122:/rhs/brick46/test3_46 / Status: Connected Number of entries: 1 Brick 10.70.35.45:/rhs/brick47/test3_47 Status: Transport endpoint is not connected Number of entries: - Brick 10.70.35.130:/rhs/brick47/test3_47 / Status: Connected Number of entries: 1 Brick 10.70.35.122:/rhs/brick47/test3_47 / Status: Connected Number of entries: 1 Brick 10.70.35.45:/rhs/brick48/test3_48 Status: Transport endpoint is not connected Number of entries: - Brick 10.70.35.130:/rhs/brick48/test3_48 / Status: Connected Number of entries: 1 Brick 10.70.35.122:/rhs/brick48/test3_48 / Status: Connected Number of entries: 1 Brick 10.70.35.45:/rhs/brick49/test3_49 Status: Transport endpoint is not connected Number of entries: - Brick 10.70.35.130:/rhs/brick49/test3_49 / Status: Connected Number of entries: 1 Brick 10.70.35.122:/rhs/brick49/test3_49 / Status: Connected Number of entries: 1 Brick 10.70.35.45:/rhs/brick50/test3_50 Status: Transport endpoint is not connected Number of entries: - Brick 10.70.35.130:/rhs/brick50/test3_50 Status: Connected Number of entries: 0 Brick 10.70.35.122:/rhs/brick50/test3_50 Status: Connected Number of entries: 0 Brick 10.70.35.45:/rhs/brick51/test3_51 Status: Connected Number of entries: 0 Brick 10.70.35.130:/rhs/brick51/test3_51 Status: Connected Number of entries: 0 Brick 10.70.35.122:/rhs/brick51/test3_51 Status: Connected Number of entries: 0 Brick 10.70.35.45:/rhs/brick60/test3_60 Status: Transport endpoint is not connected Number of entries: - Brick 10.70.35.130:/rhs/brick60/test3_60 / Status: Connected Number of entries: 1 Brick 10.70.35.122:/rhs/brick60/test3_60 / Status: Connected Number of entries: 1 Brick 10.70.35.45:/rhs/brick61/test3_61 Status: Transport endpoint is not connected Number of entries: - Brick 10.70.35.130:/rhs/brick61/test3_61 Status: Connected Number of entries: 0 Brick 10.70.35.122:/rhs/brick61/test3_61 Status: Connected Number of entries: 0 [root@dhcp35-45 ~]# gluster v status Status of volume: test3_31 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick31/test3_31 N/A N/A N N/A Brick 10.70.35.130:/rhs/brick31/test3_31 49152 0 Y 30495 Brick 10.70.35.122:/rhs/brick31/test3_31 49152 0 Y 14828 Self-heal Daemon on localhost N/A N/A Y 27795 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26963 Self-heal Daemon on 10.70.35.122 N/A N/A Y 17576 Self-heal Daemon on 10.70.35.130 N/A N/A Y 807 Task Status of Volume test3_31 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: test3_32 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick32/test3_32 N/A N/A N N/A Brick 10.70.35.130:/rhs/brick32/test3_32 49152 0 Y 30495 Brick 10.70.35.122:/rhs/brick32/test3_32 49152 0 Y 14828 Self-heal Daemon on localhost N/A N/A Y 27795 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26963 Self-heal Daemon on 10.70.35.130 N/A N/A Y 807 Self-heal Daemon on 10.70.35.122 N/A N/A Y 17576 Task Status of Volume test3_32 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: test3_33 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick33/test3_33 N/A N/A N N/A Brick 10.70.35.130:/rhs/brick33/test3_33 49152 0 Y 30495 Brick 10.70.35.122:/rhs/brick33/test3_33 49152 0 Y 14828 Self-heal Daemon on localhost N/A N/A Y 27795 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26963 Self-heal Daemon on 10.70.35.130 N/A N/A Y 807 Self-heal Daemon on 10.70.35.122 N/A N/A Y 17576 Task Status of Volume test3_33 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: test3_34 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick34/test3_34 N/A N/A N N/A Brick 10.70.35.130:/rhs/brick34/test3_34 49152 0 Y 30495 Brick 10.70.35.122:/rhs/brick34/test3_34 49152 0 Y 14828 Self-heal Daemon on localhost N/A N/A Y 27795 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26963 Self-heal Daemon on 10.70.35.122 N/A N/A Y 17576 Self-heal Daemon on 10.70.35.130 N/A N/A Y 807 Task Status of Volume test3_34 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: test3_35 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick35/test3_35 N/A N/A N N/A Brick 10.70.35.130:/rhs/brick35/test3_35 49152 0 Y 30495 Brick 10.70.35.122:/rhs/brick35/test3_35 49152 0 Y 14828 Self-heal Daemon on localhost N/A N/A Y 27795 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26963 Self-heal Daemon on 10.70.35.122 N/A N/A Y 17576 Self-heal Daemon on 10.70.35.130 N/A N/A Y 807 Task Status of Volume test3_35 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: test3_36 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick36/test3_36 N/A N/A N N/A Brick 10.70.35.130:/rhs/brick36/test3_36 49152 0 Y 30495 Brick 10.70.35.122:/rhs/brick36/test3_36 49152 0 Y 14828 Self-heal Daemon on localhost N/A N/A Y 27795 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26963 Self-heal Daemon on 10.70.35.122 N/A N/A Y 17576 Self-heal Daemon on 10.70.35.130 N/A N/A Y 807 Task Status of Volume test3_36 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: test3_37 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick37/test3_37 N/A N/A N N/A Brick 10.70.35.130:/rhs/brick37/test3_37 49152 0 Y 30495 Brick 10.70.35.122:/rhs/brick37/test3_37 49152 0 Y 14828 Self-heal Daemon on localhost N/A N/A Y 27795 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26963 Self-heal Daemon on 10.70.35.122 N/A N/A Y 17576 Self-heal Daemon on 10.70.35.130 N/A N/A Y 807 Task Status of Volume test3_37 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: test3_38 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick38/test3_38 N/A N/A N N/A Brick 10.70.35.130:/rhs/brick38/test3_38 49152 0 Y 30495 Brick 10.70.35.122:/rhs/brick38/test3_38 49152 0 Y 14828 Self-heal Daemon on localhost N/A N/A Y 27795 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26963 Self-heal Daemon on 10.70.35.130 N/A N/A Y 807 Self-heal Daemon on 10.70.35.122 N/A N/A Y 17576 Task Status of Volume test3_38 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: test3_39 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick39/test3_39 N/A N/A N N/A Brick 10.70.35.130:/rhs/brick39/test3_39 49152 0 Y 30495 Brick 10.70.35.122:/rhs/brick39/test3_39 49152 0 Y 14828 Self-heal Daemon on localhost N/A N/A Y 27795 Self-heal Daemon on 10.70.35.130 N/A N/A Y 807 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26963 Self-heal Daemon on 10.70.35.122 N/A N/A Y 17576 Task Status of Volume test3_39 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: test3_40 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick40/test3_40 N/A N/A N N/A Brick 10.70.35.130:/rhs/brick40/test3_40 49152 0 Y 30495 Brick 10.70.35.122:/rhs/brick40/test3_40 49152 0 Y 14828 Self-heal Daemon on localhost N/A N/A Y 27795 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26963 Self-heal Daemon on 10.70.35.122 N/A N/A Y 17576 Self-heal Daemon on 10.70.35.130 N/A N/A Y 807 Task Status of Volume test3_40 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: test3_41 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick41/test3_41 N/A N/A N N/A Brick 10.70.35.130:/rhs/brick41/test3_41 49152 0 Y 30495 Brick 10.70.35.122:/rhs/brick41/test3_41 49152 0 Y 14828 Self-heal Daemon on localhost N/A N/A Y 27795 Self-heal Daemon on 10.70.35.122 N/A N/A Y 17576 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26963 Self-heal Daemon on 10.70.35.130 N/A N/A Y 807 Task Status of Volume test3_41 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: test3_42 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick42/test3_42 N/A N/A N N/A Brick 10.70.35.130:/rhs/brick42/test3_42 49152 0 Y 30495 Brick 10.70.35.122:/rhs/brick42/test3_42 49152 0 Y 14828 Self-heal Daemon on localhost N/A N/A Y 27795 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26963 Self-heal Daemon on 10.70.35.122 N/A N/A Y 17576 Self-heal Daemon on 10.70.35.130 N/A N/A Y 807 Task Status of Volume test3_42 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: test3_43 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick43/test3_43 N/A N/A N N/A Brick 10.70.35.130:/rhs/brick43/test3_43 49152 0 Y 30495 Brick 10.70.35.122:/rhs/brick43/test3_43 49152 0 Y 14828 Self-heal Daemon on localhost N/A N/A Y 27795 Self-heal Daemon on 10.70.35.122 N/A N/A Y 17576 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26963 Self-heal Daemon on 10.70.35.130 N/A N/A Y 807 Task Status of Volume test3_43 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: test3_44 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick44/test3_44 N/A N/A N N/A Brick 10.70.35.130:/rhs/brick44/test3_44 49152 0 Y 30495 Brick 10.70.35.122:/rhs/brick44/test3_44 49152 0 Y 14828 Self-heal Daemon on localhost N/A N/A Y 27795 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26963 Self-heal Daemon on 10.70.35.130 N/A N/A Y 807 Self-heal Daemon on 10.70.35.122 N/A N/A Y 17576 Task Status of Volume test3_44 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: test3_45 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick45/test3_45 N/A N/A N N/A Brick 10.70.35.130:/rhs/brick45/test3_45 49152 0 Y 30495 Brick 10.70.35.122:/rhs/brick45/test3_45 49152 0 Y 14828 Self-heal Daemon on localhost N/A N/A Y 27795 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26963 Self-heal Daemon on 10.70.35.122 N/A N/A Y 17576 Self-heal Daemon on 10.70.35.130 N/A N/A Y 807 Task Status of Volume test3_45 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: test3_46 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick46/test3_46 N/A N/A N N/A Brick 10.70.35.130:/rhs/brick46/test3_46 49152 0 Y 30495 Brick 10.70.35.122:/rhs/brick46/test3_46 49152 0 Y 14828 Self-heal Daemon on localhost N/A N/A Y 27795 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26963 Self-heal Daemon on 10.70.35.130 N/A N/A Y 807 Self-heal Daemon on 10.70.35.122 N/A N/A Y 17576 Task Status of Volume test3_46 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: test3_47 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick47/test3_47 N/A N/A N N/A Brick 10.70.35.130:/rhs/brick47/test3_47 49152 0 Y 30495 Brick 10.70.35.122:/rhs/brick47/test3_47 49152 0 Y 14828 Self-heal Daemon on localhost N/A N/A Y 27795 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26963 Self-heal Daemon on 10.70.35.122 N/A N/A Y 17576 Self-heal Daemon on 10.70.35.130 N/A N/A Y 807 Task Status of Volume test3_47 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: test3_48 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick48/test3_48 N/A N/A N N/A Brick 10.70.35.130:/rhs/brick48/test3_48 49152 0 Y 30495 Brick 10.70.35.122:/rhs/brick48/test3_48 49152 0 Y 14828 Self-heal Daemon on localhost N/A N/A Y 27795 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26963 Self-heal Daemon on 10.70.35.130 N/A N/A Y 807 Self-heal Daemon on 10.70.35.122 N/A N/A Y 17576 Task Status of Volume test3_48 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: test3_49 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick49/test3_49 N/A N/A N N/A Brick 10.70.35.130:/rhs/brick49/test3_49 49152 0 Y 30495 Brick 10.70.35.122:/rhs/brick49/test3_49 49152 0 Y 14828 Self-heal Daemon on localhost N/A N/A Y 27795 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26963 Self-heal Daemon on 10.70.35.130 N/A N/A Y 807 Self-heal Daemon on 10.70.35.122 N/A N/A Y 17576 Task Status of Volume test3_49 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: test3_50 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick50/test3_50 N/A N/A N N/A Brick 10.70.35.130:/rhs/brick50/test3_50 49152 0 Y 30495 Brick 10.70.35.122:/rhs/brick50/test3_50 49152 0 Y 14828 Self-heal Daemon on localhost N/A N/A Y 27795 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26963 Self-heal Daemon on 10.70.35.130 N/A N/A Y 807 Self-heal Daemon on 10.70.35.122 N/A N/A Y 17576 Task Status of Volume test3_50 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: test3_51 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick51/test3_51 49153 0 Y 25139 Brick 10.70.35.130:/rhs/brick51/test3_51 49152 0 Y 30495 Brick 10.70.35.122:/rhs/brick51/test3_51 49152 0 Y 14828 Self-heal Daemon on localhost N/A N/A Y 27795 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26963 Self-heal Daemon on 10.70.35.130 N/A N/A Y 807 Self-heal Daemon on 10.70.35.122 N/A N/A Y 17576 Task Status of Volume test3_51 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: test3_60 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick60/test3_60 N/A N/A N N/A Brick 10.70.35.130:/rhs/brick60/test3_60 49152 0 Y 30495 Brick 10.70.35.122:/rhs/brick60/test3_60 49152 0 Y 14828 Self-heal Daemon on localhost N/A N/A Y 27795 Self-heal Daemon on 10.70.35.122 N/A N/A Y 17576 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26963 Self-heal Daemon on 10.70.35.130 N/A N/A Y 807 Task Status of Volume test3_60 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: test3_61 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick61/test3_61 N/A N/A N N/A Brick 10.70.35.130:/rhs/brick61/test3_61 49152 0 Y 30495 Brick 10.70.35.122:/rhs/brick61/test3_61 49152 0 Y 14828 Self-heal Daemon on localhost N/A N/A Y 27795 Self-heal Daemon on 10.70.35.23 N/A N/A Y 26963 Self-heal Daemon on 10.70.35.122 N/A N/A Y 17576 Self-heal Daemon on 10.70.35.130 N/A N/A Y 807 Task Status of Volume test3_61 ------------------------------------------------------------------------------ There are no active volume tasks [root@dhcp35-45 ~]#
Hi, I tried to reproduce the issue on my local environment after followed below steps but i am not able to reproduce it. Steps: 1) Created 3 1x3 volumes 2) Enable brick multiplex 3) Mount vol1 on mnt1 4) disable selfheal daemon(cluster.self-heal-daemon off) 5) Untar linux tar on mnt1 and for other vols also 6) After 2 min. Remove brick directory for vol1 on one node 7) Run below command on mnt1 to update permission for metadata changes find ./ -type f | xargs chmod 775 8) Stop/start vol1 and then check heal info output on all vols Regards Mohit Agrawal
Nag, We'd need a proper reproducer here as it looks like we couldn't hit this problem with the latest build.
I'm moving this bug back to ON_QA given we haven't been able to reproduce this issue with multiple attempts. If QE happens to failQA it again, please allow us with the setup for sometime where we can do a live debugging.
3.8.4-32: not seeing the issue again(tried 3 times in different ways). If i hit it again, i will either raise a new bug or reopen this one. Hence moving to verified
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2774