Description ============== With 873 file volumes in place, a gluster node was restarted. After gluster node restart, for most of the volumes, gluster v heal info is showing "Transport Endpoint Not Connected". Two different glusterfsd PIDs are seen in collective "gluster v status" command. But, as only one glusterfsd brick process shows up in ps -ef, the heal for all the volumes which show the other PID are reporting "Transport Endpoint Not Connected". Following are the steps performed and issues seen : 1. Setup has OCP 3.9 + CNS 3.9 along with glusterfs registry, logging and metrics configured. 2. Scaled up the setup with a total of 873 volumes. 3. It was seen that one of the node - 10.70.47.32 was in a NodeLost state due to resource constraint. Hence rebooted the node. 4. Once the node was up, glusterd took lot of time to come up. 5. Once the glusterd process came up - 5a) for around 809 volumes, the brick PID was 12641. This brick process was not showing up in "ps -ef|grep glusterfsd" and the bricks connected to this PID were in "Transport Endpoint Not Connected state" #gluster v heal <vol-name> info 5b) for the rest of the file volumes, the PID was 11393. This brick process was the only brick process listed in "ps -ef|grep glusterfsd". The heal info for these volumes did not report any issue for 10.70.46.32. 6. Also, on the node at times, we were seeing "fork: Cannot allocate memory". Hence , increased the memory from 48GB to 96 GB. Still the issue persisted [root@dhcp47-32 /]# ls /var/tmp -bash: fork: Cannot allocate memory Memory usage when Mem=48GB =========================== [root@dhcp47-32 ~]# free total used free shared buff/cache available Mem: 49289676 2771668 44024752 18300 2493256 44667312 Swap: 0 0 0 Memory usage when Mem=96GB =========================== [root@dhcp47-32 ~]# free total used free shared buff/cache available Mem: 98831776 2653424 94174832 17248 2003520 94394844 Swap: 0 0 0 Output for one such volume with "Status: Transport endpoint is not connected" ==================================================================================== sh-4.2# gluster v status vol_1971d3ad97c05e51c89ce95c3bd3753a Status of volume: vol_1971d3ad97c05e51c89ce95c3bd3753a Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.46.106:/var/lib/heketi/mounts/v g_cf10a43b398837706fcd3094301b59ae/brick_ba 1db9f827e3b35e5ce62bf160508376/brick 49152 0 Y 826 Brick 10.70.47.32:/var/lib/heketi/mounts/vg _b0a4650d964ecdf86c56c2ad334a294d/brick_371 74c6b3132d5079859080664b21047/brick 49152 0 Y 12641 Brick 10.70.46.171:/var/lib/heketi/mounts/v g_555d590bbfb592cac95eb6abde6a1d84/brick_cf c5c8d9f79deafc446a0bde84e3ae0a/brick 49152 0 Y 527 Self-heal Daemon on localhost N/A N/A Y 20886 Self-heal Daemon on dhcp46-106.lab.eng.blr. redhat.com N/A N/A Y 73690 Self-heal Daemon on dhcp46-171.lab.eng.blr. redhat.com N/A N/A Y 98661 Task Status of Volume vol_1971d3ad97c05e51c89ce95c3bd3753a ------------------------------------------------------------------------------ There are no active volume tasks sh-4.2# sh-4.2# sh-4.2# sh-4.2# gluster v heal vol_1971d3ad97c05e51c89ce95c3bd3753a info Brick 10.70.46.106:/var/lib/heketi/mounts/vg_cf10a43b398837706fcd3094301b59ae/brick_ba1db9f827e3b35e5ce62bf160508376/brick Status: Connected Number of entries: 0 Brick 10.70.47.32:/var/lib/heketi/mounts/vg_b0a4650d964ecdf86c56c2ad334a294d/brick_37174c6b3132d5079859080664b21047/brick Status: Transport endpoint is not connected Number of entries: - Brick 10.70.46.171:/var/lib/heketi/mounts/vg_555d590bbfb592cac95eb6abde6a1d84/brick_cfc5c8d9f79deafc446a0bde84e3ae0a/brick Status: Connected Number of entries: 0 Even manual heal on the selected volumes failed ================================================ sh-4.2# gluster v heal vol_1971d3ad97c05e51c89ce95c3bd3753a Launching heal operation to perform index self heal on volume vol_1971d3ad97c05e51c89ce95c3bd3753a has been unsuccessful on bricks that are down. Please check if all brick processes are running. Gluster peer status ====================== sh-4.2# gluster peer status Number of Peers: 2 Hostname: dhcp46-171.lab.eng.blr.redhat.com Uuid: 4be44c81-1fc2-429d-b4e9-3ee4b5acf719 State: Peer in Cluster (Connected) Other names: 10.70.46.171 Hostname: dhcp46-106.lab.eng.blr.redhat.com Uuid: 01f2729f-4f24-4ad2-8986-8af4cab73818 State: Peer in Cluster (Connected) sh-4.2# Output for one such volume with "Status: Connected" ===================================================== sh-4.2# gluster v heal vol_0c8f08ce143b86ea3d098b058ace70fa info Brick 10.70.46.106:/var/lib/heketi/mounts/vg_51e41d01385d095733056c28f0000db6/brick_13aa9108d9e62c35e6df905b79f87e93/brick Status: Connected Number of entries: 0 Brick 10.70.47.32:/var/lib/heketi/mounts/vg_4e448e5c4357c6327095c79ee982555b/brick_d3cd894a86bf73e3c592aad624b6abdd/brick Status: Connected Number of entries: 0 Brick 10.70.46.171:/var/lib/heketi/mounts/vg_85f98ae8b9d9e96d1b0c21409ef1a305/brick_cb03c508dde246a3844827fee687ee50/brick Status: Connected Number of entries: 0 sh-4.2# gluster v status vol_0c8f08ce143b86ea3d098b058ace70fa Status of volume: vol_0c8f08ce143b86ea3d098b058ace70fa Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.46.106:/var/lib/heketi/mounts/v g_51e41d01385d095733056c28f0000db6/brick_13 aa9108d9e62c35e6df905b79f87e93/brick 49152 0 Y 826 Brick 10.70.47.32:/var/lib/heketi/mounts/vg _4e448e5c4357c6327095c79ee982555b/brick_d3c d894a86bf73e3c592aad624b6abdd/brick 49152 0 Y 11393 Brick 10.70.46.171:/var/lib/heketi/mounts/v g_85f98ae8b9d9e96d1b0c21409ef1a305/brick_cb 03c508dde246a3844827fee687ee50/brick 49152 0 Y 527 Self-heal Daemon on localhost N/A N/A Y 20886 Self-heal Daemon on dhcp46-106.lab.eng.blr. redhat.com N/A N/A Y 73690 Self-heal Daemon on dhcp46-171.lab.eng.blr. redhat.com N/A N/A Y 98661 Task Status of Volume vol_0c8f08ce143b86ea3d098b058ace70fa ------------------------------------------------------------------------------ There are no active volume tasks sh-4.2# gluster vol status Status of volume: heketidbstorage Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.46.171:/var/lib/heketi/mounts/v g_555d590bbfb592cac95eb6abde6a1d84/brick_79 34177c0209de9f62c8b0ee9560b498/brick 49152 0 Y 527 Brick 10.70.46.106:/var/lib/heketi/mounts/v g_cf10a43b398837706fcd3094301b59ae/brick_9e c1513f5afc2f77664c59a6d41c079b/brick 49152 0 Y 826 Brick 10.70.47.32:/var/lib/heketi/mounts/vg _b0a4650d964ecdf86c56c2ad334a294d/brick_933 c85bb8b38deaa035e2d7f963b1cca/brick 49152 0 Y 11393 Self-heal Daemon on localhost N/A N/A Y 11384 Self-heal Daemon on dhcp46-171.lab.eng.blr. redhat.com N/A N/A Y 92363 Self-heal Daemon on dhcp46-106.lab.eng.blr. redhat.com N/A N/A Y 67031 gluster v heal heketidbstorage info +++++++++ Brick 10.70.46.171:/var/lib/heketi/mounts/vg_555d590bbfb592cac95eb6abde6a1d84/brick_7934177c0209de9f62c8b0ee9560b498/brick Status: Connected Number of entries: 0 Brick 10.70.46.106:/var/lib/heketi/mounts/vg_cf10a43b398837706fcd3094301b59ae/brick_9ec1513f5afc2f77664c59a6d41c079b/brick Status: Connected Number of entries: 0 Brick 10.70.47.32:/var/lib/heketi/mounts/vg_b0a4650d964ecdf86c56c2ad334a294d/brick_933c85bb8b38deaa035e2d7f963b1cca/brick Status: Connected Number of entries: 0 Versions ============== sh-4.2# rpm -qa|grep gluster glusterfs-libs-3.8.4-54.10.el7rhgs.x86_64 glusterfs-3.8.4-54.10.el7rhgs.x86_64 glusterfs-api-3.8.4-54.10.el7rhgs.x86_64 glusterfs-server-3.8.4-54.10.el7rhgs.x86_64 glusterfs-client-xlators-3.8.4-54.10.el7rhgs.x86_64 glusterfs-cli-3.8.4-54.10.el7rhgs.x86_64 glusterfs-fuse-3.8.4-54.10.el7rhgs.x86_64 glusterfs-geo-replication-3.8.4-54.10.el7rhgs.x86_64 gluster-block-0.2.1-14.1.el7rhgs.x86_64 sh-4.2# sh-4.2# uname -a Linux dhcp46-106.lab.eng.blr.redhat.com 3.10.0-862.9.1.el7.x86_64 #1 SMP Wed Jun 27 04:30:39 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux sh-4.2# Actual results: =================== Two brick PIDs are listed in gluster v status , even when one PID is actually running. Hence, for volumes with brick ID not in running state, the heal status shows "Transport Endpoint Not Connected" Expected results: ==================== All brick processes should come up after a node or a pod restart, even when the volume count is high. We should not have 2 PIDs listed for gluster volumes Additional info:
I verified this bug with 1010 volumes on following gluster rpms in cns -> sh-4.2# rpm -qa | grep gluster glusterfs-client-xlators-3.12.2-17.el7rhgs.x86_64 glusterfs-cli-3.12.2-17.el7rhgs.x86_64 python2-gluster-3.12.2-17.el7rhgs.x86_64 glusterfs-geo-replication-3.12.2-17.el7rhgs.x86_64 glusterfs-libs-3.12.2-17.el7rhgs.x86_64 glusterfs-3.12.2-17.el7rhgs.x86_64 glusterfs-api-3.12.2-17.el7rhgs.x86_64 glusterfs-fuse-3.12.2-17.el7rhgs.x86_64 glusterfs-server-3.12.2-17.el7rhgs.x86_64 gluster-block-0.2.1-25.el7rhgs.x86_64 sh-4.2# for i in $(gluster v list); do gluster v heal $i info; done | grep "Transport endpoint is not connected" sh-4.2# sh-4.2# sh-4.2# gluster v list | wc -l 1010 In ps also there was only one glusterfsd process is running for all 1010 bricks UID PID PPID C STIME TTY TIME CMD root 1 0 0 Aug24 ? 00:02:50 /usr/sbin/init root 38 1 0 Aug24 ? 00:00:00 /usr/sbin/lvmetad -f root 41 1 0 Aug24 ? 00:00:08 /usr/lib/systemd/systemd-udevd root 113 1 1 Aug24 ? 01:08:51 /usr/sbin/dmeventd -f dbus 1154 1 0 Aug24 ? 00:00:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation root 1156 1 0 Aug24 ? 00:00:00 /usr/sbin/gssproxy -D root 1175 1 0 Aug24 ? 00:00:00 /usr/sbin/crond -n root 1199 1 0 Aug24 ? 00:00:00 /usr/sbin/sshd -D root 21055 103514 0 12:51 pts/2 00:00:00 ps -ef root 23072 0 0 Aug24 pts/0 00:00:02 /bin/sh root 103514 0 0 09:59 pts/2 00:00:01 /bin/sh root 108745 1 1 Aug24 ? 01:35:53 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO rpc 108751 1 0 Aug24 ? 00:00:00 /sbin/rpcbind -w root 109041 1 0 Aug24 ? 00:00:00 /usr/bin/tcmu-runner --tcmu-log-dir /var/log/glusterfs/gluster-block root 109063 1 0 Aug24 ? 00:00:00 /usr/sbin/gluster-blockd --glfs-lru-count 15 --log-level INFO root 109237 1 1 Aug24 ? 01:27:23 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/gluster root 109246 1 9 Aug24 ? 08:59:47 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-bri I rebooted two node one by one, Everything was up and running, Hence marking it as verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2688