Description of problem: ======================== glusterd receiving responses from NULL uuid. The messages are seen in glusterd log file. [2011-12-04 00:12:05.096714] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f8d11ff3818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f8d11ff34d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f8d11ff2f3e]))) 0-management: forced unwinding frame type(glusterd mgmt) op(--(2)) called at 2011-12-04 00:06:46.119673 (xid=0x69x) [2011-12-04 00:12:05.118427] I [glusterd-rpc-ops.c:607:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received RJT from uuid: 00000000-0000-0000-0000-000000000000 [2011-12-04 00:12:05.130639] I [glusterd-handler.c:407:glusterd_friend_find] 0-glusterd: Unable to find peer by uuid [2011-12-04 00:12:05.130689] C [glusterd-rpc-ops.c:613:glusterd3_1_cluster_unlock_cbk] 0-: Unlock response received from unknown peer 00000000-0000-0000-0000-000000000000 Version-Release number of selected component (if applicable): =============================================================== [root@darrel ~]# gluster --version glusterfs 3.3.0.3rhs built on Oct 10 2012 09:16:20 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. [root@darrel ~]# uname -a Linux darrel.lab.eng.blr.redhat.com 2.6.32-220.28.1.el6.x86_64 #1 SMP Wed Oct 3 12:26:28 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux Steps to Reproduce: ====================== 1. Create a 2 node replicate volume 2. Create a fuse mount and execute the following command : "for i in `seq 1 10000`; do mkdir dir.$i ; for j in `seq 1 10` ; do dd if=/dev/input_file of=dir.$i/file.$j bs=1k count=1024 ; done ; done " 3. While dd is in progress , reboot one of the server. 4. Run the command 'gluster volume status all' from the active node. 5. once the server reboots, poweroff the server. Actual results: ============= saw glusterd receiving responses from NULL uuid.
The uuid that is being logged here is obtained from the response structure, no the one from the peerinfo. The response structure gotten by the unlock_cbk function can be an empty structure, when the cbk is called because of connection failure (which causes a call_bail or a disconnect event). This causes the messages in the logs about null uuid. This is not a problem caused by peer probe or any other peer state-machine issues.