Hide Forgot
Created attachment 570265 [details] Brick Log File Description of problem: Server failed to create frame because of connection object being NULL and doesn't send reply back to client . Hence client application hangs waiting for a response from server. [2012-03-14 23:33:43.510756] E [server-helpers.c:735:server_alloc_frame] (-->/usr/local/lib/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x360) [0x7f679d2970a9] (-->/usr/local/ lib/glusterfs/3.3.0qa27/xlator/protocol/server.so(server_finodelk+0xcb) [0x7f6793bd53db] (-->/usr/local/lib/glusterfs/3.3.0qa27/xlator/protocol/server.so(get_frame_from _request+0x89) [0x7f6793bc11c5]))) 0-server: invalid argument: conn [2012-03-14 23:33:43.510836] W [rpcsvc.c:1111:rpcsvc_error_reply] (-->/usr/local/lib/libgfrpc.so.0(rpc_transport_notify+0x130) [0x7f679d29cda8] (-->/usr/local/lib/libgf rpc.so.0(rpcsvc_notify+0x181) [0x7f679d29744c] (-->/usr/local/lib/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x37b) [0x7f679d2970c4]))) 0-: sending a RPC error reply [2012-03-14 23:33:43.511039] E [server-helpers.c:735:server_alloc_frame] (-->/usr/local/lib/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x360) [0x7f679d2970a9] (-->/usr/local/ lib/glusterfs/3.3.0qa27/xlator/protocol/server.so(server_finodelk+0xcb) [0x7f6793bd53db] (-->/usr/local/lib/glusterfs/3.3.0qa27/xlator/protocol/server.so(get_frame_from _request+0x89) [0x7f6793bc11c5]))) 0-server: invalid argument: conn [2012-03-14 23:33:43.511129] W [rpcsvc.c:1111:rpcsvc_error_reply] (-->/usr/local/lib/libgfrpc.so.0(rpc_transport_notify+0x130) [0x7f679d29cda8] (-->/usr/local/lib/libgf rpc.so.0(rpcsvc_notify+0x181) [0x7f679d29744c] (-->/usr/local/lib/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x37b) [0x7f679d2970c4]))) 0-: sending a RPC error reply [2012-03-14 23:33:43.511596] I [socket.c:2377:socket_submit_reply] 0-tcp.dstore1-server: not connected (priv->connected = 255) [2012-03-14 23:33:43.511648] E [rpcsvc.c:1078:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x233872x, Program: GlusterFS 3.3.0qa27, ProgVers: 330, Proc: 30) to rpc-transport (tcp.dstore1-server) [2012-03-14 23:33:43.512071] I [socket.c:2377:socket_submit_reply] 0-tcp.dstore1-server: not connected (priv->connected = 255) [2012-03-14 23:33:43.512104] E [rpcsvc.c:1078:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x23261x, Program: GlusterFS 3.3.0qa27, ProgVers: 330, Proc: 27) to rpc-transport (tcp.dstore1-server) [2012-03-14 23:33:43.518857] E [server.c:166:server_submit_reply] (-->/usr/local/lib/libglusterfs.so.0(default_finodelk_cbk+0x14d) [0x7f679d4c97ca] (-->/usr/local/lib/glusterfs/3.3.0qa27/xlator/debug/io-stats.so(io_stats_finodelk_cbk+0x23a) [0x7f6793df2f2b] (-->/usr/local/lib/glusterfs/3.3.0qa27/xlator/protocol/server.so(server_finodelk_cbk+0x25e) [0x7f6793bc4f6a]))) 0-: Reply submission failed Version-Release number of selected component (if applicable): 3.3.0qa27 How reproducible: often Steps to Reproduce: 1.create a replicate volume (1 X 3) 2.create fuse mounts from client 3.run ping_pong test from fuse mount: ping_pong f13 -rw 100 50 300 4.bring down a brick while ping_pong is running 5.bring back the brick (changes the graph) Actual results: ping_pong test hangs on fuse mount when a brick goes down and come back online.
Now visiting this bug again, I found that there is no issues in the logs, and server side behavior. This can happen because connection is not established when the reply was about to be sent. The ping-pong hanging issue should be related to ping-timeout not properly done or something else. Surely not an issue on the server side. Re-open with 'ping-pong tests hang on fuse mount' summary if the behavior persist on latest mountpoint. Moving to 'wontfix' status for now.