Description of problem: I compiled glusterfs with CFLAGS="-g3 -DDEBUG -lgcov --coverage" Was running sanity on a distributed replicate volume, with quota & geo-rep enabled. then I also set limit-usage of 10GB. These errors happened after brick down/up was done on multiple bricks. I was running sanity on 1 client, while running fileop, i got these errors. [2011-12-23 02:03:30.084334] W [client3_1-fops.c:1273:client3_1_finodelk_cbk] 2-vol-client-2: remote operation failed: Invalid argument [2011-12-23 02:03:30.084373] E [afr-lk-common.c:567:afr_unlock_inodelk_cbk] 2-vol-replicate-1: /system_light/run25721/dir1/dir5/file514: unlock failed on 0, reason: Invalid argument [2011-12-23 02:04:13.177107] W [client3_1-fops.c:1273:client3_1_finodelk_cbk] 2-vol-client-4: remote operation failed: Invalid argument [2011-12-23 02:04:13.177141] E [afr-lk-common.c:567:afr_unlock_inodelk_cbk] 2-vol-replicate-2: /system_light/run25721/dir2/dir5/file53: unlock failed on 0, r eason: Invalid argument [2011-12-23 02:04:42.954784] W [client3_1-fops.c:1273:client3_1_finodelk_cbk] 2-vol-client-4: remote operation failed: Invalid argument [2011-12-23 02:04:42.954816] E [afr-lk-common.c:567:afr_unlock_inodelk_cbk] 2-vol-replicate-2: /system_light/run25721/dir1/dir16/file1619: unlock failed on 0 , reason: Invalid argument [2011-12-23 02:05:31.524251] W [client3_1-fops.c:1273:client3_1_finodelk_cbk] 2-vol-client-0: remote operation failed: Invalid argument [2011-12-23 02:05:31.524285] E [afr-lk-common.c:567:afr_unlock_inodelk_cbk] 2-vol-replicate-0: /system_light/run25721/dir1/dir18/file180: unlock failed on 0, reason: Invalid argument [2011-12-23 02:05:39.144078] W [client3_1-fops.c:1273:client3_1_finodelk_cbk] 2-vol-client-2: remote operation failed: Invalid argument [2011-12-23 02:05:39.144131] E [afr-lk-common.c:567:afr_unlock_inodelk_cbk] 2-vol-replicate-1: /system_light/run25721/dir2/dir1/file18: unlock failed on 0, r eason: Invalid argument P.S: I have provided the logs to pranith, not tried more times, just wanted to log incase it was a bug. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Happens with qa19 as well. client logs: [2012-01-10 16:49:30.333224] E [afr-lk-common.c:567:afr_unlock_inodelk_cbk] 2-vol-replicate-0: /system_light/run13879/fileop_L1_10/fileop_L1_10_L2_5/fileop_d ir_10_5_29: unlock failed on 0, reason: Invalid argument [2012-01-10 16:50:31.080786] W [client3_1-fops.c:1235:client3_1_inodelk_cbk] 2-vol-client-2: remote operation failed: Invalid argument [2012-01-10 16:50:31.080818] E [afr-lk-common.c:567:afr_unlock_inodelk_cbk] 2-vol-replicate-1: /system_light/run13879/fileop_L1_16/fileop_L1_16_L2_16/fileop_ dir_16_16_9: unlock failed on 0, reason: Invalid argument [2012-01-10 16:50:32.663965] W [client3_1-fops.c:1235:client3_1_inodelk_cbk] 2-vol-client-0: remote operation failed: Invalid argument [2012-01-10 16:50:32.663992] E [afr-lk-common.c:567:afr_unlock_inodelk_cbk] 2-vol-replicate-0: /system_light/run13879/fileop_L1_16/fileop_L1_16_L2_24/fileop_ dir_16_24_7: unlock failed on 0, reason: Invalid argument [2012-01-10 16:51:16.024236] W [client3_1-fops.c:1235:client3_1_inodelk_cbk] 2-vol-client-0: remote operation failed: Invalid argument [2012-01-10 16:51:16.024264] E [afr-lk-common.c:567:afr_unlock_inodelk_cbk] 2-vol-replicate-0: /system_light/run13879/fileop_L1_21/fileop_L1_21_L2_12/fileop_ dir_21_12_22: unlock failed on 0, reason: Invalid argument [2012-01-10 16:52:30.437856] W [client3_1-fops.c:1235:client3_1_inodelk_cbk] 2-vol-client-2: remote operation failed: Invalid argument [2012-01-10 16:52:30.437887] E [afr-lk-common.c:567:afr_unlock_inodelk_cbk] 2-vol-replicate-1: /system_light/run13879/fileop_L1_28/fileop_L1_28_L2_9/fileop_d ir_28_9_16: unlock failed on 0, reason: Invalid argument [2012-01-10 16:59:01.317059] W [socket.c:1510:__socket_proto_state_machine] 0-vol-client-0: reading from socket failed. Error (Transport endpoint is not conn ected), peer (127.0.1.1:24009) [2012-01-10 16:59:01.347869] I [client.c:1885:client_rpc_notify] 0-vol-client-0: disconnected [2012-01-10 16:59:01.347973] W [socket.c:1510:__socket_proto_state_machine] 1-vol-client-0: reading from socket failed. Error (Transport endpoint is not conn ected), peer (127.0.1.1:24009) [2012-01-10 16:59:01.348029] I [client.c:1885:client_rpc_notify] 1-vol-client-0: disconnected [2012-01-10 16:59:01.348056] W [socket.c:204:__socket_rwv] 2-vol-client-0: readv failed (Connection reset by peer) [2012-01-10 16:59:01.348073] W [socket.c:1510:__socket_proto_state_machine] 2-vol-client-0: reading from socket failed. Error (Connection reset by peer), pee r (127.0.1.1:24009) [2012-01-10 16:59:01.440543] E [rpc-clnt.c:380:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x16e) [0x7ffdd0b7878d] (-->/usr/local/l ib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x1c5) [0x7ffdd0b776e1] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x45) [0x7ffdd0b76e55]))) 2-vol-cl ient-0: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2012-01-10 16:59:01.292216 [2012-01-10 16:59:01.440624] W [client3_1-fops.c:1312:client3_1_entrylk_cbk] 2-vol-client-0: remote operation failed: Transport endpoint is not connected [2012-01-10 16:59:01.440718] E [rpc-clnt.c:380:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x16e) [0x7ffdd0b7878d] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x1c5) [0x7ffdd0b776e1] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x45) [0x7ffdd0b76e55]))) 2-vol-client-0: forced unwinding frame type(GlusterFS 3.1) op(STAT(1)) called at 2012-01-10 16:59:01.293537 [2012-01-10 16:59:01.440787] W [client3_1-fops.c:418:client3_1_stat_cbk] 2-vol-client-0: remote operation failed: Transport endpoint is not connected [2012-01-10 16:59:01.440921] E [rpc-clnt.c:380:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x16e) [0x7ffdd0b7878d] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x1c5) [0x7ffdd0b776e1] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x45) [0x7ffdd0b76e55]))) 2-vol-client-0: forced unwinding frame type(GlusterFS 3.1) op(RELEASEDIR(42)) called at 2012-01-10 16:59:01.293757 Steps to reproduce: Compile glusterfs with CFLAGS="-g3 -DDEBUG -lgcov --coverage" Run fileop a distributed-replicate volume. Usage: fileop -f 30 -t; Brick a brick down & then back up do "gluster volume heal vol" client log should show these errors.
*** Bug 787656 has been marked as a duplicate of this bug. ***
I am not seeing these errors anymore.
CHANGE: http://review.gluster.com/2993 (features/locks: Avoid race in creating domains) merged in master by Anand Avati (avati)
CHANGE: http://review.gluster.com/2998 (features/locks: Avoid race in pl_inode_get) merged in master by Anand Avati (avati)
Not seeing these anymore.