Bug 770080 - [glusterfs-3.3.0qa18]: errors like "unlock failed on 0, reason: Invalid argument" in client
Summary: [glusterfs-3.3.0qa18]: errors like "unlock failed on 0, reason: Invalid argum...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: replicate
Version: 3.3-beta
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
Assignee: Pranith Kumar K
QA Contact:
URL:
Whiteboard:
: 787656 (view as bug list)
Depends On:
Blocks: 817967
TreeView+ depends on / blocked
 
Reported: 2011-12-23 10:16 UTC by Rahul C S
Modified: 2013-07-24 17:33 UTC (History)
3 users (show)

Fixed In Version: glusterfs-3.4.0
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-07-24 17:33:33 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions: 1f3a0dd4742a2fcd3215aee4a5e22125d7ea4f4d
Embargoed:


Attachments (Terms of Use)

Description Rahul C S 2011-12-23 10:16:24 UTC
Description of problem:
I compiled glusterfs with CFLAGS="-g3 -DDEBUG -lgcov --coverage"

Was running sanity on a distributed replicate volume, with quota & geo-rep enabled. then I also set limit-usage of 10GB. These errors happened after brick down/up was done on multiple bricks.

I was running sanity on 1 client, while running fileop, i got these errors.

[2011-12-23 02:03:30.084334] W [client3_1-fops.c:1273:client3_1_finodelk_cbk] 2-vol-client-2: remote operation failed: Invalid argument
[2011-12-23 02:03:30.084373] E [afr-lk-common.c:567:afr_unlock_inodelk_cbk] 2-vol-replicate-1: /system_light/run25721/dir1/dir5/file514: unlock failed on 0,
reason: Invalid argument
[2011-12-23 02:04:13.177107] W [client3_1-fops.c:1273:client3_1_finodelk_cbk] 2-vol-client-4: remote operation failed: Invalid argument
[2011-12-23 02:04:13.177141] E [afr-lk-common.c:567:afr_unlock_inodelk_cbk] 2-vol-replicate-2: /system_light/run25721/dir2/dir5/file53: unlock failed on 0, r
eason: Invalid argument
[2011-12-23 02:04:42.954784] W [client3_1-fops.c:1273:client3_1_finodelk_cbk] 2-vol-client-4: remote operation failed: Invalid argument
[2011-12-23 02:04:42.954816] E [afr-lk-common.c:567:afr_unlock_inodelk_cbk] 2-vol-replicate-2: /system_light/run25721/dir1/dir16/file1619: unlock failed on 0
, reason: Invalid argument
[2011-12-23 02:05:31.524251] W [client3_1-fops.c:1273:client3_1_finodelk_cbk] 2-vol-client-0: remote operation failed: Invalid argument
[2011-12-23 02:05:31.524285] E [afr-lk-common.c:567:afr_unlock_inodelk_cbk] 2-vol-replicate-0: /system_light/run25721/dir1/dir18/file180: unlock failed on 0,
 reason: Invalid argument
[2011-12-23 02:05:39.144078] W [client3_1-fops.c:1273:client3_1_finodelk_cbk] 2-vol-client-2: remote operation failed: Invalid argument
[2011-12-23 02:05:39.144131] E [afr-lk-common.c:567:afr_unlock_inodelk_cbk] 2-vol-replicate-1: /system_light/run25721/dir2/dir1/file18: unlock failed on 0, r
eason: Invalid argument


P.S: I have provided the logs to pranith, not tried more times, just wanted to log incase it was a bug.
Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 1 Rahul C S 2012-01-10 11:47:19 UTC
Happens with qa19 as well.

client logs:
[2012-01-10 16:49:30.333224] E [afr-lk-common.c:567:afr_unlock_inodelk_cbk] 2-vol-replicate-0: /system_light/run13879/fileop_L1_10/fileop_L1_10_L2_5/fileop_d
ir_10_5_29: unlock failed on 0, reason: Invalid argument
[2012-01-10 16:50:31.080786] W [client3_1-fops.c:1235:client3_1_inodelk_cbk] 2-vol-client-2: remote operation failed: Invalid argument
[2012-01-10 16:50:31.080818] E [afr-lk-common.c:567:afr_unlock_inodelk_cbk] 2-vol-replicate-1: /system_light/run13879/fileop_L1_16/fileop_L1_16_L2_16/fileop_
dir_16_16_9: unlock failed on 0, reason: Invalid argument
[2012-01-10 16:50:32.663965] W [client3_1-fops.c:1235:client3_1_inodelk_cbk] 2-vol-client-0: remote operation failed: Invalid argument
[2012-01-10 16:50:32.663992] E [afr-lk-common.c:567:afr_unlock_inodelk_cbk] 2-vol-replicate-0: /system_light/run13879/fileop_L1_16/fileop_L1_16_L2_24/fileop_
dir_16_24_7: unlock failed on 0, reason: Invalid argument
[2012-01-10 16:51:16.024236] W [client3_1-fops.c:1235:client3_1_inodelk_cbk] 2-vol-client-0: remote operation failed: Invalid argument
[2012-01-10 16:51:16.024264] E [afr-lk-common.c:567:afr_unlock_inodelk_cbk] 2-vol-replicate-0: /system_light/run13879/fileop_L1_21/fileop_L1_21_L2_12/fileop_
dir_21_12_22: unlock failed on 0, reason: Invalid argument
[2012-01-10 16:52:30.437856] W [client3_1-fops.c:1235:client3_1_inodelk_cbk] 2-vol-client-2: remote operation failed: Invalid argument
[2012-01-10 16:52:30.437887] E [afr-lk-common.c:567:afr_unlock_inodelk_cbk] 2-vol-replicate-1: /system_light/run13879/fileop_L1_28/fileop_L1_28_L2_9/fileop_d
ir_28_9_16: unlock failed on 0, reason: Invalid argument
[2012-01-10 16:59:01.317059] W [socket.c:1510:__socket_proto_state_machine] 0-vol-client-0: reading from socket failed. Error (Transport endpoint is not conn
ected), peer (127.0.1.1:24009)
[2012-01-10 16:59:01.347869] I [client.c:1885:client_rpc_notify] 0-vol-client-0: disconnected
[2012-01-10 16:59:01.347973] W [socket.c:1510:__socket_proto_state_machine] 1-vol-client-0: reading from socket failed. Error (Transport endpoint is not conn
ected), peer (127.0.1.1:24009)
[2012-01-10 16:59:01.348029] I [client.c:1885:client_rpc_notify] 1-vol-client-0: disconnected
[2012-01-10 16:59:01.348056] W [socket.c:204:__socket_rwv] 2-vol-client-0: readv failed (Connection reset by peer)
[2012-01-10 16:59:01.348073] W [socket.c:1510:__socket_proto_state_machine] 2-vol-client-0: reading from socket failed. Error (Connection reset by peer), pee
r (127.0.1.1:24009)
[2012-01-10 16:59:01.440543] E [rpc-clnt.c:380:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x16e) [0x7ffdd0b7878d] (-->/usr/local/l
ib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x1c5) [0x7ffdd0b776e1] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x45) [0x7ffdd0b76e55]))) 2-vol-cl
ient-0: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2012-01-10 16:59:01.292216
[2012-01-10 16:59:01.440624] W [client3_1-fops.c:1312:client3_1_entrylk_cbk] 2-vol-client-0: remote operation failed: Transport endpoint is not connected
[2012-01-10 16:59:01.440718] E [rpc-clnt.c:380:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x16e) [0x7ffdd0b7878d] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x1c5) [0x7ffdd0b776e1] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x45) [0x7ffdd0b76e55]))) 2-vol-client-0: forced unwinding frame type(GlusterFS 3.1) op(STAT(1)) called at 2012-01-10 16:59:01.293537
[2012-01-10 16:59:01.440787] W [client3_1-fops.c:418:client3_1_stat_cbk] 2-vol-client-0: remote operation failed: Transport endpoint is not connected
[2012-01-10 16:59:01.440921] E [rpc-clnt.c:380:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x16e) [0x7ffdd0b7878d] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x1c5) [0x7ffdd0b776e1] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x45) [0x7ffdd0b76e55]))) 2-vol-client-0: forced unwinding frame type(GlusterFS 3.1) op(RELEASEDIR(42)) called at 2012-01-10 16:59:01.293757

Steps to reproduce:
Compile glusterfs with CFLAGS="-g3 -DDEBUG -lgcov --coverage"
Run fileop a distributed-replicate volume. Usage: fileop -f 30 -t;
Brick a brick down & then back up
do "gluster volume heal vol"
client log should show these errors.

Comment 2 Pranith Kumar K 2012-02-07 06:19:25 UTC
*** Bug 787656 has been marked as a duplicate of this bug. ***

Comment 3 Pranith Kumar K 2012-02-22 13:08:01 UTC
I am not seeing these errors anymore.

Comment 4 Anand Avati 2012-03-21 18:15:22 UTC
CHANGE: http://review.gluster.com/2993 (features/locks: Avoid race in creating domains) merged in master by Anand Avati (avati)

Comment 5 Anand Avati 2012-03-21 18:16:24 UTC
CHANGE: http://review.gluster.com/2998 (features/locks: Avoid race in pl_inode_get) merged in master by Anand Avati (avati)

Comment 6 Rahul C S 2012-04-05 08:11:38 UTC
Not seeing these anymore.


Note You need to log in before you can comment on or make changes to this bug.