Bug 788051

Summary: [b708b18b833d1f2ba4da394884bc762a821ff56b]: self-heal failing due to invalid arguement
Product: [Community] GlusterFS Reporter: Raghavendra Bhat <rabhat>
Component: replicateAssignee: Pranith Kumar K <pkarampu>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: high    
Version: mainlineCC: gluster-bugs, shwetha.h.panduranga
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.4.0 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-07-24 17:43:59 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: glusterfs-3.3.0qa40 Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 817967    

Description Raghavendra Bhat 2012-02-07 10:52:48 UTC 
Description of problem:
Self-heal daemon is giving failure log messages because of invalid argument errors.



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:
[2012-02-07 16:19:46.543052] I [rpc-clnt.c:1604:rpc_clnt_reconfig] 0-mirror-client-0: changing port to 24009 (from 0)
[2012-02-07 16:19:46.543123] I [rpc-clnt.c:1604:rpc_clnt_reconfig] 0-mirror-client-1: changing port to 24010 (from 0)
[2012-02-07 16:19:47.145883] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 9
[2012-02-07 16:19:47.146006] W [socket.c:1874:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not suppor
ted
[2012-02-07 16:19:50.528078] I [client-handshake.c:1104:select_server_supported_programs] 0-mirror-client-0: Using Program GlusterFS 3git, Num
 (1298437), Version (330)
[2012-02-07 16:19:50.528888] I [client-handshake.c:925:client_setvolume_cbk] 0-mirror-client-0: Connected to 127.0.0.1:24009, attached to remo
te volume '/mnt/sda7/export3'.
[2012-02-07 16:19:50.528978] I [afr-common.c:3455:afr_notify] 0-mirror-replicate-0: Subvolume 'mirror-client-0' came back up; going online.
[2012-02-07 16:19:50.529073] I [client-handshake.c:1104:select_server_supported_programs] 0-mirror-client-1: Using Program GlusterFS 3git, Num
 (1298437), Version (330)
[2012-02-07 16:19:50.529599] I [client-handshake.c:925:client_setvolume_cbk] 0-mirror-client-1: Connected to 127.0.0.1:24010, attached to remo
te volume '/mnt/sda8/export3'.
[2012-02-07 16:19:50.529659] I [afr-common.c:3459:afr_notify] 0-mirror-replicate-0: subvol 1 came up, start crawl
[2012-02-07 16:19:50.529703] I [afr-common.c:3554:afr_notify] 0-mirror-replicate-0: All subvolumes came up, start crawl
[2012-02-07 16:19:50.529757] I [afr-self-heald.c:460:afr_proactive_self_heal] 0-mirror-replicate-0: starting crawl for -1
[2012-02-07 16:19:50.545835] I [afr-self-heald.c:270:afr_find_child_position] 0-mirror-replicate-0: child 0 is 1
[2012-02-07 16:19:50.546687] I [afr-self-heald.c:270:afr_find_child_position] 0-mirror-replicate-0: child 1 is 1
[2012-02-07 16:19:50.547633] I [afr-common.c:1825:afr_set_root_inode_on_first_lookup] 0-mirror-replicate-0: added root inode
[2012-02-07 16:19:50.547955] I [afr-common.c:1288:afr_launch_self_heal] 0-mirror-replicate-0: background  entry self-heal triggered. path: /, 
reason: lookup detected pending operations
[2012-02-07 16:19:50.552187] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-mirror-replicate-0: missing entry /fstest_6803ea15
5f2f482962eebc72217c2a20 on mirror-client-1
[2012-02-07 16:19:50.560520] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /fstest_66acf28ffb9c6
e83dadf07a563b4bd01 on subvolume mirror-client-0 => -1 (No such file or directory)
[2012-02-07 16:19:50.560886] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /fstest_c357f34681b67
76ff82e63ec2cbd3ee4 on subvolume mirror-client-0 => -1 (No such file or directory)
[2012-02-07 16:19:50.563727] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /fstest_c46f7c85509bc
63b75f002c2e93dd079 on subvolume mirror-client-0 => -1 (No such file or directory)
[2012-02-07 16:19:50.563823] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /fstest_cc08dfa4f7797
6bf340464e0b29f16f9 on subvolume mirror-client-0 => -1 (No such file or directory)
[2012-02-07 16:19:50.563901] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /fstest_9c2b111c4d420
352e72609992217aa67 on subvolume mirror-client-0 => -1 (No such file or directory)
[2012-02-07 16:19:50.564018] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /fstest_f67022f6cacde
0254c9ca70b03b1acb1 on subvolume mirror-client-0 => -1 (No such file or directory)
[2012-02-07 16:19:50.564129] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /fstest_f015fc76d6fd5
26f2a0fda817318a324 on subvolume mirror-client-0 => -1 (No such file or directory)
[2012-02-07 16:19:50.564245] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /clients on subvolume
 mirror-client-0 => -1 (No such file or directory)
[2012-02-07 16:19:50.564376] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /fstest_2e198c0986e6d
8fc4a3c9e97ed9c6ace on subvolume mirror-client-0 => -1 (No such file or directory)
[2012-02-07 16:19:50.569703] I [afr-self-heal-common.c:2022:afr_self_heal_completion_cbk] 0-mirror-replicate-0: background  entry self-heal co
mpleted on /
[2012-02-07 16:19:50.571964] I [afr-common.c:1288:afr_launch_self_heal] 0-mirror-replicate-0: background  meta-data entry self-heal triggered. path: /fstest_66acf28ffb9c6e83dadf07a563b4bd01, reason: lookup detected pending operations
[2012-02-07 16:19:50.575342] I [afr-self-heal-common.c:2022:afr_self_heal_completion_cbk] 0-mirror-replicate-0: background  meta-data entry self-heal completed on /fstest_66acf28ffb9c6e83dadf07a563b4bd01
[2012-02-07 16:19:50.576907] I [afr-common.c:1288:afr_launch_self_heal] 0-mirror-replicate-0: background  meta-data entry self-heal triggered. path: /fstest_c357f34681b6776ff82e63ec2cbd3ee4, reason: lookup detected pending operations
[2012-02-07 16:19:50.580991] I [afr-self-heal-common.c:2022:afr_self_heal_completion_cbk] 0-mirror-replicate-0: background  meta-data entry self-heal completed on /fstest_c357f34681b6776ff82e63ec2cbd3ee4
[2012-02-07 16:19:50.582378] I [afr-common.c:1288:afr_launch_self_heal] 0-mirror-replicate-0: background  meta-data entry self-heal triggered. path: /fstest_c46f7c85509bc63b75f002c2e93dd079, reason: lookup detected pending operations
[2012-02-07 16:19:50.585625] I [afr-self-heal-common.c:2022:afr_self_heal_completion_cbk] 0-mirror-replicate-0: background  meta-data entry self-heal completed on /fstest_c46f7c85509bc63b75f002c2e93dd079
[2012-02-07 16:19:50.586952] I [afr-common.c:1288:afr_launch_self_heal] 0-mirror-replicate-0: background  meta-data entry self-heal triggered. path: /fstest_cc08dfa4f77976bf340464e0b29f16f9, reason: lookup detected pending operations
[2012-02-07 16:19:50.590271] I [afr-self-heal-common.c:2022:afr_self_heal_completion_cbk] 0-mirror-replicate-0: background  meta-data entry self-heal completed on /fstest_cc08dfa4f77976bf340464e0b29f16f9
[2012-02-07 16:19:50.591937] I [afr-common.c:1288:afr_launch_self_heal] 0-mirror-replicate-0: background  meta-data entry self-heal triggered. path: /fstest_9c2b111c4d420352e72609992217aa67, reason: lookup detected pending operations
[2012-02-07 16:19:50.597015] I [afr-self-heal-common.c:2022:afr_self_heal_completion_cbk] 0-mirror-replicate-0: background  meta-data entry self-heal completed on /fstest_9c2b111c4d420352e72609992217aa67
[2012-02-07 16:19:50.599013] I [afr-common.c:1288:afr_launch_self_heal] 0-mirror-replicate-0: background  meta-data entry self-heal triggered. path: /fstest_f67022f6cacde0254c9ca70b03b1acb1, reason: lookup detected pending operations
[2012-02-07 16:19:50.604245] I [afr-self-heal-common.c:2022:afr_self_heal_completion_cbk] 0-mirror-replicate-0: background  meta-data entry self-heal completed on /fstest_f67022f6cacde0254c9ca70b03b1acb1
[2012-02-07 16:19:50.607108] I [afr-common.c:1288:afr_launch_self_heal] 0-mirror-replicate-0: background  meta-data entry self-heal triggered. path: /fstest_f015fc76d6fd526f2a0fda817318a324, reason: lookup detected pending operations
[2012-02-07 16:19:50.613933] I [afr-self-heal-common.c:2022:afr_self_heal_completion_cbk] 0-mirror-replicate-0: background  meta-data entry self-heal completed on /fstest_f015fc76d6fd526f2a0fda817318a324
[2012-02-07 16:19:50.617238] I [afr-common.c:1288:afr_launch_self_heal] 0-mirror-replicate-0: background  meta-data entry self-heal triggered. path: /clients, reason: lookup detected pending operations
[2012-02-07 16:19:50.630993] W [client3_1-fops.c:2287:client3_1_lookup_cbk] 0-mirror-client-0: remote operation failed: Invalid argument. Path: /clients/client1
[2012-02-07 16:19:50.631079] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /clients/client1 on subvolume mirror-client-0 => -1 (Invalid argument)
[2012-02-07 16:19:50.633192] W [client3_1-fops.c:2287:client3_1_lookup_cbk] 0-mirror-client-0: remote operation failed: Invalid argument. Path: /clients/client9
[2012-02-07 16:19:50.633240] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /clients/client9 on subvolume mirror-client-0 => -1 (Invalid argument)
[2012-02-07 16:19:50.633307] W [client3_1-fops.c:2287:client3_1_lookup_cbk] 0-mirror-client-1: remote operation failed: Invalid argument. Path: /clients/client1
[2012-02-07 16:19:50.633439] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /clients/client1 on subvolume mirror-client-1 => -1 (Invalid argument)
[2012-02-07 16:19:50.633486] E [afr-self-heal-common.c:1275:afr_sh_common_lookup_cbk] 0-mirror-replicate-0: Failed to lookup /clients/client1,:


Expected results:


Additional info:
Comment 1 Pranith Kumar K 2012-02-22 13:08:51 UTC 
Not able to reproduce this bug. Johnny can you try once.
Comment 2 Anand Avati 2012-04-12 22:08:34 UTC 
CHANGE: http://review.gluster.com/3137 (libglusterfs: loc_copy should set name only if src->name is set) merged in master by Anand Avati (avati)
Comment 3 Shwetha Panduranga 2012-04-13 10:16:25 UTC 
Steps to recreate the bug:-
--------------------------

1) create a replicate volume (1X3).start the volume
2) create fuse mount
3) from the fuse mount, create a directory <dir1> and create 10 files in it.
4) unmount the fuse mount
5) delete 5 files from the backend on one of the bricks
6) create fuse mount
7) cd <dir1> ; execute 'ls'
Comment 4 Raghavendra Bhat 2012-05-08 13:13:11 UTC 
Checked with glusterfs-3.3.0qa40 and its fixed now. Followed the steps and did not get Invalid argument errors in log.

[2012-05-08 18:42:04.451291] I [afr-common.c:1347:afr_launch_self_heal] 0-mirror-replicate-0:   entry self-heal triggered. path: /dir, reason: checksums of directory differ
[2012-05-08 18:42:04.476583] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /dir/9 on subvolume mirror-client-0 => -1 (No such file or directory)
[2012-05-08 18:42:04.507030] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /dir/7 on subvolume mirror-client-0 => -1 (No such file or directory)
[2012-05-08 18:42:04.526571] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /dir/8 on subvolume mirror-client-0 => -1 (No such file or directory)
[2012-05-08 18:42:04.526772] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /dir/1 on subvolume mirror-client-0 => -1 (No such file or directory)
[2012-05-08 18:42:04.526920] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /dir/5 on subvolume mirror-client-0 => -1 (No such file or directory)
[2012-05-08 18:42:04.563085] I [afr-self-heal-common.c:2054:afr_self_heal_completion_cbk] 0-mirror-replicate-0: background  entry self-heal completed on /dir