Description of problem: Self-heal daemon is giving failure log messages because of invalid argument errors. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: [2012-02-07 16:19:46.543052] I [rpc-clnt.c:1604:rpc_clnt_reconfig] 0-mirror-client-0: changing port to 24009 (from 0) [2012-02-07 16:19:46.543123] I [rpc-clnt.c:1604:rpc_clnt_reconfig] 0-mirror-client-1: changing port to 24010 (from 0) [2012-02-07 16:19:47.145883] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 9 [2012-02-07 16:19:47.146006] W [socket.c:1874:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not suppor ted [2012-02-07 16:19:50.528078] I [client-handshake.c:1104:select_server_supported_programs] 0-mirror-client-0: Using Program GlusterFS 3git, Num (1298437), Version (330) [2012-02-07 16:19:50.528888] I [client-handshake.c:925:client_setvolume_cbk] 0-mirror-client-0: Connected to 127.0.0.1:24009, attached to remo te volume '/mnt/sda7/export3'. [2012-02-07 16:19:50.528978] I [afr-common.c:3455:afr_notify] 0-mirror-replicate-0: Subvolume 'mirror-client-0' came back up; going online. [2012-02-07 16:19:50.529073] I [client-handshake.c:1104:select_server_supported_programs] 0-mirror-client-1: Using Program GlusterFS 3git, Num (1298437), Version (330) [2012-02-07 16:19:50.529599] I [client-handshake.c:925:client_setvolume_cbk] 0-mirror-client-1: Connected to 127.0.0.1:24010, attached to remo te volume '/mnt/sda8/export3'. [2012-02-07 16:19:50.529659] I [afr-common.c:3459:afr_notify] 0-mirror-replicate-0: subvol 1 came up, start crawl [2012-02-07 16:19:50.529703] I [afr-common.c:3554:afr_notify] 0-mirror-replicate-0: All subvolumes came up, start crawl [2012-02-07 16:19:50.529757] I [afr-self-heald.c:460:afr_proactive_self_heal] 0-mirror-replicate-0: starting crawl for -1 [2012-02-07 16:19:50.545835] I [afr-self-heald.c:270:afr_find_child_position] 0-mirror-replicate-0: child 0 is 1 [2012-02-07 16:19:50.546687] I [afr-self-heald.c:270:afr_find_child_position] 0-mirror-replicate-0: child 1 is 1 [2012-02-07 16:19:50.547633] I [afr-common.c:1825:afr_set_root_inode_on_first_lookup] 0-mirror-replicate-0: added root inode [2012-02-07 16:19:50.547955] I [afr-common.c:1288:afr_launch_self_heal] 0-mirror-replicate-0: background entry self-heal triggered. path: /, reason: lookup detected pending operations [2012-02-07 16:19:50.552187] I [afr-self-heal-entry.c:647:afr_sh_entry_expunge_entry_cbk] 0-mirror-replicate-0: missing entry /fstest_6803ea15 5f2f482962eebc72217c2a20 on mirror-client-1 [2012-02-07 16:19:50.560520] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /fstest_66acf28ffb9c6 e83dadf07a563b4bd01 on subvolume mirror-client-0 => -1 (No such file or directory) [2012-02-07 16:19:50.560886] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /fstest_c357f34681b67 76ff82e63ec2cbd3ee4 on subvolume mirror-client-0 => -1 (No such file or directory) [2012-02-07 16:19:50.563727] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /fstest_c46f7c85509bc 63b75f002c2e93dd079 on subvolume mirror-client-0 => -1 (No such file or directory) [2012-02-07 16:19:50.563823] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /fstest_cc08dfa4f7797 6bf340464e0b29f16f9 on subvolume mirror-client-0 => -1 (No such file or directory) [2012-02-07 16:19:50.563901] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /fstest_9c2b111c4d420 352e72609992217aa67 on subvolume mirror-client-0 => -1 (No such file or directory) [2012-02-07 16:19:50.564018] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /fstest_f67022f6cacde 0254c9ca70b03b1acb1 on subvolume mirror-client-0 => -1 (No such file or directory) [2012-02-07 16:19:50.564129] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /fstest_f015fc76d6fd5 26f2a0fda817318a324 on subvolume mirror-client-0 => -1 (No such file or directory) [2012-02-07 16:19:50.564245] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /clients on subvolume mirror-client-0 => -1 (No such file or directory) [2012-02-07 16:19:50.564376] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /fstest_2e198c0986e6d 8fc4a3c9e97ed9c6ace on subvolume mirror-client-0 => -1 (No such file or directory) [2012-02-07 16:19:50.569703] I [afr-self-heal-common.c:2022:afr_self_heal_completion_cbk] 0-mirror-replicate-0: background entry self-heal co mpleted on / [2012-02-07 16:19:50.571964] I [afr-common.c:1288:afr_launch_self_heal] 0-mirror-replicate-0: background meta-data entry self-heal triggered. path: /fstest_66acf28ffb9c6e83dadf07a563b4bd01, reason: lookup detected pending operations [2012-02-07 16:19:50.575342] I [afr-self-heal-common.c:2022:afr_self_heal_completion_cbk] 0-mirror-replicate-0: background meta-data entry self-heal completed on /fstest_66acf28ffb9c6e83dadf07a563b4bd01 [2012-02-07 16:19:50.576907] I [afr-common.c:1288:afr_launch_self_heal] 0-mirror-replicate-0: background meta-data entry self-heal triggered. path: /fstest_c357f34681b6776ff82e63ec2cbd3ee4, reason: lookup detected pending operations [2012-02-07 16:19:50.580991] I [afr-self-heal-common.c:2022:afr_self_heal_completion_cbk] 0-mirror-replicate-0: background meta-data entry self-heal completed on /fstest_c357f34681b6776ff82e63ec2cbd3ee4 [2012-02-07 16:19:50.582378] I [afr-common.c:1288:afr_launch_self_heal] 0-mirror-replicate-0: background meta-data entry self-heal triggered. path: /fstest_c46f7c85509bc63b75f002c2e93dd079, reason: lookup detected pending operations [2012-02-07 16:19:50.585625] I [afr-self-heal-common.c:2022:afr_self_heal_completion_cbk] 0-mirror-replicate-0: background meta-data entry self-heal completed on /fstest_c46f7c85509bc63b75f002c2e93dd079 [2012-02-07 16:19:50.586952] I [afr-common.c:1288:afr_launch_self_heal] 0-mirror-replicate-0: background meta-data entry self-heal triggered. path: /fstest_cc08dfa4f77976bf340464e0b29f16f9, reason: lookup detected pending operations [2012-02-07 16:19:50.590271] I [afr-self-heal-common.c:2022:afr_self_heal_completion_cbk] 0-mirror-replicate-0: background meta-data entry self-heal completed on /fstest_cc08dfa4f77976bf340464e0b29f16f9 [2012-02-07 16:19:50.591937] I [afr-common.c:1288:afr_launch_self_heal] 0-mirror-replicate-0: background meta-data entry self-heal triggered. path: /fstest_9c2b111c4d420352e72609992217aa67, reason: lookup detected pending operations [2012-02-07 16:19:50.597015] I [afr-self-heal-common.c:2022:afr_self_heal_completion_cbk] 0-mirror-replicate-0: background meta-data entry self-heal completed on /fstest_9c2b111c4d420352e72609992217aa67 [2012-02-07 16:19:50.599013] I [afr-common.c:1288:afr_launch_self_heal] 0-mirror-replicate-0: background meta-data entry self-heal triggered. path: /fstest_f67022f6cacde0254c9ca70b03b1acb1, reason: lookup detected pending operations [2012-02-07 16:19:50.604245] I [afr-self-heal-common.c:2022:afr_self_heal_completion_cbk] 0-mirror-replicate-0: background meta-data entry self-heal completed on /fstest_f67022f6cacde0254c9ca70b03b1acb1 [2012-02-07 16:19:50.607108] I [afr-common.c:1288:afr_launch_self_heal] 0-mirror-replicate-0: background meta-data entry self-heal triggered. path: /fstest_f015fc76d6fd526f2a0fda817318a324, reason: lookup detected pending operations [2012-02-07 16:19:50.613933] I [afr-self-heal-common.c:2022:afr_self_heal_completion_cbk] 0-mirror-replicate-0: background meta-data entry self-heal completed on /fstest_f015fc76d6fd526f2a0fda817318a324 [2012-02-07 16:19:50.617238] I [afr-common.c:1288:afr_launch_self_heal] 0-mirror-replicate-0: background meta-data entry self-heal triggered. path: /clients, reason: lookup detected pending operations [2012-02-07 16:19:50.630993] W [client3_1-fops.c:2287:client3_1_lookup_cbk] 0-mirror-client-0: remote operation failed: Invalid argument. Path: /clients/client1 [2012-02-07 16:19:50.631079] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /clients/client1 on subvolume mirror-client-0 => -1 (Invalid argument) [2012-02-07 16:19:50.633192] W [client3_1-fops.c:2287:client3_1_lookup_cbk] 0-mirror-client-0: remote operation failed: Invalid argument. Path: /clients/client9 [2012-02-07 16:19:50.633240] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /clients/client9 on subvolume mirror-client-0 => -1 (Invalid argument) [2012-02-07 16:19:50.633307] W [client3_1-fops.c:2287:client3_1_lookup_cbk] 0-mirror-client-1: remote operation failed: Invalid argument. Path: /clients/client1 [2012-02-07 16:19:50.633439] E [afr-self-heal-common.c:998:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /clients/client1 on subvolume mirror-client-1 => -1 (Invalid argument) [2012-02-07 16:19:50.633486] E [afr-self-heal-common.c:1275:afr_sh_common_lookup_cbk] 0-mirror-replicate-0: Failed to lookup /clients/client1,: Expected results: Additional info:
Not able to reproduce this bug. Johnny can you try once.
CHANGE: http://review.gluster.com/3137 (libglusterfs: loc_copy should set name only if src->name is set) merged in master by Anand Avati (avati)
Steps to recreate the bug:- -------------------------- 1) create a replicate volume (1X3).start the volume 2) create fuse mount 3) from the fuse mount, create a directory <dir1> and create 10 files in it. 4) unmount the fuse mount 5) delete 5 files from the backend on one of the bricks 6) create fuse mount 7) cd <dir1> ; execute 'ls'
Checked with glusterfs-3.3.0qa40 and its fixed now. Followed the steps and did not get Invalid argument errors in log. [2012-05-08 18:42:04.451291] I [afr-common.c:1347:afr_launch_self_heal] 0-mirror-replicate-0: entry self-heal triggered. path: /dir, reason: checksums of directory differ [2012-05-08 18:42:04.476583] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /dir/9 on subvolume mirror-client-0 => -1 (No such file or directory) [2012-05-08 18:42:04.507030] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /dir/7 on subvolume mirror-client-0 => -1 (No such file or directory) [2012-05-08 18:42:04.526571] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /dir/8 on subvolume mirror-client-0 => -1 (No such file or directory) [2012-05-08 18:42:04.526772] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /dir/1 on subvolume mirror-client-0 => -1 (No such file or directory) [2012-05-08 18:42:04.526920] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-mirror-replicate-0: path /dir/5 on subvolume mirror-client-0 => -1 (No such file or directory) [2012-05-08 18:42:04.563085] I [afr-self-heal-common.c:2054:afr_self_heal_completion_cbk] 0-mirror-replicate-0: background entry self-heal completed on /dir