Hide Forgot
Description of problem: ======================= Found invalid arguments errors on geo-replication client logs and stale file handle warning in snapd lags. Uss is enabled on the volumes but no snapshot operations are performed. No lookup on .snaps is carried. These errors in geo-rep logs are ambiguous as no snapshot operation is performed by user. Geo-Replication client logs: ============================ [2016-04-13 18:24:58.299211] E [snapview-client.c:92:svc_inode_ctx_get] (-->/usr/lib64/glusterfs/3.7.9/xlator/performance/md-cache.so(mdc_lookup_cbk+0x12a) [0x7fa4031b4a3a] -->/usr/lib64/glusterfs/3.7.9/xlator/features/snapview-client.so(+0xa2ab) [0x7fa402fa62ab] -->/usr/lib64/glusterfs/3.7.9/xlator/features/snapview-client.so(svc_inode_ctx_get+0xa8) [0x7fa402f9ecd8] ) 0-vol0-snapview-client: invalid argument: inode [Invalid argument] [2016-04-13 18:24:58.302570] E [snapview-client.c:92:svc_inode_ctx_get] (-->/usr/lib64/glusterfs/3.7.9/xlator/performance/md-cache.so(mdc_lookup_cbk+0x12a) [0x7fa4031b4a3a] -->/usr/lib64/glusterfs/3.7.9/xlator/features/snapview-client.so(+0xa2ab) [0x7fa402fa62ab] -->/usr/lib64/glusterfs/3.7.9/xlator/features/snapview-client.so(svc_inode_ctx_get+0xa8) [0x7fa402f9ecd8] ) 0-vol0-snapview-client: invalid argument: inode [Invalid argument] [2016-04-13 18:24:58.310297] E [snapview-client.c:92:svc_inode_ctx_get] (-->/usr/lib64/glusterfs/3.7.9/xlator/performance/md-cache.so(mdc_lookup_cbk+0x12a) [0x7fa4031b4a3a] -->/usr/lib64/glusterfs/3.7.9/xlator/features/snapview-client.so(+0xa2ab) [0x7fa402fa62ab] -->/usr/lib64/glusterfs/3.7.9/xlator/features/snapview-client.so(svc_inode_ctx_get+0xa8) [0x7fa402f9ecd8] ) 0-vol0-snapview-client: invalid argument: inode [Invalid argument] Snapd logs: =========== [2016-04-13 18:24:58.333051] W [MSGID: 115005] [server-resolve.c:126:resolve_gfid_cbk] 0-vol0-server: 24455707-e959-470c-9ce8-169993fa89cd: failed to resolve (Stale file handle) [Stale file handle] The message "W [MSGID: 115005] [server-resolve.c:126:resolve_gfid_cbk] 0-vol0-server: 24455707-e959-470c-9ce8-169993fa89cd: failed to resolve (Stale file handle) [Stale file handle]" repeated 5 times between [2016-04-13 18:24:58.333051] and [2016-04-13 18:24:58.340688] [2016-04-13 18:24:58.345005] W [MSGID: 115005] [server-resolve.c:126:resolve_gfid_cbk] 0-vol0-server: 13bb5cca-f82b-4209-b404-0d1314859fb6: failed to resolve (Stale file handle) [Stale file handle] The message "W [MSGID: 115005] [server-resolve.c:126:resolve_gfid_cbk] 0-vol0-server: cc135b54-ad15-4a56-8037-923956a06d3f: failed to resolve (Stale file handle) [Stale file handle]" repeated 4 times between [2016-04-13 18:24:57.841812] and [2016-04-13 18:24:58.356258] [2016-04-13 18:24:58.359004] W [MSGID: 115005] [server-resolve.c:126:resolve_gfid_cbk] 0-vol0-server: eadff54f-6208-459b-877b-36eaab39deda: failed to resolve (Stale file handle) [Stale file handle] [2016-04-13 18:24:58.360232] W [MSGID: 115005] [server-resolve.c:126:resolve_gfid_cbk] 0-vol0-server: cc135b54-ad15-4a56-8037-923956a06d3f: failed to resolve (Stale file handle) [Stale file handle] [2016-04-13 18:24:58.361814] W [MSGID: 115005] [server-resolve.c:126:resolve_gfid_cbk] 0-vol0-server: eadff54f-6208-459b-877b-36eaab39deda: failed to resolve (Stale file handle) [Stale file handle] These errors are reported in huge numbers. Version-Release number of selected component (if applicable): ============================================================= glusterfs-3.7.9-1.el7rhgs.x86_64 How reproducible: ================= 2/2 Steps Carried: 1. Ran geo-replication automated cases once with root and once with non-root. 2. fops carried in order are create,chmod,chown,chgrp,symlink,hardlink,rename,truncate,remove
For reference: These errors are reported multiple times at slave log for every entry synced to slave when bricks on Master volume were offline. [root@dhcp37-52 geo-replication-slaves]# grep -ri "snapview-client.c:92:svc_inode_ctx_get" e4348759-bb69-415f-b8f7-94fbcb3a2533\:gluster%3A%2F%2F127.0.0.1%3ASlave.gluster.log | wc -l 1243 [root@dhcp37-52 geo-replication-slaves]#