Hide Forgot
oops.. pressed enter before adding contents On Mon, Mar 7, 2011 at 5:16 PM, Rahul C S <rahulcbhat> wrote: Hi Amar, Distributed-replicate setup, ran dbench -t 200 250 iozone -i 0 -i 1 -i 2 -i 3 -i 4 -i 5 -i 6 -i 7 -i 8 -i 9 -i 10 -i 11 -i 12 -s 100m -r 22k on both fuse and nfs clients Also over the weekend on fuse mount, i ran the /opt/qa/tools/kernel_compile.sh script From today morning detecting this in the brick log: [2011-03-06 20:15:06.306354] C [inode.c:218:__is_dentry_cyclic] /export/inode: detected cyclic loop formation during inode linkage. inode (1/00000000-0000-0000-0000-000000000001) linking under itself as vol Following is the output on the client. [root@centos-qa-client-3 fuse]# ls -l total 8 drwxrwxr-x 5 root root 4096 Mar 6 19:57 linux-2.6.31.1 ?--------- ? ? ? ? ? rep ?--------- ? ? ? ? ? vol [root@centos-qa-client-3 nfs]# ls -l total 319464 -rw-r--r-- 1 root root 326762496 Mar 6 20:03 heavyf drwxrwxr-x 5 root root 8192 Mar 6 19:57 linux-2.6.31.1 drwxr-xr-x 5 root root 4096 Mar 6 20:03 rep drwxr-xr-x 2 root root 8192 Mar 6 14:24 vol [root@centos-qa-client-3 nfs]# rm -rf * rm: WARNING: Circular directory structure. This almost certainly means that you have a corrupted file system. NOTIFY YOUR SYSTEM MANAGER. The following directory is part of the cycle: `rep/vol' rm: WARNING: Circular directory structure. This almost certainly means that you have a corrupted file system. NOTIFY YOUR SYSTEM MANAGER. The following directory is part of the cycle: `rep/vol/vol/vol' Logs: [2011-03-06 20:02:41.530062] D [afr-common.c:555:afr_lookup_collect_xattr] fore-replicate-0: metadata self-heal is pending for /. [2011-03-06 20:02:41.530380] D [afr-common.c:555:afr_lookup_collect_xattr] fore-replicate-0: metadata self-heal is pending for /. [2011-03-06 20:02:41.530436] I [afr-common.c:672:afr_lookup_done] fore-replicate-0: split brain detected during lookup of /. [2011-03-06 20:02:41.530456] I [afr-common.c:716:afr_lookup_done] fore-replicate-0: background meta-data data self-heal triggered. path: / [2011-03-06 20:02:41.530963] D [afr-self-heal-metadata.c:682:afr_sh_post_nonblocking_inodelk_cbk] fore-replicate-0: Non Blocking inodelks done. Proceeding to FOP [2011-03-06 20:02:41.531503] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] fore-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes [2011-03-06 20:02:41.531537] D [afr-lk-common.c:415:transaction_lk_op] fore-replicate-0: lk op is for a self heal [2011-03-06 20:02:41.531949] D [afr-self-heal-metadata.c:81:afr_sh_metadata_done] fore-replicate-0: aborting selfheal of / [2011-03-06 20:02:41.532006] I [afr-self-heal-common.c:1527:afr_self_heal_completion_cbk] fore-replicate-0: background meta-data data self-heal completed on / [2011-03-06 20:02:41.533937] D [afr-common.c:555:afr_lookup_collect_xattr] fore-replicate-0: metadata self-heal is pending for /. [2011-03-06 20:02:41.534119] D [afr-common.c:555:afr_lookup_collect_xattr] fore-replicate-0: metadata self-heal is pending for /. [2011-03-06 20:02:41.534145] I [afr-common.c:672:afr_lookup_done] fore-replicate-0: split brain detected during lookup of /. [2011-03-06 20:02:41.534168] I [afr-common.c:716:afr_lookup_done] fore-replicate-0: background meta-data data self-heal triggered. path: / [2011-03-06 20:02:41.534511] D [afr-self-heal-metadata.c:682:afr_sh_post_nonblocking_inodelk_cbk] fore-replicate-0: Non Blocking inodelks done. Proceeding to FOP [2011-03-06 20:02:41.534921] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] fore-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes [2011-03-06 20:02:41.534951] D [afr-lk-common.c:415:transaction_lk_op] fore-replicate-0: lk op is for a self heal [2011-03-06 20:02:41.535276] D [afr-self-heal-metadata.c:81:afr_sh_metadata_done] fore-replicate-0: aborting selfheal of / [2011-03-06 20:02:41.535301] I [afr-self-heal-common.c:1527:afr_self_heal_completion_cbk] fore-replicate-0: background meta-data data self-heal completed on / [2011-03-06 20:02:41.537830] D [dht-layout.c:584:dht_layout_normalize] fore-dht: directory /vol looked up first time [2011-03-06 20:02:41.537853] D [dht-common.c:172:dht_lookup_dir_cbk] fore-dht: fixing assignment on /vol [2011-03-06 20:02:41.538841] D [afr-lk-common.c:410:transaction_lk_op] fore-replicate-0: lk op is for a transaction [2011-03-06 20:02:41.538871] D [afr-transaction.c:973:afr_post_nonblocking_inodelk_cbk] fore-replicate-0: Non blocking inodelks failed. Proceeding to blocking [2011-03-06 20:02:41.539046] D [afr-lk-common.c:410:transaction_lk_op] fore-replicate-1: lk op is for a transaction [2011-03-06 20:02:41.539075] D [afr-transaction.c:973:afr_post_nonblocking_inodelk_cbk] fore-replicate-1: Non blocking inodelks failed. Proceeding to blocking [2011-03-06 20:02:41.540148] D [afr-lk-common.c:964:afr_lock_blocking] fore-replicate-0: unable to lock on even one child [2011-03-06 20:02:41.540180] D [afr-lk-common.c:410:transaction_lk_op] fore-replicate-0: lk op is for a transaction [2011-03-06 20:02:41.540197] D [afr-transaction.c:948:afr_post_blocking_inodelk_cbk] fore-replicate-0: Blocking inodelks failed. [2011-03-06 20:02:41.540332] D [afr-lk-common.c:964:afr_lock_blocking] fore-replicate-1: unable to lock on even one child [2011-03-06 20:02:41.540349] D [afr-lk-common.c:410:transaction_lk_op] fore-replicate-1: lk op is for a transaction [2011-03-06 20:02:41.540364] D [afr-transaction.c:948:afr_post_blocking_inodelk_cbk] fore-replicate-1: Blocking inodelks failed. [2011-03-06 20:02:41.541271] D [afr-lk-common.c:410:transaction_lk_op] fore-replicate-0: lk op is for a transaction [2011-03-06 20:02:41.541298] D [afr-transaction.c:973:afr_post_nonblocking_inodelk_cbk] fore-replicate-0: Non blocking inodelks failed. Proceeding to blocking [2011-03-06 20:02:41.541403] D [afr-lk-common.c:410:transaction_lk_op] fore-replicate-1: lk op is for a transaction [2011-03-06 20:02:41.541427] D [afr-transaction.c:973:afr_post_nonblocking_inodelk_cbk] fore-replicate-1: Non blocking inodelks failed. Proceeding to blocking [2011-03-06 20:02:41.542416] D [afr-lk-common.c:964:afr_lock_blocking] fore-replicate-0: unable to lock on even one child [2011-03-06 20:02:41.542442] D [afr-lk-common.c:410:transaction_lk_op] fore-replicate-0: lk op is for a transaction [2011-03-06 20:02:41.542456] D [afr-transaction.c:948:afr_post_blocking_inodelk_cbk] fore-replicate-0: Blocking inodelks failed. [2011-03-06 20:02:41.542616] D [afr-lk-common.c:964:afr_lock_blocking] fore-replicate-1: unable to lock on even one child [2011-03-06 20:02:41.542653] D [afr-lk-common.c:410:transaction_lk_op] fore-replicate-1: lk op is for a transaction [2011-03-06 20:02:41.542668] D [afr-transaction.c:948:afr_post_blocking_inodelk_cbk] fore-replicate-1: Blocking inodelks failed. [2011-03-06 20:02:41.542703] C [inode.c:218:__is_dentry_cyclic] fore/inode: detected cyclic loop formation during inode linkage. inode (1/00000000-0000-0000-0000-000000000001) linking under itself as vol
This bug is still there and seen in glusterfs-3.2.0qa13. Apr 17 06:11:04 centos-rahul-2 GlusterFS[30639]: [2011-04-17 06:11:04.923296] C [inode.c:232:__is_dentry_cyclic] 0-/export/mirror/inode: detected cyclic loop formation during inode linkage. inode (1/00000000-0000-0000-0000-000000000001) linking under itself as
Rahul/Johny, Can you guys check if this issue arises on the clean backend. (I suspect what Avati said is right, ie, if there was some directory which was export of gluster volume inside the current export directory, this can happen). If not seen in next week, I will be closing the bug, and we can re-open this if found. Regards, Amar
with patch http://patches.gluster.com/patch/7837 (for bug 764797), this bug should not be seen at all (from 3.1+ versions). Please re-open the bug if seen again (while testing master branch). Closing with works for me.