Bug 764228 (GLUSTER-2496)

Summary:	cyclic directory structure:
Product:	[Community] GlusterFS	Reporter:	Amar Tumballi <amarts>
Component:	core	Assignee:	Anand Avati <aavati>
Status:	CLOSED WORKSFORME	QA Contact:
Severity:	medium	Docs Contact:
Priority:	medium
Version:	mainline	CC:	chrisw, gluster-bugs, rabhat, vraman
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	---
Regression:	---	Mount Type:	fuse
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Amar Tumballi 2011-03-08 04:56:57 UTC

oops.. pressed enter before adding contents




On Mon, Mar 7, 2011 at 5:16 PM, Rahul C S <rahulcbhat> wrote:
Hi Amar,

Distributed-replicate setup, ran 
dbench -t 200 250
iozone -i 0 -i 1 -i 2 -i 3 -i 4 -i 5 -i 6 -i 7 -i 8 -i 9 -i 10 -i 11 -i 12 -s 100m -r 22k
on both fuse and nfs clients

Also over the weekend on fuse mount, i ran the /opt/qa/tools/kernel_compile.sh script 

From today morning detecting this in the brick log: 
[2011-03-06 20:15:06.306354] C [inode.c:218:__is_dentry_cyclic] /export/inode: detected cyclic loop formation during inode linkage. inode (1/00000000-0000-0000-0000-000000000001) linking under itself as vol

Following is the output on the client.
[root@centos-qa-client-3 fuse]# ls -l
total 8
drwxrwxr-x 5 root root 4096 Mar  6 19:57 linux-2.6.31.1
?--------- ? ?    ?       ?            ? rep
?--------- ? ?    ?       ?            ? vol


[root@centos-qa-client-3 nfs]# ls -l
total 319464
-rw-r--r-- 1 root root 326762496 Mar  6 20:03 heavyf
drwxrwxr-x 5 root root      8192 Mar  6 19:57 linux-2.6.31.1
drwxr-xr-x 5 root root      4096 Mar  6 20:03 rep
drwxr-xr-x 2 root root      8192 Mar  6 14:24 vol

[root@centos-qa-client-3 nfs]# rm -rf *
rm: WARNING: Circular directory structure.
This almost certainly means that you have a corrupted file system.
NOTIFY YOUR SYSTEM MANAGER.
The following directory is part of the cycle:
  `rep/vol'

rm: WARNING: Circular directory structure.
This almost certainly means that you have a corrupted file system.
NOTIFY YOUR SYSTEM MANAGER.
The following directory is part of the cycle:
  `rep/vol/vol/vol'



Logs:


[2011-03-06 20:02:41.530062] D [afr-common.c:555:afr_lookup_collect_xattr] fore-replicate-0: metadata self-heal is pending for /.
[2011-03-06 20:02:41.530380] D [afr-common.c:555:afr_lookup_collect_xattr] fore-replicate-0: metadata self-heal is pending for /.
[2011-03-06 20:02:41.530436] I [afr-common.c:672:afr_lookup_done] fore-replicate-0: split brain detected during lookup of /.
[2011-03-06 20:02:41.530456] I [afr-common.c:716:afr_lookup_done] fore-replicate-0: background  meta-data data self-heal triggered. path: /
[2011-03-06 20:02:41.530963] D [afr-self-heal-metadata.c:682:afr_sh_post_nonblocking_inodelk_cbk] fore-replicate-0: Non Blocking inodelks done. Proceeding to FOP
[2011-03-06 20:02:41.531503] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] fore-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2011-03-06 20:02:41.531537] D [afr-lk-common.c:415:transaction_lk_op] fore-replicate-0: lk op is for a self heal
[2011-03-06 20:02:41.531949] D [afr-self-heal-metadata.c:81:afr_sh_metadata_done] fore-replicate-0: aborting selfheal of /
[2011-03-06 20:02:41.532006] I [afr-self-heal-common.c:1527:afr_self_heal_completion_cbk] fore-replicate-0: background  meta-data data self-heal completed on /
[2011-03-06 20:02:41.533937] D [afr-common.c:555:afr_lookup_collect_xattr] fore-replicate-0: metadata self-heal is pending for /.
[2011-03-06 20:02:41.534119] D [afr-common.c:555:afr_lookup_collect_xattr] fore-replicate-0: metadata self-heal is pending for /.
[2011-03-06 20:02:41.534145] I [afr-common.c:672:afr_lookup_done] fore-replicate-0: split brain detected during lookup of /.
[2011-03-06 20:02:41.534168] I [afr-common.c:716:afr_lookup_done] fore-replicate-0: background  meta-data data self-heal triggered. path: /
[2011-03-06 20:02:41.534511] D [afr-self-heal-metadata.c:682:afr_sh_post_nonblocking_inodelk_cbk] fore-replicate-0: Non Blocking inodelks done. Proceeding to FOP
[2011-03-06 20:02:41.534921] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] fore-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2011-03-06 20:02:41.534951] D [afr-lk-common.c:415:transaction_lk_op] fore-replicate-0: lk op is for a self heal
[2011-03-06 20:02:41.535276] D [afr-self-heal-metadata.c:81:afr_sh_metadata_done] fore-replicate-0: aborting selfheal of /
[2011-03-06 20:02:41.535301] I [afr-self-heal-common.c:1527:afr_self_heal_completion_cbk] fore-replicate-0: background  meta-data data self-heal completed on /
[2011-03-06 20:02:41.537830] D [dht-layout.c:584:dht_layout_normalize] fore-dht: directory /vol looked up first time
[2011-03-06 20:02:41.537853] D [dht-common.c:172:dht_lookup_dir_cbk] fore-dht: fixing assignment on /vol
[2011-03-06 20:02:41.538841] D [afr-lk-common.c:410:transaction_lk_op] fore-replicate-0: lk op is for a transaction
[2011-03-06 20:02:41.538871] D [afr-transaction.c:973:afr_post_nonblocking_inodelk_cbk] fore-replicate-0: Non blocking inodelks failed. Proceeding to blocking
[2011-03-06 20:02:41.539046] D [afr-lk-common.c:410:transaction_lk_op] fore-replicate-1: lk op is for a transaction
[2011-03-06 20:02:41.539075] D [afr-transaction.c:973:afr_post_nonblocking_inodelk_cbk] fore-replicate-1: Non blocking inodelks failed. Proceeding to blocking
[2011-03-06 20:02:41.540148] D [afr-lk-common.c:964:afr_lock_blocking] fore-replicate-0: unable to lock on even one child
[2011-03-06 20:02:41.540180] D [afr-lk-common.c:410:transaction_lk_op] fore-replicate-0: lk op is for a transaction
[2011-03-06 20:02:41.540197] D [afr-transaction.c:948:afr_post_blocking_inodelk_cbk] fore-replicate-0: Blocking inodelks failed.
[2011-03-06 20:02:41.540332] D [afr-lk-common.c:964:afr_lock_blocking] fore-replicate-1: unable to lock on even one child
[2011-03-06 20:02:41.540349] D [afr-lk-common.c:410:transaction_lk_op] fore-replicate-1: lk op is for a transaction
[2011-03-06 20:02:41.540364] D [afr-transaction.c:948:afr_post_blocking_inodelk_cbk] fore-replicate-1: Blocking inodelks failed.
[2011-03-06 20:02:41.541271] D [afr-lk-common.c:410:transaction_lk_op] fore-replicate-0: lk op is for a transaction
[2011-03-06 20:02:41.541298] D [afr-transaction.c:973:afr_post_nonblocking_inodelk_cbk] fore-replicate-0: Non blocking inodelks failed. Proceeding to blocking
[2011-03-06 20:02:41.541403] D [afr-lk-common.c:410:transaction_lk_op] fore-replicate-1: lk op is for a transaction
[2011-03-06 20:02:41.541427] D [afr-transaction.c:973:afr_post_nonblocking_inodelk_cbk] fore-replicate-1: Non blocking inodelks failed. Proceeding to blocking
[2011-03-06 20:02:41.542416] D [afr-lk-common.c:964:afr_lock_blocking] fore-replicate-0: unable to lock on even one child
[2011-03-06 20:02:41.542442] D [afr-lk-common.c:410:transaction_lk_op] fore-replicate-0: lk op is for a transaction
[2011-03-06 20:02:41.542456] D [afr-transaction.c:948:afr_post_blocking_inodelk_cbk] fore-replicate-0: Blocking inodelks failed.
[2011-03-06 20:02:41.542616] D [afr-lk-common.c:964:afr_lock_blocking] fore-replicate-1: unable to lock on even one child
[2011-03-06 20:02:41.542653] D [afr-lk-common.c:410:transaction_lk_op] fore-replicate-1: lk op is for a transaction
[2011-03-06 20:02:41.542668] D [afr-transaction.c:948:afr_post_blocking_inodelk_cbk] fore-replicate-1: Blocking inodelks failed.
[2011-03-06 20:02:41.542703] C [inode.c:218:__is_dentry_cyclic] fore/inode: detected cyclic loop formation during inode linkage. inode (1/00000000-0000-0000-0000-000000000001) linking under itself as vol

Comment 1 Amar Tumballi 2011-03-08 07:55:07 UTC

Comment 2 Raghavendra Bhat 2011-04-19 10:39:28 UTC

This bug is still there and seen in glusterfs-3.2.0qa13.


Apr 17 06:11:04 centos-rahul-2 GlusterFS[30639]: [2011-04-17 06:11:04.923296] C [inode.c:232:__is_dentry_cyclic] 0-/export/mirror/inode: detected cyclic loop formation during inode linkage. inode (1/00000000-0000-0000-0000-000000000001) linking under itself as

Comment 3 Amar Tumballi 2011-06-08 15:32:23 UTC

Rahul/Johny,

Can you guys check if this issue arises on the clean backend. (I suspect what Avati said is right, ie, if there was some directory which was export of gluster volume inside the current export directory, this can happen). If not seen in next week, I will be closing the bug, and we can re-open this if found.

Regards,
Amar

Comment 4 Amar Tumballi 2011-07-14 05:23:19 UTC

with patch http://patches.gluster.com/patch/7837 (for bug 764797), this bug should not be seen at all (from 3.1+ versions). Please re-open the bug if seen again (while testing master branch).

Closing with works for me.