Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1643716

Summary: "OSError: [Errno 40] Too many levels of symbolic links" when syncing deletion of directory hierarchy
Product: [Community] GlusterFS Reporter: Christian Lohmaier <lohmaier+rhbz>
Component: geo-replicationAssignee: Shwetha K Acharya <sacharya>
Status: CLOSED NOTABUG QA Contact:
Severity: high Docs Contact:
Priority: high    
Version: 4.1CC: atumball, bugs, pasik, sunkumar
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-27 12:29:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Christian Lohmaier 2018-10-27 19:52:47 UTC
Description of problem:
using geo-replication with deep directory and lots of hardlinks can lead to "too many levels of symbolic links" error when files should be removed on the slave

Version-Release number of selected component (if applicable):
all of 4.1, but also in 3.12


How reproducible:
needs special directory, in our case trees of node modules or ruby gems 

Steps to Reproduce:
1. use rsnapshot to create backups of a virtual machine on the master
2. rsnapshot creates hardlinked copy of the synced tree to daily.0 which then get rotated
3. so .sync → daily.0, mv daily.0 daily.1 & .sync → daily.0
4. at end of rotation daily.6 is renamed to _delete.someid and gets removed.

Actual results:
the geo-replication of that deletion triggers the "too many levels of symbolic links" error, but after repeated attempts then finally manages to delete the hierarchy nevertheless, but geo-replication is forced back to history crawl and the process takes way too long.

Expected results:
no problems in removing files that it managed to create without problem earlier.

Additional info:
ruby gems and node collections are likely to trigger it, but need to be largish, but by no means excessive

one example:
# find /master_mnt/path/to/site/bundle/ruby/2.1.0/gems/ -type f |wc -l
4972
# find /master_mnt/path/to/site/bundle/ruby/2.1.0/gems/ -type d |wc -l
918

the contents don't change frequently, so most of the files are hardlinks with 8 to 12 targets (.sync, daily.0 to daily.6, weekly.0 to weekly.3, _delete)

Comment 1 Shwetha K Acharya 2019-05-14 13:34:32 UTC
This bug is fixed in gluster 5 release.

Bugzilla link: https://bugzilla.redhat.com/show_bug.cgi?id=1597540, updating to next version will solve this problem.

Please confirm your gluster version and kindly close this bug.

Comment 2 Shwetha K Acharya 2019-06-27 12:29:44 UTC
I am closing this bug as the issue is addressed in release 5. Feel free to open new issue if you still face any problem.

Comment 3 Red Hat Bugzilla 2023-09-14 04:41:25 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days