Bug 1643716 - "OSError: [Errno 40] Too many levels of symbolic links" when syncing deletion of directory hierarchy [NEEDINFO]
Summary: "OSError: [Errno 40] Too many levels of symbolic links" when syncing deletion...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: GlusterFS
Classification: Community
Component: geo-replication
Version: 4.1
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
Assignee: Shwetha K Acharya
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-10-27 19:52 UTC by Christian Lohmaier
Modified: 2019-06-27 12:29 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-27 12:29:44 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
sacharya: needinfo? (lohmaier+rhbz)


Attachments (Terms of Use)

Description Christian Lohmaier 2018-10-27 19:52:47 UTC
Description of problem:
using geo-replication with deep directory and lots of hardlinks can lead to "too many levels of symbolic links" error when files should be removed on the slave

Version-Release number of selected component (if applicable):
all of 4.1, but also in 3.12


How reproducible:
needs special directory, in our case trees of node modules or ruby gems 

Steps to Reproduce:
1. use rsnapshot to create backups of a virtual machine on the master
2. rsnapshot creates hardlinked copy of the synced tree to daily.0 which then get rotated
3. so .sync → daily.0, mv daily.0 daily.1 & .sync → daily.0
4. at end of rotation daily.6 is renamed to _delete.someid and gets removed.

Actual results:
the geo-replication of that deletion triggers the "too many levels of symbolic links" error, but after repeated attempts then finally manages to delete the hierarchy nevertheless, but geo-replication is forced back to history crawl and the process takes way too long.

Expected results:
no problems in removing files that it managed to create without problem earlier.

Additional info:
ruby gems and node collections are likely to trigger it, but need to be largish, but by no means excessive

one example:
# find /master_mnt/path/to/site/bundle/ruby/2.1.0/gems/ -type f |wc -l
4972
# find /master_mnt/path/to/site/bundle/ruby/2.1.0/gems/ -type d |wc -l
918

the contents don't change frequently, so most of the files are hardlinks with 8 to 12 targets (.sync, daily.0 to daily.6, weekly.0 to weekly.3, _delete)

Comment 1 Shwetha K Acharya 2019-05-14 13:34:32 UTC
This bug is fixed in gluster 5 release.

Bugzilla link: https://bugzilla.redhat.com/show_bug.cgi?id=1597540, updating to next version will solve this problem.

Please confirm your gluster version and kindly close this bug.

Comment 2 Shwetha K Acharya 2019-06-27 12:29:44 UTC
I am closing this bug as the issue is addressed in release 5. Feel free to open new issue if you still face any problem.


Note You need to log in before you can comment on or make changes to this bug.