Hide Forgot
Description of problem: hardlink sync through xsync crawl resulted in duplicate files in the same directory. # ls -l /mnt/slave/level00/level10/level20/hardlink_to_files/ | grep 52970470%%10ACFYD6Z4 -rw-r--r-- 2 root root 1160 Nov 28 14:10 52970470%%10ACFYD6Z4 -rw-r--r-- 1 root root 1160 Nov 28 14:10 52970470%%10ACFYD6Z4 Version-Release number of selected component (if applicable):glusterfs-3.4.0.45geo-1 How reproducible:didn't try to reproduce Steps to Reproduce: 1.create and start a geo-rep relationship between master and slave. 2.create files using the command, ./crefi.py -n 1000 --multi -b 10 -d 10 --random --max=2K --min=1K /mnt/master/ 3.let them sync to slave, 4.stop geo-rep session, 5../crefi.py -n 1000 --multi -b 10 -d 10 --random --max=2K --min=1K --fop=hardlink /mnt/master/ 6. after completing the creation of hardlinks, start the geo-rep session Actual results: results in duplicate files in the same directory. Expected results: It shouldn't create duplicate files. Additional info:
This I was able to hit it again with the same build. But this time is didn't happen in /mnt/slave/level00/hardlink_to_files/ but it happened in /mnt/slave/level00/ which is usual. The script I ran, for i in {1..20} do ./crefi.py -n 100 --multi -b 10 -d 10 --random --max=2K --min=1K /mnt/master/ sleep 100 ./crefi.py -n 100 --multi -b 10 -d 10 --random --max=2K --min=1K --fop=chmod /mnt/master/ sleep 100 ./crefi.py -n 100 --multi -b 10 -d 10 --random --max=2K --min=1K --fop=chown /mnt/master/ sleep 100 ./crefi.py -n 100 --multi -b 10 -d 10 --random --max=2K --min=1K --fop=chgrp /mnt/master/ sleep 100 ./crefi.py -n 100 --multi -b 10 -d 10 --random --max=2K --min=1K --fop=symlink /mnt/master/ sleep 100 ./crefi.py -n 100 --multi -b 10 -d 10 --random --max=2K --min=1K --fop=hardlink /mnt/master/ sleep 100 ./crefi.py -n 100 --multi -b 10 -d 10 --random --max=2K --min=1K --fop=rename /mnt/master/ sleep 500 done which would incrimentaly create files and do operations on all of the files at that moment. This took some 24 hours to complete, in between there were many disconnections, which resulted in xsync crawls. This might have triggered hardlink syncing through hybrid crawl. But if that was the case the duplicate should have been in hardlinks_to_files directories, it happened in /mnt/slave/level00/. Since it is the issue with duplicate files, I have updated it here.
Vijay, If a hardlink created to a file falls into the subvolume other than on which file resides, dht linkto file gets created on the hashed subvolume and hardlink is created on the cached subvolume. Since the geo-rep daemon runs per brick, the xync changelog is created for the linkto file and also the original hardlink file. Both gets processed by the sync daemon resulting in race to form duplicate files of same name. The patch of ignoring dht linkto files will solve this problem from geo-rep side. https://code.engineering.redhat.com/gerrit/17336 This will fix the issue from geo-rep side, but it seems there is a larger issue of race outside geo-rep where it is allowing to process the file creation of same name in same location. -Kotresh H R
This has happened again on the build glusterfs-3.4.0.58rhs-1.
As per the Comment 3, link-to files are ignored. and with the following patch, Hardlink is handled as link in XSync crawl. http://review.gluster.org/#/c/5934/ This bug can be closed once verified as below, 1. Create data and Hardlinks in Master Volume. 2. Geo-replication Create and Start. 3. All Regular files and Hardlinks checksums should match.
Closing this bug since RHGS 2.1 release reached EOL. Required bugs are cloned to RHGS 3.1. Please re-open this issue if found again.