Description of problem: ======================= In a geo-replication setup where master is tiered and slave is non-tiered volume, some of the hardlinks are synced as T file while some are synced properly. For example: ============ Master volume hardlink files: [root@dj scripts]# ls -l /mnt/vol0/thread0/level00/hardlink_to_files/ total 26 -rw-r--r--. 2 root root 10707 Jan 21 04:11 56a002ca%%IUUZUI3ZFV -rw-r--r--. 2 root root 15094 Jan 21 2016 56a002ca%%VLS8G8PZ11 [root@dj scripts]# Synced to slave as: [root@dj scripts]# ls -l /mnt/vol1/thread0/level00/hardlink_to_files/ total 11 -rw-r--r--. 1 root root 10707 Jan 21 04:11 56a002ca%%IUUZUI3ZFV ---------T. 1 root root 0 Jan 21 2016 56a002ca%%VLS8G8PZ11 [root@dj scripts]# [root@dj scripts]# cat /mnt/vol1/thread0/level00/hardlink_to_files/56a002ca%%VLS8G8PZ11 [root@dj scripts]# For the files which is synced properly, the changelogs records are as follows: ============================================================================== Active Hot Tier changelog records in processed: ----------------------------------------------- [root@dhcp37-165 .processed]# grep -i "03c926fd-4cc0-4d41-a8be-50efa70916a4" * Binary file archive_201601.tar matches CHANGELOG.1453329701:E 03c926fd-4cc0-4d41-a8be-50efa70916a4 CREATE 33188 0 0 84eae93e-208b-4890-aa09-1ee223d0f313%2F56a00268%25%25S5CGBPUMYP CHANGELOG.1453329701:D 03c926fd-4cc0-4d41-a8be-50efa70916a4 CHANGELOG.1453329791:E 03c926fd-4cc0-4d41-a8be-50efa70916a4 LINK b8bb1da7-1d1b-4600-982d-d6243d6e952b%2F56a002ca%25%25IUUZUI3ZFV [root@dhcp37-165 .processed]# Active Cold Tier changelog records in processed: ------------------------------------------------ [root@dhcp37-165 .processed]# grep -i "03c926fd-4cc0-4d41-a8be-50efa70916a4" * Binary file archive_201601.tar matches CHANGELOG.1453329693:E 03c926fd-4cc0-4d41-a8be-50efa70916a4 MKNOD 33280 0 0 84eae93e-208b-4890-aa09-1ee223d0f313%2F56a00268%25%25S5CGBPUMYP CHANGELOG.1453329800:E 03c926fd-4cc0-4d41-a8be-50efa70916a4 MKNOD 33280 0 0 b8bb1da7-1d1b-4600-982d-d6243d6e952b%2F56a002ca%25%25IUUZUI3ZFV [root@dhcp37-165 .processed]# For the files which is not synced properly, the changelogs records are as follows: ============================================================================== Active Hot Tier changelog records in processed: ----------------------------------------------- [root@dhcp37-165 .processed]# grep -i "85f08137-c61a-4b81-8264-23e9510a203a" * Binary file archive_201601.tar matches CHANGELOG.1453329701:E 85f08137-c61a-4b81-8264-23e9510a203a CREATE 33188 0 0 84eae93e-208b-4890-aa09-1ee223d0f313%2F56a00268%25%25S0KZN4J05C CHANGELOG.1453329701:D 85f08137-c61a-4b81-8264-23e9510a203a CHANGELOG.1453329791:E 85f08137-c61a-4b81-8264-23e9510a203a LINK b8bb1da7-1d1b-4600-982d-d6243d6e952b%2F56a002ca%25%25VLS8G8PZ11 [root@dhcp37-165 .processed]# Active Cold Tier changelog records in processed: ------------------------------------------------ [root@dhcp37-165 .processed]# grep -i "85f08137-c61a-4b81-8264-23e9510a203a" * Binary file archive_201601.tar matches CHANGELOG.1453329693:E 85f08137-c61a-4b81-8264-23e9510a203a MKNOD 33280 0 0 84eae93e-208b-4890-aa09-1ee223d0f313%2F56a00268%25%25S0KZN4J05C [root@dhcp37-165 .processed]# Another Active Cold Tier changelog recorded followin in processed: ------------------------------------------------------------------ [root@dhcp37-155 .processed]# grep -i "85f08137-c61a-4b81-8264-23e9510a203a" * Binary file archive_201601.tar matches CHANGELOG.1453329789:E 85f08137-c61a-4b81-8264-23e9510a203a MKNOD 33280 0 0 b8bb1da7-1d1b-4600-982d-d6243d6e952b%2F56a002ca%25%25VLS8G8PZ11 [root@dhcp37-155 .processed]# Above test is done by disabling quick-read on slave to avoid client side crash. Version-Release number of selected component (if applicable): ============================================================= glusterfs-3.7.5-16.el7rhgs.x86_64 How reproducible: ================= Always Steps Carried: ============== 1. Create master and slave cluster 2. Create master as tiered volume (HT: 2x2, CT: 2x(4+2) 3. Create slave volume (2x2) and disable quick-read 4. Create geo-replication session between master and slave 5. Mount master volume 6. Create set of data on master volume, after a while promotion demotion should start 7. Create hardlinks of the files 8. Check for the hardlink files entry created on slave 9. Check the permissions of the hardlinks files synced to slave Actual results: =============== Many hardlink files are synced as T files to slave Expected results: ================= Files should be synced as regular link files
RCA: Since with Tiering internal MKNOD is recorded, hardlinks are also recorded as MKNOD. Geo-replication will create hardlinks as new files in Slave instead of hardlink. This new file will have the same GFID as original file but not hardlinked. Data may copied to original file alone when sync happens through rsync.
Downstream patch sent https://code.engineering.redhat.com/gerrit/#/c/66678/
Verified with build: glusterfs-3.7.5-18.el7rhgs.x86_64 Note: quick-read is turned off on slave side. Create 10k files and another 10k hardlinks while promote/demotes were inprogress. All the hardlinks got sync to slave and arequal checksum matches as: [root@dj scripts]# ./arequal-checksum -p /mnt/master/ Entry counts Regular files : 20099 Directories : 2551 Symbolic links : 270 Other : 0 Total : 22920 Metadata checksums Regular files : 486e85 Directories : 24d481 Symbolic links : 3e9 Other : 3e9 Checksums Regular files : 3304cde9564cea9b7c58ed0b9b952814 Directories : 521f19232d2d2244 Symbolic links : 30611b563e514719 Other : 0 Total : 2d222297dea5a7d2 [root@dj scripts]# [root@dj scripts]# ./arequal-checksum -p /mnt/slave/ Entry counts Regular files : 20099 Directories : 2551 Symbolic links : 270 Other : 0 Total : 22920 Metadata checksums Regular files : 486e85 Directories : 24d481 Symbolic links : 3e9 Other : 3e9 Checksums Regular files : 3304cde9564cea9b7c58ed0b9b952814 Directories : 521f19232d2d2244 Symbolic links : 30611b563e514719 Other : 0 Total : 2d222297dea5a7d2 [root@dj scripts]# Moving the bug to verified state. Will run the regression and update if the issue is reproducible.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0193.html