Description of problem: On a tiered vol, detach tier was triggered when few files were in hot tier and were continuously appended from fuse mount. Detach tier status showed failed (due to bz#1332957). however, even with the issue seen in bz#1332957, migration of files from hot tier to cold tier should have been completed. Volume Name: spiderman Type: Tier Volume ID: 543b6e31-326c-42a2-8e4f-e121ba72a1f5 Status: Started Number of Bricks: 10 Transport-type: tcp Hot Tier : Hot Tier Type : Distributed-Replicate Number of Bricks: 2 x 2 = 4 Brick1: 10.70.47.171:/bricks/brick4/l1 Brick2: 10.70.47.128:/bricks/brick4/l1 Brick3: 10.70.47.187:/bricks/brick4/l1 Brick4: 10.70.46.103:/bricks/brick4/l1 Cold Tier: Cold Tier Type : Disperse Number of Bricks: 1 x (4 + 2) = 6 Brick5: 10.70.47.187:/bricks/brick0/l1 Brick6: 10.70.46.103:/bricks/brick0/l1 Brick7: 10.70.47.128:/bricks/brick0/l1 Brick8: 10.70.47.171:/bricks/brick0/l1 Brick9: 10.70.47.187:/bricks/brick1/l1 Brick10: 10.70.46.103:/bricks/brick1/l1 Options Reconfigured: diagnostics.client-log-level: INFO cluster.tier-mode: cache features.ctr-enabled: on features.quota-deem-statfs: on features.inode-quota: on features.quota: on performance.readdir-ahead: on cluster.enable-shared-storage: enable nfs-ganesha: disable Version-Release number of selected component (if applicable): glusterfs-server-3.7.9-3.el7rhgs.x86_64 How reproducible: 1/1 Steps to Reproduce: 1. From a tiered volume (hot tier: 2x2, cold tier: 1x(4+2)), detach tier and commit it to make it a mere disperse volume 2. create 100 files from fuse mount 3. attach tier 4. keep appending data to 100 files created in step-2, so they are all promoted to hot tier 5. detach tier - allow detach tier to complete 6. Detach tier operation failed due to bz#1332957 7. check for files in hot tier Actual results: Not all files are migrated to cold tier Expected results: All files should be migrated from hot tier to cold tier Additional info: sosreports will be attached shortly.
If the Deatch tier has failed(for whatever reason, in this case the failure of removal of xattr) in the beginning that mean definitely files are not migrated to COLD. So Wont this be a redundant bug. Suggestion : 1. Either we close this bug saying it deduplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1332957, and update 1332957 saying files are not still in hot. 2. Or Change the title/summary of 1334234 saying files detach tier failed due to failure of removal of xattr, hence files are still there in hot. Because the title now may be misleading saying "Files remain in hot tier even after detach tier completes", giving the impression on a sunny day, where detach tier is fine, files are still in HOT. Also make this bug depends on 1332957.
Oops My bad! you are correct. We do it at the end of detach tier start i.e by then file would have been move. This is not a redundant bug. We should have files moved to cold.
Patch posted upstream : review.gluster.org/#/c/14285/
RCA: Previously we had wrongly placed the clearing tier-fix-layout-complete xattr before the joining of migration threads. This would lead to situations where failure of clearing the xattr would cause the premature death of migration threads. Fix: Now we clear the xattr only after the data movement threads join, ensuring that all migration is done.
Downstream patch : https://code.engineering.redhat.com/gerrit/#/c/74382/
The issue reported in the bug is no more seen in build - glusterfs-3.7.9-5. Detach tier completes successfully and migrates all files to cold tier. Moving the bug to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1240