Description of problem: ========================= Consider a 3 x 2 distribute-replicate volume. If any of the brick is offline when a distribute subvolume is removed there will be pending self-heals to be performed on the offline brick of the files/dirs migrated from the removed bricks . After successful completion of remove-brick operation (start, status 'completed', commit) if the source brick goes offline even before sync brick comes online (connectivity issues) and sync brick comes online creating files/directories from mount point when source brick is offline will result in the data loss (loss of files / directories which got migrated as part of remove-brick operation) . Version-Release number of selected component (if applicable): =========================================================== glusterfs 3.4.0.59rhs built on Feb 4 2014 08:44:13 How reproducible: ================ Often Steps to Reproduce: ====================== 1. Create 3 x 2 distribute-replicate volume. Start the volume. 2. Create fuse mount. 3. Bring down brick5 offline. 4. Create 10 files from mount point. 5. Bring back brick5 online. 6. Wait for self-heal to happen 7. Bring brick6 offline. 8. remove the distribute sub-volume-1 from the volume. 9. Wait for the migration to complete and then commit the remove-brick operation. 10. Bring brick5 offline. 11. Bring back brick6 online. 12. create files from mount point. 13. Bring brick5 online. Actual results: =============== Self-heal happens from brick6 to brick5. All the files migrated to brick5 as part of remove brick operation when brick6 was offline will be lost. Expected results: ================= TBD
Case 2:- ======== Consider a 3 x 2 distribute-replicate volume. If any of the brick is offline when a distribute subvolume is removed there will be pending self-heals to be performed on the offline brick of the files/dirs migrated from the removed bricks. After successful remove-brick operation , even though the graph has changed and brick5, brick6 now gets client-2 and client-3 afr change-log attributes, Opendir still refers to the previous stale client-4 and client-5 change-log attributes of the brick5 and brick6 and self-heals all the data. Steps to Reproduce: ====================== 1. Create 3 x 2 distribute-replicate volume. Start the volume. 2. Create fuse mount. 3. Bring down brick5 offline. 4. Create 10 files from mount point. 5. Bring back brick5 online. 6. Wait for self-heal to happen 7. Bring brick6 offline. 8. remove the distribute sub-volume-1 from the volume. 9. Wait for the migration to complete and then commit the remove-brick operation. 10. Bring back brick6 online. 11. perform "ls -l" from mount point. self-heal happens from brick5 to brick6 and the stale change-logs are still referred.
Case 1 will be fixed with the persistent changelog implementation to be implemented in Denali. But it needs to be documented as a known issue for Corbett. Hence adding to BZ 1035040. doc text needs to be set. Case 2 happens if before self-heal, afr_opendir happens for the first time on an inode which triggers a conservative merge. This is the expected behaviour by design.
Please review the edited doc text and sign off.
Updated as suggested. Please sign off.
Looks good to me.
Thank you for submitting this issue for consideration in Red Hat Gluster Storage. The release for which you requested us to review, is now End of Life. Please See https://access.redhat.com/support/policy/updates/rhs/ If you can reproduce this bug against a currently maintained version of Red Hat Gluster Storage, please feel free to file a new report against the current release.