Description of problem: ========================== On a 2x3 dis-rep volume, renamed few files from mount. While rename is in progress 1 brick both the subvolumes went offline. Once the rename is complete, the bricks are brought online. self-heal indicated the self-heal is completed. But when tried to get the renamed files, they were missing from mountpoint. Observation: ============= The directory E_dir_1 was renamed to E_dir_renamed_1. After the brick came online, the E_dir_1 is recreated under E_dir_renamed_1 with exactly same entries as E_dir_renamed_1. Version-Release number of selected component (if applicable): =============================================================== glusterfs-3.7.5-13.el6rhs.x86_64 How reproducible: =============== Often Steps to Reproduce: ====================== 1. create a 2x3 dis-rep cold and hot tiered volume. start the volume. Create fuse mount. 2. Create files on mount point. 3.Repeat the following steps for the operations: (create file/dirs, copy files/dirs from files created, renamed few files and dirs) a. starting the operation on the mount b. bring down certain bricks from each subvolume c. after the operation is complete calculate arequal-checksum d. bring back the bricks e. wait for self-heal to complete f. once self-heal is complete, calculate arequal-checksum g. compare checksums calculated at (c) and (f). they should be same h. bring down bricks from each subvolume i. calculate arequal-checksum j. compare checksums calculated at (f) and (j). they should be same. Actual results: ================== With rename operation, the arequal-checksum mismated b/w (c) and (f). 2015-12-30 08:31:33,624 INFO compare_arequal_checksum_mount Arequal-Checksum on mount cutlass.lab.eng.blr.redhat.com:/mnt/glusterfs1 : 'after-op-before-self-heal-rename' is: Entry counts Regular files : 1597 Directories : 105 Symbolic links : 0 Other : 0 Total : 1702 Metadata checksums Regular files : 48974c Directories : 24d74c Symbolic links : 3e9 Other : 3e9 Checksums Regular files : 85e9d8cb4c9db47a45fbadc1302f1f0f Directories : 7a041170126a5d7f Symbolic links : 0 Other : 0 Total : ba16647a6ed8f60a 2015-12-30 08:31:33,624 INFO compare_arequal_checksum_mount Arequal-Checksum on mount cutlass.lab.eng.blr.redhat.com:/mnt/glusterfs1 : 'after-self-heal-rename' is: Entry counts Regular files : 1582 Directories : 105 Symbolic links : 0 Other : 0 Total : 1687 Metadata checksums Regular files : 2cb0 Directories : 24d74c Symbolic links : 3e9 Other : 3e9 Checksums Regular files : dabaae4e2af4bf861ed63a104b16459b Directories : 39041079166d425e Symbolic links : 0 Other : 0 Total : fd688427778fb843 2015-12-30 08:31:33,624 ERROR compare_arequal_checksum_mount Checksums on mount cutlass.lab.eng.blr.redhat.com:/mnt/glusterfs1 of 'after-op-before-self-heal-rename' and 'after-self-heal-rename' doesn't match 2015-12-30 08:31:33,624 INFO run Executing find /mnt/glusterfs1 | uniq -d on cutlass.lab.eng.blr.redhat.com 2015-12-30 08:31:34,681 INFO run "find /mnt/glusterfs1 | uniq -d" on cutlass.lab.eng.blr.redhat.com: RETCODE is 0 2015-12-30 08:31:34,681 INFO get_duplicate_entries_from_mount No Duplicate Entries found under cutlass.lab.eng.blr.redhat.com:/mnt/glusterfs1 2015-12-30 08:31:34,682 ERROR get_missing_entries_from_mount Missing entries from mount when comparing entries 'after-op-before-self-heal-rename' and entries 'after-self-heal-rename': /mnt/glusterfs1/E_dir_renamed_1/E_dir_1/E_file_renamed_11 /mnt/glusterfs1/E_dir_renamed_1/E_dir_1/E_file_renamed_10 /mnt/glusterfs1/E_dir_renamed_1/E_dir_1/E_file_renamed_13 /mnt/glusterfs1/E_dir_renamed_1/E_dir_1/E_file_renamed_12 /mnt/glusterfs1/E_dir_renamed_1/E_dir_1/E_file_renamed_15 /mnt/glusterfs1/E_dir_renamed_1/E_dir_1/E_file_renamed_14 /mnt/glusterfs1/E_dir_renamed_1/E_dir_1/E_file_renamed_9 /mnt/glusterfs1/E_dir_renamed_1/E_dir_1/E_file_renamed_8 /mnt/glusterfs1/E_dir_renamed_1/E_dir_1/E_file_renamed_1 /mnt/glusterfs1/E_dir_renamed_1/E_dir_1/E_file_renamed_3 /mnt/glusterfs1/E_dir_renamed_1/E_dir_1/E_file_renamed_2 /mnt/glusterfs1/E_dir_renamed_1/E_dir_1/E_file_renamed_5 /mnt/glusterfs1/E_dir_renamed_1/E_dir_1/E_file_renamed_4 /mnt/glusterfs1/E_dir_renamed_1/E_dir_1/E_file_renamed_7 /mnt/glusterfs1/E_dir_renamed_1/E_dir_1/E_file_renamed_6 Expected results: ==================== arequal-checksum should be same after self-heal Additional info: =================== [root@rhsauto019:~] Dec-30-2015 03:28:07 $gluster v info Volume Name: testvol Type: Distributed-Replicate Volume ID: 19b1e5ca-af1d-4361-8b4d-d13992fcc185 Status: Started Number of Bricks: 2 x 3 = 6 Transport-type: tcp Bricks: Brick1: rhsauto019.lab.eng.blr.redhat.com:/bricks/brick0/testvol_brick0 Brick2: rhsauto020.lab.eng.blr.redhat.com:/bricks/brick0/testvol_brick1 Brick3: rhsauto021.lab.eng.blr.redhat.com:/bricks/brick0/testvol_brick2 Brick4: rhsauto022.lab.eng.blr.redhat.com:/bricks/brick0/testvol_brick3 Brick5: rhsauto019.lab.eng.blr.redhat.com:/bricks/brick1/testvol_brick4 Brick6: rhsauto020.lab.eng.blr.redhat.com:/bricks/brick1/testvol_brick5 Options Reconfigured: performance.readdir-ahead: on
SosReport: http://rhsqe-repo.lab.eng.blr.redhat.com/bugs_necessary_info/1294732/