Description of problem: ========================== During rebalance of files : 1) The destination file is created with LINKFILE mode, linkto xattr is set and truncated with the file size same as the source. 2) After creating the file on the hashed subvolume rebalance process checks for the "free space" available on the destination bricks. If the data movement is attempted from higher disk space to lower disk space, then data is not migrated on to the hashed subvolume. But the link to file on the destination brick is not truncated to zero bytes . Even though the migration was skipped , link to file will have the same file size same as the cached sub-volume's file size. Version-Release number of selected component (if applicable): ============================================================ glusterfs 3.4.0.59rhs built on Feb 4 2014 08:44:13 How reproducible: Steps to Reproduce: ========================== 1. Create 1 x 3 replicate volume. Start the volume. Create files/dirs from mount point. NOTE : Each the brick had 1.7TB of space 2. Add 3 more bricks to the volume making it 2 x 3 distribute-replicate volume. NOTE : Each brick had 50GB of space 3. Start rebalance. Rebalance of all the files are skipped. Actual results: ================== 1. Link files were created with file size same as the source file size. 2. Rebalance was skipped on all the files. On the hashed sub-volume one of the directory listing info: ========================================================= root@king [Feb-10-2014-12:54:08] >ls -lh /rhs/bricks/exporter/user2/file* ---------T 2 root root 0 Feb 7 16:39 /rhs/bricks/exporter/user2/file1 ---------T 2 root root 2.0M Feb 7 16:39 /rhs/bricks/exporter/user2/file10 ---------T 2 root root 2.0M Feb 7 16:39 /rhs/bricks/exporter/user2/file11 ---------T 2 root root 1.2M Feb 7 16:39 /rhs/bricks/exporter/user2/file12 ---------T 2 root root 1.1M Feb 7 16:39 /rhs/bricks/exporter/user2/file13 ---------T 2 root root 0 Feb 7 16:39 /rhs/bricks/exporter/user2/file14 ---------T 2 root root 0 Feb 7 16:39 /rhs/bricks/exporter/user2/file15 ---------T 2 root root 1.5M Feb 7 16:39 /rhs/bricks/exporter/user2/file16 ---------T 2 root root 1.1M Feb 7 16:39 /rhs/bricks/exporter/user2/file17 ---------T 2 root root 1.4M Feb 7 16:39 /rhs/bricks/exporter/user2/file18 ---------T 2 root root 2.0M Feb 7 16:39 /rhs/bricks/exporter/user2/file19 ---------T 2 root root 0 Feb 7 16:39 /rhs/bricks/exporter/user2/file2 ---------T 2 root root 0 Feb 7 16:39 /rhs/bricks/exporter/user2/file20 ---------T 2 root root 0 Feb 7 16:39 /rhs/bricks/exporter/user2/file21 ---------T 2 root root 0 Feb 7 16:39 /rhs/bricks/exporter/user2/file22 ---------T 2 root root 0 Feb 7 16:39 /rhs/bricks/exporter/user2/file23 ---------T 2 root root 1.6M Feb 7 16:39 /rhs/bricks/exporter/user2/file24 ---------T 2 root root 1.5M Feb 7 16:39 /rhs/bricks/exporter/user2/file25 ---------T 2 root root 1.5M Feb 7 16:39 /rhs/bricks/exporter/user2/file3 ---------T 2 root root 1.5M Feb 7 16:39 /rhs/bricks/exporter/user2/file4 ---------T 2 root root 0 Feb 7 16:39 /rhs/bricks/exporter/user2/file5 ---------T 2 root root 0 Feb 7 16:39 /rhs/bricks/exporter/user2/file6 ---------T 2 root root 1.4M Feb 7 16:39 /rhs/bricks/exporter/user2/file7 ---------T 2 root root 0 Feb 7 16:39 /rhs/bricks/exporter/user2/file8 ---------T 2 root root 1.7M Feb 7 16:39 /rhs/bricks/exporter/user2/file9 root@king [Feb-10-2014-12:54:30] > Expected results: ==================== Link files should be created with zero byte size. Additional Info: ==================== root@rhs-client11 [Feb-10-2014- 2:18:46] >gluster v info Volume Name: exporter Type: Distributed-Replicate Volume ID: 26eef3b2-e712-4f2c-ade6-9a26e3c85cc7 Status: Started Number of Bricks: 2 x 3 = 6 Transport-type: tcp Bricks: Brick1: rhs-client11:/rhs/bricks/exporter Brick2: rhs-client12:/rhs/bricks/exporter Brick3: rhs-client13:/rhs/bricks/exporter Brick4: king:/rhs/bricks/exporter Brick5: darrel:/rhs/bricks/exporter Brick6: hicks:/rhs/bricks/exporter Options Reconfigured: cluster.entry-self-heal: off features.quota: on
SOS Reports : http://rhsqe-repo.lab.eng.blr.redhat.com/bugs_necessary_info/1063162/
Updating the bug with my findings: I had had a look at Shwetha's setup (and the code of course) before the bug was raised. This problem is caused due to the order in which the following two actions are performed on a file about to be migrated by the rebalance process: a. determining the disk space availability on the dest and src subvols; b. opening fd on the destination sub-volume and truncating the link file on the destination to the size of the original file on the source sub-volume. The rebalance process does (b) first followed by (a) (when it actually should be doing (a) followed by (b)). And upon sensing that the disk space available on the destination is less than that on the src sub-volume, the rebalance process aborts migration of the file in question, thereby leaving the link file with a non-zero size on the destination sub-volume. Note: * Truncating the file to a higher size does not cause data blocks to be allocated for the file. * The last change on this area of code was on 2013-09-17 and is not introduced in Corbett.
Cloning this to 3.1. To be fixed in future.