Bug 1063162

Summary:	rebalance : link files are created with size same as the source file size but not as zero byte file
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	spandura
Component:	distribute	Assignee:	Nithya Balachandran <nbalacha>
Status:	CLOSED DEFERRED	QA Contact:	storage-qa-internal <storage-qa-internal>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	2.1	CC:	kdhananj, nlevinki, spalai, vbellur
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1270750 1286131 (view as bug list)		Environment:
Last Closed:	2015-11-27 11:42:12 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1270750, 1286131

Description spandura 2014-02-10 07:28:29 UTC

Description of problem:
==========================
During rebalance of files :

1) The destination file is created with LINKFILE mode, linkto xattr is set and truncated with the file size same as the source. 

2) After creating the file on the hashed subvolume rebalance process checks for the "free space" available on the destination bricks. If the data movement is  attempted from higher disk space to lower disk space, then data is not migrated on to the hashed subvolume. 

But the link to file on the destination brick is not truncated to zero bytes . Even though the migration was skipped , link to file will have the same  file size same as the cached sub-volume's file size. 

Version-Release number of selected component (if applicable):
============================================================
glusterfs 3.4.0.59rhs built on Feb  4 2014 08:44:13

How reproducible:


Steps to Reproduce:
==========================
1. Create 1 x 3 replicate volume. Start the volume. Create files/dirs from mount point. 

NOTE : Each the brick had 1.7TB of space

2. Add 3 more bricks to the volume making it 2 x 3 distribute-replicate volume.

NOTE : Each brick had 50GB of space 

3. Start rebalance. Rebalance of all the files are skipped. 

Actual results:
==================
1. Link files were created with file size same as the source file size. 

2. Rebalance was skipped on all the files. 

On the hashed sub-volume one of the directory listing info:
=========================================================
root@king [Feb-10-2014-12:54:08] >ls -lh /rhs/bricks/exporter/user2/file*
---------T 2 root root    0 Feb  7 16:39 /rhs/bricks/exporter/user2/file1
---------T 2 root root 2.0M Feb  7 16:39 /rhs/bricks/exporter/user2/file10
---------T 2 root root 2.0M Feb  7 16:39 /rhs/bricks/exporter/user2/file11
---------T 2 root root 1.2M Feb  7 16:39 /rhs/bricks/exporter/user2/file12
---------T 2 root root 1.1M Feb  7 16:39 /rhs/bricks/exporter/user2/file13
---------T 2 root root    0 Feb  7 16:39 /rhs/bricks/exporter/user2/file14
---------T 2 root root    0 Feb  7 16:39 /rhs/bricks/exporter/user2/file15
---------T 2 root root 1.5M Feb  7 16:39 /rhs/bricks/exporter/user2/file16
---------T 2 root root 1.1M Feb  7 16:39 /rhs/bricks/exporter/user2/file17
---------T 2 root root 1.4M Feb  7 16:39 /rhs/bricks/exporter/user2/file18
---------T 2 root root 2.0M Feb  7 16:39 /rhs/bricks/exporter/user2/file19
---------T 2 root root    0 Feb  7 16:39 /rhs/bricks/exporter/user2/file2
---------T 2 root root    0 Feb  7 16:39 /rhs/bricks/exporter/user2/file20
---------T 2 root root    0 Feb  7 16:39 /rhs/bricks/exporter/user2/file21
---------T 2 root root    0 Feb  7 16:39 /rhs/bricks/exporter/user2/file22
---------T 2 root root    0 Feb  7 16:39 /rhs/bricks/exporter/user2/file23
---------T 2 root root 1.6M Feb  7 16:39 /rhs/bricks/exporter/user2/file24
---------T 2 root root 1.5M Feb  7 16:39 /rhs/bricks/exporter/user2/file25
---------T 2 root root 1.5M Feb  7 16:39 /rhs/bricks/exporter/user2/file3
---------T 2 root root 1.5M Feb  7 16:39 /rhs/bricks/exporter/user2/file4
---------T 2 root root    0 Feb  7 16:39 /rhs/bricks/exporter/user2/file5
---------T 2 root root    0 Feb  7 16:39 /rhs/bricks/exporter/user2/file6
---------T 2 root root 1.4M Feb  7 16:39 /rhs/bricks/exporter/user2/file7
---------T 2 root root    0 Feb  7 16:39 /rhs/bricks/exporter/user2/file8
---------T 2 root root 1.7M Feb  7 16:39 /rhs/bricks/exporter/user2/file9
root@king [Feb-10-2014-12:54:30] >

Expected results:
====================
Link files should be created with zero byte size. 

Additional Info:
====================

root@rhs-client11 [Feb-10-2014- 2:18:46] >gluster v info
 
Volume Name: exporter
Type: Distributed-Replicate
Volume ID: 26eef3b2-e712-4f2c-ade6-9a26e3c85cc7
Status: Started
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: rhs-client11:/rhs/bricks/exporter
Brick2: rhs-client12:/rhs/bricks/exporter
Brick3: rhs-client13:/rhs/bricks/exporter
Brick4: king:/rhs/bricks/exporter
Brick5: darrel:/rhs/bricks/exporter
Brick6: hicks:/rhs/bricks/exporter
Options Reconfigured:
cluster.entry-self-heal: off
features.quota: on

Comment 2 spandura 2014-02-10 10:01:55 UTC

SOS Reports : http://rhsqe-repo.lab.eng.blr.redhat.com/bugs_necessary_info/1063162/

Comment 3 Krutika Dhananjay 2014-02-11 11:51:33 UTC

Updating the bug with my findings:

I had had a look at Shwetha's setup (and the code of course) before the bug was raised. This problem is caused due to the order in which the following two actions are performed on a file about to be migrated by the rebalance process:

a. determining the disk space availability on the dest and src subvols;
b. opening fd on the destination sub-volume and truncating the link file on the destination to the size of the original file on the source sub-volume.

The rebalance process does (b) first followed by (a) (when it actually should be doing (a) followed by (b)). And upon sensing that the disk space available on the destination is less than that on the src sub-volume, the rebalance process aborts migration of the file in question, thereby leaving the link file with a non-zero size on the destination sub-volume.

Note:
* Truncating the file to a higher size does not cause data blocks to be allocated for the file.
* The last change on this area of code was on 2013-09-17 and is not introduced in Corbett.

Comment 4 Susant Kumar Palai 2015-11-27 11:42:12 UTC

Cloning this to 3.1. To be fixed in future.