Bug 1212822

Summary:

Data Tiering: link files getting created on hot tier for all the files in cold tier

Product:

[Community] GlusterFS

Reporter:

Nag Pavan Chilakam <nchilaka>

Component:

tiering

Assignee:

Joseph Elwin Fernandes <josferna>

Status:

CLOSED NOTABUG

QA Contact:

bugs <bugs>

Severity:

urgent

Docs Contact:

Priority:

urgent

Version:

mainline

CC:

bugs, dlambrig, sankarshan

Target Milestone:

---

Keywords:

Triaged

Target Release:

---

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2015-04-22 15:28:27 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

1186580, 1260923

Attachments:

Description	Flags
cli logs	none

Description Nag Pavan Chilakam 2015-04-17 12:01:36 UTC

Description of problem:
======================
I created a 3 way replica volume and created some files and directory from mount point, then on attaching a tier and when i issued ls command, all the files in cold tier have a link file created in hot tier. This is not right. Files must either exist on hot or on cold tier. Link files get created when we rename a file or when a new brick is attached in the same dht range(tier layer is not in same dht range, as it is seperate range)
From my knowledge, this was seen even with a EC volume on tier and hence should be independent of volume type with tier

Version-Release number of selected component (if applicable):
=============================================================
[root@interstellar ~]# gluster --version
glusterfs 3.7dev built on Apr 13 2015 07:14:27
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@interstellar ~]# rpm -qa|grep gluster
glusterfs-3.7dev-0.994.gitf522001.el6.x86_64
glusterfs-resource-agents-3.7dev-0.994.gitf522001.el6.noarch
glusterfs-cli-3.7dev-0.994.gitf522001.el6.x86_64
glusterfs-api-devel-3.7dev-0.994.gitf522001.el6.x86_64
glusterfs-fuse-3.7dev-0.994.gitf522001.el6.x86_64
glusterfs-extra-xlators-3.7dev-0.994.gitf522001.el6.x86_64
glusterfs-rdma-3.7dev-0.994.gitf522001.el6.x86_64
glusterfs-api-3.7dev-0.994.gitf522001.el6.x86_64
glusterfs-devel-3.7dev-0.994.gitf522001.el6.x86_64
glusterfs-debuginfo-3.7dev-0.994.gitf522001.el6.x86_64
glusterfs-libs-3.7dev-0.994.gitf522001.el6.x86_64
glusterfs-geo-replication-3.7dev-0.994.gitf522001.el6.x86_64
glusterfs-server-3.7dev-0.994.gitf522001.el6.x86_64
glusterfs-regression-tests-3.7dev-0.994.gitf522001.el6.x86_64


Steps to Reproduce:
===================

1.create a 3 way replica and start and mount vol
gluster v create rep3 rep 3 ninja:/rhs/brick1/rep3a interstellar:/rhs/brick1/rep3a transformers:/rhs/brick1/rep3a interstellar:/rhs/brick1/rep3b ninja:/rhs/brick1/rep3b transformers:/rhs/brick1/rep3b 
volume create: rep3: success: please start the volume to access data
[root@interstellar ~]# gluster v info rep3
 
Volume Name: rep3
Type: Distributed-Replicate
Volume ID: 738f401f-5382-4763-b2a5-ebc71bcd0b28
Status: Created
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: ninja:/rhs/brick1/rep3a
Brick2: interstellar:/rhs/brick1/rep3a
Brick3: transformers:/rhs/brick1/rep3a
Brick4: interstellar:/rhs/brick1/rep3b
Brick5: ninja:/rhs/brick1/rep3b
Brick6: transformers:/rhs/brick1/rep3b
[root@interstellar ~]# gluster v status rep3
Volume rep3 is not started
[root@interstellar ~]# gluster v start rep3
volume start: rep3: success
[root@interstellar ~]# gluster v status rep3
Status of volume: rep3
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick ninja:/rhs/brick1/rep3a               49234     0          Y       14452
Brick interstellar:/rhs/brick1/rep3a        49187     0          Y       60206
Brick transformers:/rhs/brick1/rep3a        49172     0          Y       14930
Brick interstellar:/rhs/brick1/rep3b        49188     0          Y       60223
Brick ninja:/rhs/brick1/rep3b               49235     0          Y       14471
Brick transformers:/rhs/brick1/rep3b        49173     0          Y       14948
NFS Server on localhost                     N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       60249
NFS Server on 10.70.34.56                   N/A       N/A        N       N/A  
Self-heal Daemon on 10.70.34.56             N/A       N/A        Y       24275
NFS Server on ninja                         N/A       N/A        N       N/A  
Self-heal Daemon on ninja                   N/A       N/A        Y       14497
NFS Server on transformers                  N/A       N/A        N       N/A  
Self-heal Daemon on transformers            N/A       N/A        Y       14974
 
Task Status of Volume rep3
------------------------------------------------------------------------------
There are no active volume tasks
 
[root@interstellar ~]# gluster v info rep3
 
Volume Name: rep3
Type: Distributed-Replicate
Volume ID: 738f401f-5382-4763-b2a5-ebc71bcd0b28
Status: Started
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: ninja:/rhs/brick1/rep3a
Brick2: interstellar:/rhs/brick1/rep3a
Brick3: transformers:/rhs/brick1/rep3a
Brick4: interstellar:/rhs/brick1/rep3b
Brick5: ninja:/rhs/brick1/rep3b
Brick6: transformers:/rhs/brick1/rep3b
[root@interstellar ~]# gluster v status rep3^C
[root@interstellar ~]# 
[root@interstellar ~]# gluster v status rep3
Status of volume: rep3
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick ninja:/rhs/brick1/rep3a               49234     0          Y       14452
Brick interstellar:/rhs/brick1/rep3a        49187     0          Y       60206
Brick transformers:/rhs/brick1/rep3a        49172     0          Y       14930
Brick interstellar:/rhs/brick1/rep3b        49188     0          Y       60223
Brick ninja:/rhs/brick1/rep3b               49235     0          Y       14471
Brick transformers:/rhs/brick1/rep3b        49173     0          Y       14948
NFS Server on localhost                     N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       60249
NFS Server on 10.70.34.56                   N/A       N/A        N       N/A  
Self-heal Daemon on 10.70.34.56             N/A       N/A        Y       24275
NFS Server on ninja                         N/A       N/A        N       N/A  
Self-heal Daemon on ninja                   N/A       N/A        Y       14497
NFS Server on transformers                  N/A       N/A        N       N/A  
Self-heal Daemon on transformers            N/A       N/A        Y       14974
 
Task Status of Volume rep3
------------------------------------------------------------------------------
There are no active volume tasks


2. 
create some files on mount and also a dir


3.
Now attach a tier 
[root@interstellar ~]# gluster v attach-tier rep3 replica 3 ninja:/rhs/brick1/rep3-tier interstellar:/rhs/brick1/rep3-tier transformers:/rhs/brick1/rep3-tier
volume add-brick: success
[root@interstellar ~]# gluster v info rep3
 
Volume Name: rep3
Type: Tier
Volume ID: 738f401f-5382-4763-b2a5-ebc71bcd0b28
Status: Started
Number of Bricks: 3 x 3 = 9
Transport-type: tcp
Bricks:
Brick1: transformers:/rhs/brick1/rep3-tier
Brick2: interstellar:/rhs/brick1/rep3-tier
Brick3: ninja:/rhs/brick1/rep3-tier
Brick4: ninja:/rhs/brick1/rep3a
Brick5: interstellar:/rhs/brick1/rep3a
Brick6: transformers:/rhs/brick1/rep3a
Brick7: interstellar:/rhs/brick1/rep3b
Brick8: ninja:/rhs/brick1/rep3b
Brick9: transformers:/rhs/brick1/rep3b
[root@interstellar ~]# gluster v status rep3
Status of volume: rep3
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick transformers:/rhs/brick1/rep3-tier    49174     0          Y       15092
Brick interstellar:/rhs/brick1/rep3-tier    49189     0          Y       60438
Brick ninja:/rhs/brick1/rep3-tier           49236     0          Y       14618
Brick ninja:/rhs/brick1/rep3a               49234     0          Y       14452
Brick interstellar:/rhs/brick1/rep3a        49187     0          Y       60206
Brick transformers:/rhs/brick1/rep3a        49172     0          Y       14930
Brick interstellar:/rhs/brick1/rep3b        49188     0          Y       60223
Brick ninja:/rhs/brick1/rep3b               49235     0          Y       14471
Brick transformers:/rhs/brick1/rep3b        49173     0          Y       14948
NFS Server on localhost                     N/A       N/A        N       N/A  
NFS Server on 10.70.34.56                   N/A       N/A        N       N/A  
NFS Server on transformers                  N/A       N/A        N       N/A  
NFS Server on ninja                         N/A       N/A        N       N/A  
 
Task Status of Volume rep3
------------------------------------------------------------------------------
There are no active volume tasks
 



4. 
Now check the files on bricks it can be seen as below:
[root@interstellar ~]# getfattr -d -e hex -m . /rhs/brick*/rep*
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/rep3a
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000000000007ffa08f1
trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28

# file: rhs/brick1/rep3b
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000007ffa08f2ffffffff
trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28

# file: rhs/brick1/rep3-tier
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28

[root@interstellar ~]# getfattr -d -e hex -m . /rhs/brick*/rep*
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/rep3a
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000000000007ffa08f1
trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28
trusted.tier-gfid=0x0000000100000000000000009ff31dab

# file: rhs/brick1/rep3b
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000007ffa08f2ffffffff
trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28
trusted.tier-gfid=0x0000000100000000000000009ff31dab

# file: rhs/brick1/rep3-tier
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x000000010000000000000000ffffffff
trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28
trusted.tier-gfid=0x00000001000000009ff31dacffffffff

[root@interstellar ~]# ll /rhs/brick1/rep*/rhs/brick1/rep3a:
total 0
drwxr-xr-x. 2 root root 20 Jan 17 16:54 dir1
-rw-r--r--. 2 root root  0 Apr 17  2015 f4
-rw-r--r--. 2 root root  0 Apr 17  2015 f6
-rw-r--r--. 2 root root  0 Apr 17  2015 f7
-rw-r--r--. 2 root root  0 Apr 17  2015 f8
-rw-r--r--. 2 root root  0 Apr 17  2015 f9

/rhs/brick1/rep3b:
total 4
drwxr-xr-x. 2 root root  6 Jan 17 16:54 dir1
-rw-r--r--. 2 root root 18 Jan 17 16:54 f1
-rw-r--r--. 2 root root  0 Apr 17  2015 f10
-rw-r--r--. 2 root root  0 Apr 17  2015 f2
-rw-r--r--. 2 root root  0 Apr 17  2015 f3
-rw-r--r--. 2 root root  0 Apr 17  2015 f5

/rhs/brick1/rep3-tier:
total 0
---------T. 2 root root 0 Jan 17 17:11 f1
---------T. 2 root root 0 Jan 17 17:11 f10
---------T. 2 root root 0 Jan 17 17:11 f2
---------T. 2 root root 0 Jan 17 17:11 f3
---------T. 2 root root 0 Jan 17 17:11 f4
---------T. 2 root root 0 Jan 17 17:11 f5
---------T. 2 root root 0 Jan 17 17:11 f6
---------T. 2 root root 0 Jan 17 17:11 f7
---------T. 2 root root 0 Jan 17 17:11 f8
---------T. 2 root root 0 Jan 17 17:11 f9
[root@interstellar ~]# 


[root@ninja ~]# 
[root@ninja ~]# getfattr -d -e hex -m . /rhs/brick*/rep*
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/rep3a
security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000000000007ffa08f1
trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28

# file: rhs/brick1/rep3b
security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000007ffa08f2ffffffff
trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28

# file: rhs/brick1/rep3-tier
security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28

[root@ninja ~]# getfattr -d -e hex -m . /rhs/brick*/rep*
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/rep3a
security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000000000007ffa08f1
trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28
trusted.tier-gfid=0x0000000100000000000000009ff31dab

# file: rhs/brick1/rep3b
security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000007ffa08f2ffffffff
trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28
trusted.tier-gfid=0x0000000100000000000000009ff31dab

# file: rhs/brick1/rep3-tier
security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x000000010000000000000000ffffffff
trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28
trusted.tier-gfid=0x00000001000000009ff31dacffffffff

[root@ninja ~]# ll /rhs/brick1/rep*
/rhs/brick1/rep3a:
total 0
drwxr-xr-x. 2 root root 20 Apr 17 16:53 dir1
-rw-r--r--. 2 root root  0 Apr 17 16:53 f4
-rw-r--r--. 2 root root  0 Apr 17 16:53 f6
-rw-r--r--. 2 root root  0 Apr 17 16:53 f7
-rw-r--r--. 2 root root  0 Apr 17 16:53 f8
-rw-r--r--. 2 root root  0 Apr 17 16:53 f9

/rhs/brick1/rep3b:
total 4
drwxr-xr-x. 2 root root  6 Apr 17 16:53 dir1
-rw-r--r--. 2 root root 18 Apr 17 16:53 f1
-rw-r--r--. 2 root root  0 Apr 17 16:53 f10
-rw-r--r--. 2 root root  0 Apr 17 16:53 f2
-rw-r--r--. 2 root root  0 Apr 17 16:53 f3
-rw-r--r--. 2 root root  0 Apr 17 16:53 f5

/rhs/brick1/rep3-tier:
total 0
---------T. 2 root root 0 Apr 17 17:10 f1
---------T. 2 root root 0 Apr 17 17:10 f10
---------T. 2 root root 0 Apr 17 17:10 f2
---------T. 2 root root 0 Apr 17 17:10 f3
---------T. 2 root root 0 Apr 17 17:10 f4
---------T. 2 root root 0 Apr 17 17:10 f5
---------T. 2 root root 0 Apr 17 17:10 f6
---------T. 2 root root 0 Apr 17 17:10 f7
---------T. 2 root root 0 Apr 17 17:10 f8
---------T. 2 root root 0 Apr 17 17:10 f9
[root@ninja ~]# 





Expected results:
================
no link files should be created. 
If really the files have been heated then they must be moved to hot tier,else remain in cold tier

Comment 1 Nag Pavan Chilakam 2015-04-17 12:09:49 UTC

Created attachment 1015555 [details]
cli logs

Comment 2 Nag Pavan Chilakam 2015-04-17 12:17:18 UTC

sosreport rhsqe-repo:/var/www/html/sosreports/1212822

Comment 3 Dan Lambright 2015-04-22 15:28:27 UTC

This statement 

"Link files get created when we rename a file or when a new brick is attached in the same dht range(tier layer is not in same dht range, as it is seperate range)"

.. is false. The two tiers are in the same range, and the analogy "a new brick is created in the same range" is exactly what we are doing. A link file is indeed expected on both tiers when we migrate a file between them, this is correct behavior. 

Closing.