Description of problem: ====================== I created a 3 way replica volume and created some files and directory from mount point, then on attaching a tier and when i issued ls command, all the files in cold tier have a link file created in hot tier. This is not right. Files must either exist on hot or on cold tier. Link files get created when we rename a file or when a new brick is attached in the same dht range(tier layer is not in same dht range, as it is seperate range) From my knowledge, this was seen even with a EC volume on tier and hence should be independent of volume type with tier Version-Release number of selected component (if applicable): ============================================================= [root@interstellar ~]# gluster --version glusterfs 3.7dev built on Apr 13 2015 07:14:27 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. [root@interstellar ~]# rpm -qa|grep gluster glusterfs-3.7dev-0.994.gitf522001.el6.x86_64 glusterfs-resource-agents-3.7dev-0.994.gitf522001.el6.noarch glusterfs-cli-3.7dev-0.994.gitf522001.el6.x86_64 glusterfs-api-devel-3.7dev-0.994.gitf522001.el6.x86_64 glusterfs-fuse-3.7dev-0.994.gitf522001.el6.x86_64 glusterfs-extra-xlators-3.7dev-0.994.gitf522001.el6.x86_64 glusterfs-rdma-3.7dev-0.994.gitf522001.el6.x86_64 glusterfs-api-3.7dev-0.994.gitf522001.el6.x86_64 glusterfs-devel-3.7dev-0.994.gitf522001.el6.x86_64 glusterfs-debuginfo-3.7dev-0.994.gitf522001.el6.x86_64 glusterfs-libs-3.7dev-0.994.gitf522001.el6.x86_64 glusterfs-geo-replication-3.7dev-0.994.gitf522001.el6.x86_64 glusterfs-server-3.7dev-0.994.gitf522001.el6.x86_64 glusterfs-regression-tests-3.7dev-0.994.gitf522001.el6.x86_64 Steps to Reproduce: =================== 1.create a 3 way replica and start and mount vol gluster v create rep3 rep 3 ninja:/rhs/brick1/rep3a interstellar:/rhs/brick1/rep3a transformers:/rhs/brick1/rep3a interstellar:/rhs/brick1/rep3b ninja:/rhs/brick1/rep3b transformers:/rhs/brick1/rep3b volume create: rep3: success: please start the volume to access data [root@interstellar ~]# gluster v info rep3 Volume Name: rep3 Type: Distributed-Replicate Volume ID: 738f401f-5382-4763-b2a5-ebc71bcd0b28 Status: Created Number of Bricks: 2 x 3 = 6 Transport-type: tcp Bricks: Brick1: ninja:/rhs/brick1/rep3a Brick2: interstellar:/rhs/brick1/rep3a Brick3: transformers:/rhs/brick1/rep3a Brick4: interstellar:/rhs/brick1/rep3b Brick5: ninja:/rhs/brick1/rep3b Brick6: transformers:/rhs/brick1/rep3b [root@interstellar ~]# gluster v status rep3 Volume rep3 is not started [root@interstellar ~]# gluster v start rep3 volume start: rep3: success [root@interstellar ~]# gluster v status rep3 Status of volume: rep3 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ninja:/rhs/brick1/rep3a 49234 0 Y 14452 Brick interstellar:/rhs/brick1/rep3a 49187 0 Y 60206 Brick transformers:/rhs/brick1/rep3a 49172 0 Y 14930 Brick interstellar:/rhs/brick1/rep3b 49188 0 Y 60223 Brick ninja:/rhs/brick1/rep3b 49235 0 Y 14471 Brick transformers:/rhs/brick1/rep3b 49173 0 Y 14948 NFS Server on localhost N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 60249 NFS Server on 10.70.34.56 N/A N/A N N/A Self-heal Daemon on 10.70.34.56 N/A N/A Y 24275 NFS Server on ninja N/A N/A N N/A Self-heal Daemon on ninja N/A N/A Y 14497 NFS Server on transformers N/A N/A N N/A Self-heal Daemon on transformers N/A N/A Y 14974 Task Status of Volume rep3 ------------------------------------------------------------------------------ There are no active volume tasks [root@interstellar ~]# gluster v info rep3 Volume Name: rep3 Type: Distributed-Replicate Volume ID: 738f401f-5382-4763-b2a5-ebc71bcd0b28 Status: Started Number of Bricks: 2 x 3 = 6 Transport-type: tcp Bricks: Brick1: ninja:/rhs/brick1/rep3a Brick2: interstellar:/rhs/brick1/rep3a Brick3: transformers:/rhs/brick1/rep3a Brick4: interstellar:/rhs/brick1/rep3b Brick5: ninja:/rhs/brick1/rep3b Brick6: transformers:/rhs/brick1/rep3b [root@interstellar ~]# gluster v status rep3^C [root@interstellar ~]# [root@interstellar ~]# gluster v status rep3 Status of volume: rep3 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ninja:/rhs/brick1/rep3a 49234 0 Y 14452 Brick interstellar:/rhs/brick1/rep3a 49187 0 Y 60206 Brick transformers:/rhs/brick1/rep3a 49172 0 Y 14930 Brick interstellar:/rhs/brick1/rep3b 49188 0 Y 60223 Brick ninja:/rhs/brick1/rep3b 49235 0 Y 14471 Brick transformers:/rhs/brick1/rep3b 49173 0 Y 14948 NFS Server on localhost N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 60249 NFS Server on 10.70.34.56 N/A N/A N N/A Self-heal Daemon on 10.70.34.56 N/A N/A Y 24275 NFS Server on ninja N/A N/A N N/A Self-heal Daemon on ninja N/A N/A Y 14497 NFS Server on transformers N/A N/A N N/A Self-heal Daemon on transformers N/A N/A Y 14974 Task Status of Volume rep3 ------------------------------------------------------------------------------ There are no active volume tasks 2. create some files on mount and also a dir 3. Now attach a tier [root@interstellar ~]# gluster v attach-tier rep3 replica 3 ninja:/rhs/brick1/rep3-tier interstellar:/rhs/brick1/rep3-tier transformers:/rhs/brick1/rep3-tier volume add-brick: success [root@interstellar ~]# gluster v info rep3 Volume Name: rep3 Type: Tier Volume ID: 738f401f-5382-4763-b2a5-ebc71bcd0b28 Status: Started Number of Bricks: 3 x 3 = 9 Transport-type: tcp Bricks: Brick1: transformers:/rhs/brick1/rep3-tier Brick2: interstellar:/rhs/brick1/rep3-tier Brick3: ninja:/rhs/brick1/rep3-tier Brick4: ninja:/rhs/brick1/rep3a Brick5: interstellar:/rhs/brick1/rep3a Brick6: transformers:/rhs/brick1/rep3a Brick7: interstellar:/rhs/brick1/rep3b Brick8: ninja:/rhs/brick1/rep3b Brick9: transformers:/rhs/brick1/rep3b [root@interstellar ~]# gluster v status rep3 Status of volume: rep3 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick transformers:/rhs/brick1/rep3-tier 49174 0 Y 15092 Brick interstellar:/rhs/brick1/rep3-tier 49189 0 Y 60438 Brick ninja:/rhs/brick1/rep3-tier 49236 0 Y 14618 Brick ninja:/rhs/brick1/rep3a 49234 0 Y 14452 Brick interstellar:/rhs/brick1/rep3a 49187 0 Y 60206 Brick transformers:/rhs/brick1/rep3a 49172 0 Y 14930 Brick interstellar:/rhs/brick1/rep3b 49188 0 Y 60223 Brick ninja:/rhs/brick1/rep3b 49235 0 Y 14471 Brick transformers:/rhs/brick1/rep3b 49173 0 Y 14948 NFS Server on localhost N/A N/A N N/A NFS Server on 10.70.34.56 N/A N/A N N/A NFS Server on transformers N/A N/A N N/A NFS Server on ninja N/A N/A N N/A Task Status of Volume rep3 ------------------------------------------------------------------------------ There are no active volume tasks 4. Now check the files on bricks it can be seen as below: [root@interstellar ~]# getfattr -d -e hex -m . /rhs/brick*/rep* getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/rep3a security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000000000007ffa08f1 trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28 # file: rhs/brick1/rep3b security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000007ffa08f2ffffffff trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28 # file: rhs/brick1/rep3-tier security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28 [root@interstellar ~]# getfattr -d -e hex -m . /rhs/brick*/rep* getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/rep3a security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000000000007ffa08f1 trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28 trusted.tier-gfid=0x0000000100000000000000009ff31dab # file: rhs/brick1/rep3b security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000007ffa08f2ffffffff trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28 trusted.tier-gfid=0x0000000100000000000000009ff31dab # file: rhs/brick1/rep3-tier security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x000000010000000000000000ffffffff trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28 trusted.tier-gfid=0x00000001000000009ff31dacffffffff [root@interstellar ~]# ll /rhs/brick1/rep*/rhs/brick1/rep3a: total 0 drwxr-xr-x. 2 root root 20 Jan 17 16:54 dir1 -rw-r--r--. 2 root root 0 Apr 17 2015 f4 -rw-r--r--. 2 root root 0 Apr 17 2015 f6 -rw-r--r--. 2 root root 0 Apr 17 2015 f7 -rw-r--r--. 2 root root 0 Apr 17 2015 f8 -rw-r--r--. 2 root root 0 Apr 17 2015 f9 /rhs/brick1/rep3b: total 4 drwxr-xr-x. 2 root root 6 Jan 17 16:54 dir1 -rw-r--r--. 2 root root 18 Jan 17 16:54 f1 -rw-r--r--. 2 root root 0 Apr 17 2015 f10 -rw-r--r--. 2 root root 0 Apr 17 2015 f2 -rw-r--r--. 2 root root 0 Apr 17 2015 f3 -rw-r--r--. 2 root root 0 Apr 17 2015 f5 /rhs/brick1/rep3-tier: total 0 ---------T. 2 root root 0 Jan 17 17:11 f1 ---------T. 2 root root 0 Jan 17 17:11 f10 ---------T. 2 root root 0 Jan 17 17:11 f2 ---------T. 2 root root 0 Jan 17 17:11 f3 ---------T. 2 root root 0 Jan 17 17:11 f4 ---------T. 2 root root 0 Jan 17 17:11 f5 ---------T. 2 root root 0 Jan 17 17:11 f6 ---------T. 2 root root 0 Jan 17 17:11 f7 ---------T. 2 root root 0 Jan 17 17:11 f8 ---------T. 2 root root 0 Jan 17 17:11 f9 [root@interstellar ~]# [root@ninja ~]# [root@ninja ~]# getfattr -d -e hex -m . /rhs/brick*/rep* getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/rep3a security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000000000007ffa08f1 trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28 # file: rhs/brick1/rep3b security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000007ffa08f2ffffffff trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28 # file: rhs/brick1/rep3-tier security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28 [root@ninja ~]# getfattr -d -e hex -m . /rhs/brick*/rep* getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/rep3a security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000000000007ffa08f1 trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28 trusted.tier-gfid=0x0000000100000000000000009ff31dab # file: rhs/brick1/rep3b security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000007ffa08f2ffffffff trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28 trusted.tier-gfid=0x0000000100000000000000009ff31dab # file: rhs/brick1/rep3-tier security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x000000010000000000000000ffffffff trusted.glusterfs.volume-id=0x738f401f53824763b2a5ebc71bcd0b28 trusted.tier-gfid=0x00000001000000009ff31dacffffffff [root@ninja ~]# ll /rhs/brick1/rep* /rhs/brick1/rep3a: total 0 drwxr-xr-x. 2 root root 20 Apr 17 16:53 dir1 -rw-r--r--. 2 root root 0 Apr 17 16:53 f4 -rw-r--r--. 2 root root 0 Apr 17 16:53 f6 -rw-r--r--. 2 root root 0 Apr 17 16:53 f7 -rw-r--r--. 2 root root 0 Apr 17 16:53 f8 -rw-r--r--. 2 root root 0 Apr 17 16:53 f9 /rhs/brick1/rep3b: total 4 drwxr-xr-x. 2 root root 6 Apr 17 16:53 dir1 -rw-r--r--. 2 root root 18 Apr 17 16:53 f1 -rw-r--r--. 2 root root 0 Apr 17 16:53 f10 -rw-r--r--. 2 root root 0 Apr 17 16:53 f2 -rw-r--r--. 2 root root 0 Apr 17 16:53 f3 -rw-r--r--. 2 root root 0 Apr 17 16:53 f5 /rhs/brick1/rep3-tier: total 0 ---------T. 2 root root 0 Apr 17 17:10 f1 ---------T. 2 root root 0 Apr 17 17:10 f10 ---------T. 2 root root 0 Apr 17 17:10 f2 ---------T. 2 root root 0 Apr 17 17:10 f3 ---------T. 2 root root 0 Apr 17 17:10 f4 ---------T. 2 root root 0 Apr 17 17:10 f5 ---------T. 2 root root 0 Apr 17 17:10 f6 ---------T. 2 root root 0 Apr 17 17:10 f7 ---------T. 2 root root 0 Apr 17 17:10 f8 ---------T. 2 root root 0 Apr 17 17:10 f9 [root@ninja ~]# Expected results: ================ no link files should be created. If really the files have been heated then they must be moved to hot tier,else remain in cold tier
Created attachment 1015555 [details] cli logs
sosreport rhsqe-repo:/var/www/html/sosreports/1212822
This statement "Link files get created when we rename a file or when a new brick is attached in the same dht range(tier layer is not in same dht range, as it is seperate range)" .. is false. The two tiers are in the same range, and the analogy "a new brick is created in the same range" is exactly what we are doing. A link file is indeed expected on both tiers when we migrate a file between them, this is correct behavior. Closing.