On a fuse mount, When we do a named lookup of a file which are on cold tier, a link-to file is supposed to be created. But this doesnt happen consistently. For eg: If I have a set of files say f1,f2,,,,f30 and i do "ls f1" , "ls f2", "ls f3" I see link file in hot tier only for file f1 The remaining linkfile creations fails. Workaround:Umount and remount the volume steps to reproduce: ================== 1)create a tier volume 2)attach a tier 3)mount vol on fuse 4)create some files say f1,f2....f30 5)wait for all to get demoted 6)Now issue a lookup using ls for each file it can be seen only the first file link file creation passes
It sounds like the lookups are hitting the revalidate code path in the lookup here. That would not create a linkto file.
With the latest code, the behavior has changed: New files created on tiered volumes always should create a T file on the cold sub volume, and the data on the hot sub volume (with the exception for when the hot sub volume is full). Legacy files, i.e. those that existed before the tier was attached, should create a T file on promotion, if I understand the inner DHT workings. However, See Nithya's comment about not creating a T file on revalidate (expected behavior); the DHT logic is subtle and the rules may not always be what we think. It may not be valid to assume a T file always will exist for legacy files. I ran a test: 1. created 10 legacy files on normal volume 2. attach tier in test mode 3. touched files 4. waited for promotion 5. observed T files created on cold tier. Note also, the T file itself is an internal abstraction that is not customer facing. Given those points, we should revise this bug to confirm it needs to be open or not.
On 3.7.5-7 I tested and found the hashed subvol is now cold tier. hence linkfiles in hot tier will not be created (unless disk is full, for which another bug is there) Hence this problem is solved [root@zod ~]# ll /*/brick*/linkf* /rhs/brick1/linkf: total 8 drwxr-xr-x. 2 root root 8192 Dec 2 18:35 cdir ---------T. 2 root root 0 Dec 2 18:36 f12 ---------T. 2 root root 0 Dec 2 18:36 f13 ---------T. 2 root root 0 Dec 2 18:36 f16 ---------T. 2 root root 0 Dec 2 18:36 f17 ---------T. 2 root root 0 Dec 2 18:36 f21 ---------T. 2 root root 0 Dec 2 18:36 f24 ---------T. 2 root root 0 Dec 2 18:36 f25 ---------T. 2 root root 0 Dec 2 18:36 f26 ---------T. 2 root root 0 Dec 2 18:36 f27 ---------T. 2 root root 0 Dec 2 18:36 f28 ---------T. 2 root root 0 Dec 2 18:36 f30 ---------T. 2 root root 0 Dec 2 18:36 f32 ---------T. 2 root root 0 Dec 2 18:36 f34 ---------T. 2 root root 0 Dec 2 18:36 f35 ---------T. 2 root root 0 Dec 2 18:36 f36 ---------T. 2 root root 0 Dec 2 18:36 f39 ---------T. 2 root root 0 Dec 2 18:36 f4 ---------T. 2 root root 0 Dec 2 18:36 f40 ---------T. 2 root root 0 Dec 2 18:36 f41 ---------T. 2 root root 0 Dec 2 18:36 f43 ---------T. 2 root root 0 Dec 2 18:36 f44 ---------T. 2 root root 0 Dec 2 18:36 f48 ---------T. 2 root root 0 Dec 2 18:36 f49 ---------T. 2 root root 0 Dec 2 18:36 f50 ---------T. 2 root root 0 Dec 2 18:36 f6 ---------T. 2 root root 0 Dec 2 18:36 f7 ---------T. 2 root root 0 Dec 2 18:36 f8 ---------T. 2 root root 0 Dec 2 18:36 f9 /rhs/brick2/linkf: total 8 drwxr-xr-x. 2 root root 8192 Dec 2 18:35 cdir ---------T. 2 root root 0 Dec 2 18:36 f1 ---------T. 2 root root 0 Dec 2 18:36 f10 ---------T. 2 root root 0 Dec 2 18:36 f11 ---------T. 2 root root 0 Dec 2 18:36 f14 ---------T. 2 root root 0 Dec 2 18:36 f15 ---------T. 2 root root 0 Dec 2 18:36 f18 ---------T. 2 root root 0 Dec 2 18:36 f19 ---------T. 2 root root 0 Dec 2 18:36 f2 ---------T. 2 root root 0 Dec 2 18:36 f20 ---------T. 2 root root 0 Dec 2 18:36 f22 ---------T. 2 root root 0 Dec 2 18:36 f23 ---------T. 2 root root 0 Dec 2 18:36 f29 ---------T. 2 root root 0 Dec 2 18:36 f3 ---------T. 2 root root 0 Dec 2 18:36 f31 ---------T. 2 root root 0 Dec 2 18:36 f33 ---------T. 2 root root 0 Dec 2 18:36 f37 ---------T. 2 root root 0 Dec 2 18:36 f38 ---------T. 2 root root 0 Dec 2 18:36 f42 ---------T. 2 root root 0 Dec 2 18:36 f45 ---------T. 2 root root 0 Dec 2 18:36 f46 ---------T. 2 root root 0 Dec 2 18:36 f47 ---------T. 2 root root 0 Dec 2 18:36 f5 /rhs/brick6/linkf_hot: total 180 drwxr-xr-x. 2 root root 8192 Dec 2 18:36 cdir -rw-r--r--. 2 root root 3 Dec 2 18:35 f10 -rw-r--r--. 2 root root 3 Dec 2 18:35 f11 -rw-r--r--. 2 root root 3 Dec 2 18:35 f12 -rw-r--r--. 2 root root 3 Dec 2 18:35 f13 -rw-r--r--. 2 root root 3 Dec 2 18:35 f14 -rw-r--r--. 2 root root 3 Dec 2 18:35 f15 -rw-r--r--. 2 root root 3 Dec 2 18:35 f16 -rw-r--r--. 2 root root 3 Dec 2 18:35 f17 -rw-r--r--. 2 root root 3 Dec 2 18:35 f21 -rw-r--r--. 2 root root 3 Dec 2 18:35 f23 -rw-r--r--. 2 root root 3 Dec 2 18:35 f24 -rw-r--r--. 2 root root 3 Dec 2 18:35 f25 -rw-r--r--. 2 root root 3 Dec 2 18:35 f26 -rw-r--r--. 2 root root 3 Dec 2 18:35 f27 -rw-r--r--. 2 root root 3 Dec 2 18:35 f28 -rw-r--r--. 2 root root 3 Dec 2 18:35 f29 -rw-r--r--. 2 root root 3 Dec 2 18:35 f3 -rw-r--r--. 2 root root 3 Dec 2 18:35 f30 -rw-r--r--. 2 root root 3 Dec 2 18:35 f31 -rw-r--r--. 2 root root 3 Dec 2 18:35 f32 -rw-r--r--. 2 root root 3 Dec 2 18:35 f33 -rw-r--r--. 2 root root 3 Dec 2 18:35 f34 -rw-r--r--. 2 root root 3 Dec 2 18:35 f35 -rw-r--r--. 2 root root 3 Dec 2 18:35 f36 -rw-r--r--. 2 root root 3 Dec 2 18:35 f37 -rw-r--r--. 2 root root 3 Dec 2 18:35 f38 -rw-r--r--. 2 root root 3 Dec 2 18:35 f39 -rw-r--r--. 2 root root 3 Dec 2 18:35 f4 -rw-r--r--. 2 root root 3 Dec 2 18:35 f40 -rw-r--r--. 2 root root 3 Dec 2 18:35 f41 -rw-r--r--. 2 root root 3 Dec 2 18:35 f42 -rw-r--r--. 2 root root 3 Dec 2 18:35 f43 -rw-r--r--. 2 root root 3 Dec 2 18:35 f44 -rw-r--r--. 2 root root 3 Dec 2 18:35 f45 -rw-r--r--. 2 root root 3 Dec 2 18:35 f47 -rw-r--r--. 2 root root 3 Dec 2 18:35 f48 -rw-r--r--. 2 root root 3 Dec 2 18:35 f49 -rw-r--r--. 2 root root 3 Dec 2 18:35 f5 -rw-r--r--. 2 root root 3 Dec 2 18:35 f50 -rw-r--r--. 2 root root 3 Dec 2 18:35 f6 -rw-r--r--. 2 root root 3 Dec 2 18:35 f7 -rw-r--r--. 2 root root 3 Dec 2 18:35 f8 -rw-r--r--. 2 root root 3 Dec 2 18:35 f9 /rhs/brick7/linkf_hot: total 28 drwxr-xr-x. 2 root root 161 Dec 2 18:36 cdir -rw-r--r--. 2 root root 3 Dec 2 18:35 f1 -rw-r--r--. 2 root root 3 Dec 2 18:35 f18 -rw-r--r--. 2 root root 3 Dec 2 18:35 f19 -rw-r--r--. 2 root root 3 Dec 2 18:35 f2 -rw-r--r--. 2 root root 3 Dec 2 18:35 f20 -rw-r--r--. 2 root root 3 Dec 2 18:35 f22 -rw-r--r--. 2 root root 3 Dec 2 18:35 f46 [root@zod ~]# ll /rhs/brick7/linkf_hot|wc -l [root@zod ~]# gluster v tier linkf status;gluster v rebal link status Node Promoted files Demoted files Status --------- --------- --------- --------- localhost 80 0 in progress 10.70.34.43 0 0 in progress Tiering Migration Functionality: linkf: success volume rebalance: link: failed: Volume link does not exist [root@zod ~]# gluster v tier linkf status;gluster v rebal linkf status Node Promoted files Demoted files Status --------- --------- --------- --------- localhost 80 0 in progress 10.70.34.43 0 0 in progress Tiering Migration Functionality: linkf: success Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 80 0Bytes 80 0 0 in progress 142.00 10.70.34.43 0 0Bytes 80 0 0 in progress 142.00 volume rebalance: linkf: success [root@zod ~]# gluster v tier linkf status;gluster v rebal linkf status Node Promoted files Demoted files Status --------- --------- --------- --------- localhost 80 0 in progress 10.70.34.43 0 0 in progress Tiering Migration Functionality: linkf: success Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 80 0Bytes 80 0 0 in progress 145.00 10.70.34.43 0 0Bytes 80 0 0 in progress 145.00 volume rebalance: linkf: success [root@zod ~]# llgluster bash: llgluster: command not found... [root@zod ~]# gluster v info linkf Volume Name: linkf Type: Tier Volume ID: bbc5d760-2b00-40c4-964b-71705866cafd Status: Started Number of Bricks: 8 Transport-type: tcp Hot Tier : Hot Tier Type : Distributed-Replicate Number of Bricks: 2 x 2 = 4 Brick1: yarrow:/rhs/brick6/linkf_hot Brick2: zod:/rhs/brick6/linkf_hot Brick3: yarrow:/rhs/brick7/linkf_hot Brick4: zod:/rhs/brick7/linkf_hot Cold Tier: Cold Tier Type : Distributed-Replicate Number of Bricks: 2 x 2 = 4 Brick5: zod:/rhs/brick1/linkf Brick6: yarrow:/rhs/brick1/linkf Brick7: zod:/rhs/brick2/linkf Brick8: yarrow:/rhs/brick2/linkf Options Reconfigured: cluster.tier-mode: test features.ctr-enabled: on performance.readdir-ahead: on [root@zod ~]#
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0193.html