Bug 1276051

Summary:	Data Tiering:inconsistent linkfile creation when lookups issued on cold tier files
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Nag Pavan Chilakam <nchilaka>
Component:	tier	Assignee:	Nithya Balachandran <nbalacha>
Status:	CLOSED ERRATA	QA Contact:	Nag Pavan Chilakam <nchilaka>
Severity:	medium	Docs Contact:
Priority:	high
Version:	rhgs-3.1	CC:	asrivast, dlambrig, nbalacha, nchilaka, rhs-bugs, sankarshan, storage-qa-internal
Target Milestone:	---	Keywords:	ZStream
Target Release:	RHGS 3.1.2
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	gluster-3.5.7	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-03-01 05:47:23 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1260783, 1260923

Description Nag Pavan Chilakam 2015-10-28 14:02:58 UTC

On a fuse mount, When we do a named lookup of a file which are on cold tier, a link-to file is supposed to be created. But this doesnt happen consistently.
For eg: If I have a set of files say f1,f2,,,,f30 and i do "ls f1" , "ls f2", "ls f3"
I see link file in hot tier only for file f1

The remaining linkfile creations fails.


Workaround:Umount and remount the volume


steps to reproduce:
==================
1)create a tier volume
2)attach a tier
3)mount vol on fuse
4)create some files say f1,f2....f30
5)wait for all to get demoted 
6)Now issue a lookup using ls for each file



it can be seen only the first file link file creation passes

Comment 2 Nithya Balachandran 2015-11-06 12:19:47 UTC

It sounds like the lookups are hitting the revalidate code path in the lookup here. That would not create a linkto file.

Comment 4 Dan Lambright 2015-11-24 20:25:58 UTC

With the latest code, the behavior has changed:

New files created on tiered volumes always should create a T file on the cold sub volume, and the data on the hot sub volume (with the exception for when the hot sub volume is full).  

Legacy files, i.e. those that existed before the tier was attached, should create a T file on promotion, if I understand the inner DHT workings. However, See Nithya's comment about not creating a T file on revalidate (expected behavior); the DHT logic is subtle and the rules may not always be what we think. It may not be valid to assume a T file always will exist for legacy files.

I ran a test:

1. created 10 legacy files on normal volume
2. attach tier in test mode
3. touched  files
4. waited for promotion
5. observed T files created on cold tier.

Note also, the T file itself is an internal abstraction that is not customer facing.

Given those points, we should revise this bug to confirm it needs to be open or not.

Comment 5 Nag Pavan Chilakam 2015-12-02 13:10:59 UTC

On 3.7.5-7 I tested and found the hashed subvol is now cold tier.
hence linkfiles in hot tier will not be created (unless disk is full, for which another bug is there)
Hence this problem is solved


[root@zod ~]# ll /*/brick*/linkf*
/rhs/brick1/linkf:
total 8
drwxr-xr-x. 2 root root 8192 Dec  2 18:35 cdir
---------T. 2 root root    0 Dec  2 18:36 f12
---------T. 2 root root    0 Dec  2 18:36 f13
---------T. 2 root root    0 Dec  2 18:36 f16
---------T. 2 root root    0 Dec  2 18:36 f17
---------T. 2 root root    0 Dec  2 18:36 f21
---------T. 2 root root    0 Dec  2 18:36 f24
---------T. 2 root root    0 Dec  2 18:36 f25
---------T. 2 root root    0 Dec  2 18:36 f26
---------T. 2 root root    0 Dec  2 18:36 f27
---------T. 2 root root    0 Dec  2 18:36 f28
---------T. 2 root root    0 Dec  2 18:36 f30
---------T. 2 root root    0 Dec  2 18:36 f32
---------T. 2 root root    0 Dec  2 18:36 f34
---------T. 2 root root    0 Dec  2 18:36 f35
---------T. 2 root root    0 Dec  2 18:36 f36
---------T. 2 root root    0 Dec  2 18:36 f39
---------T. 2 root root    0 Dec  2 18:36 f4
---------T. 2 root root    0 Dec  2 18:36 f40
---------T. 2 root root    0 Dec  2 18:36 f41
---------T. 2 root root    0 Dec  2 18:36 f43
---------T. 2 root root    0 Dec  2 18:36 f44
---------T. 2 root root    0 Dec  2 18:36 f48
---------T. 2 root root    0 Dec  2 18:36 f49
---------T. 2 root root    0 Dec  2 18:36 f50
---------T. 2 root root    0 Dec  2 18:36 f6
---------T. 2 root root    0 Dec  2 18:36 f7
---------T. 2 root root    0 Dec  2 18:36 f8
---------T. 2 root root    0 Dec  2 18:36 f9

/rhs/brick2/linkf:
total 8
drwxr-xr-x. 2 root root 8192 Dec  2 18:35 cdir
---------T. 2 root root    0 Dec  2 18:36 f1
---------T. 2 root root    0 Dec  2 18:36 f10
---------T. 2 root root    0 Dec  2 18:36 f11
---------T. 2 root root    0 Dec  2 18:36 f14
---------T. 2 root root    0 Dec  2 18:36 f15
---------T. 2 root root    0 Dec  2 18:36 f18
---------T. 2 root root    0 Dec  2 18:36 f19
---------T. 2 root root    0 Dec  2 18:36 f2
---------T. 2 root root    0 Dec  2 18:36 f20
---------T. 2 root root    0 Dec  2 18:36 f22
---------T. 2 root root    0 Dec  2 18:36 f23
---------T. 2 root root    0 Dec  2 18:36 f29
---------T. 2 root root    0 Dec  2 18:36 f3
---------T. 2 root root    0 Dec  2 18:36 f31
---------T. 2 root root    0 Dec  2 18:36 f33
---------T. 2 root root    0 Dec  2 18:36 f37
---------T. 2 root root    0 Dec  2 18:36 f38
---------T. 2 root root    0 Dec  2 18:36 f42
---------T. 2 root root    0 Dec  2 18:36 f45
---------T. 2 root root    0 Dec  2 18:36 f46
---------T. 2 root root    0 Dec  2 18:36 f47
---------T. 2 root root    0 Dec  2 18:36 f5

/rhs/brick6/linkf_hot:
total 180
drwxr-xr-x. 2 root root 8192 Dec  2 18:36 cdir
-rw-r--r--. 2 root root    3 Dec  2 18:35 f10
-rw-r--r--. 2 root root    3 Dec  2 18:35 f11
-rw-r--r--. 2 root root    3 Dec  2 18:35 f12
-rw-r--r--. 2 root root    3 Dec  2 18:35 f13
-rw-r--r--. 2 root root    3 Dec  2 18:35 f14
-rw-r--r--. 2 root root    3 Dec  2 18:35 f15
-rw-r--r--. 2 root root    3 Dec  2 18:35 f16
-rw-r--r--. 2 root root    3 Dec  2 18:35 f17
-rw-r--r--. 2 root root    3 Dec  2 18:35 f21
-rw-r--r--. 2 root root    3 Dec  2 18:35 f23
-rw-r--r--. 2 root root    3 Dec  2 18:35 f24
-rw-r--r--. 2 root root    3 Dec  2 18:35 f25
-rw-r--r--. 2 root root    3 Dec  2 18:35 f26
-rw-r--r--. 2 root root    3 Dec  2 18:35 f27
-rw-r--r--. 2 root root    3 Dec  2 18:35 f28
-rw-r--r--. 2 root root    3 Dec  2 18:35 f29
-rw-r--r--. 2 root root    3 Dec  2 18:35 f3
-rw-r--r--. 2 root root    3 Dec  2 18:35 f30
-rw-r--r--. 2 root root    3 Dec  2 18:35 f31
-rw-r--r--. 2 root root    3 Dec  2 18:35 f32
-rw-r--r--. 2 root root    3 Dec  2 18:35 f33
-rw-r--r--. 2 root root    3 Dec  2 18:35 f34
-rw-r--r--. 2 root root    3 Dec  2 18:35 f35
-rw-r--r--. 2 root root    3 Dec  2 18:35 f36
-rw-r--r--. 2 root root    3 Dec  2 18:35 f37
-rw-r--r--. 2 root root    3 Dec  2 18:35 f38
-rw-r--r--. 2 root root    3 Dec  2 18:35 f39
-rw-r--r--. 2 root root    3 Dec  2 18:35 f4
-rw-r--r--. 2 root root    3 Dec  2 18:35 f40
-rw-r--r--. 2 root root    3 Dec  2 18:35 f41
-rw-r--r--. 2 root root    3 Dec  2 18:35 f42
-rw-r--r--. 2 root root    3 Dec  2 18:35 f43
-rw-r--r--. 2 root root    3 Dec  2 18:35 f44
-rw-r--r--. 2 root root    3 Dec  2 18:35 f45
-rw-r--r--. 2 root root    3 Dec  2 18:35 f47
-rw-r--r--. 2 root root    3 Dec  2 18:35 f48
-rw-r--r--. 2 root root    3 Dec  2 18:35 f49
-rw-r--r--. 2 root root    3 Dec  2 18:35 f5
-rw-r--r--. 2 root root    3 Dec  2 18:35 f50
-rw-r--r--. 2 root root    3 Dec  2 18:35 f6
-rw-r--r--. 2 root root    3 Dec  2 18:35 f7
-rw-r--r--. 2 root root    3 Dec  2 18:35 f8
-rw-r--r--. 2 root root    3 Dec  2 18:35 f9

/rhs/brick7/linkf_hot:
total 28
drwxr-xr-x. 2 root root 161 Dec  2 18:36 cdir
-rw-r--r--. 2 root root   3 Dec  2 18:35 f1
-rw-r--r--. 2 root root   3 Dec  2 18:35 f18
-rw-r--r--. 2 root root   3 Dec  2 18:35 f19
-rw-r--r--. 2 root root   3 Dec  2 18:35 f2
-rw-r--r--. 2 root root   3 Dec  2 18:35 f20
-rw-r--r--. 2 root root   3 Dec  2 18:35 f22
-rw-r--r--. 2 root root   3 Dec  2 18:35 f46
[root@zod ~]# ll /rhs/brick7/linkf_hot|wc -l




[root@zod ~]# gluster v tier linkf status;gluster v rebal link status
Node                 Promoted files       Demoted files        Status              
---------            ---------            ---------            ---------           
localhost            80                   0                    in progress         
10.70.34.43          0                    0                    in progress         
Tiering Migration Functionality: linkf: success
volume rebalance: link: failed: Volume link does not exist
[root@zod ~]# gluster v tier linkf status;gluster v rebal linkf status
Node                 Promoted files       Demoted files        Status              
---------            ---------            ---------            ---------           
localhost            80                   0                    in progress         
10.70.34.43          0                    0                    in progress         
Tiering Migration Functionality: linkf: success
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost               80        0Bytes            80             0             0          in progress             142.00
                             10.70.34.43                0        0Bytes            80             0             0          in progress             142.00
volume rebalance: linkf: success
[root@zod ~]# gluster v tier linkf status;gluster v rebal linkf status
Node                 Promoted files       Demoted files        Status              
---------            ---------            ---------            ---------           
localhost            80                   0                    in progress         
10.70.34.43          0                    0                    in progress         
Tiering Migration Functionality: linkf: success
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost               80        0Bytes            80             0             0          in progress             145.00
                             10.70.34.43                0        0Bytes            80             0             0          in progress             145.00
volume rebalance: linkf: success
[root@zod ~]# llgluster
bash: llgluster: command not found...
[root@zod ~]# gluster v info linkf
 
Volume Name: linkf
Type: Tier
Volume ID: bbc5d760-2b00-40c4-964b-71705866cafd
Status: Started
Number of Bricks: 8
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick1: yarrow:/rhs/brick6/linkf_hot
Brick2: zod:/rhs/brick6/linkf_hot
Brick3: yarrow:/rhs/brick7/linkf_hot
Brick4: zod:/rhs/brick7/linkf_hot
Cold Tier:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick5: zod:/rhs/brick1/linkf
Brick6: yarrow:/rhs/brick1/linkf
Brick7: zod:/rhs/brick2/linkf
Brick8: yarrow:/rhs/brick2/linkf
Options Reconfigured:
cluster.tier-mode: test
features.ctr-enabled: on
performance.readdir-ahead: on
[root@zod ~]#

Comment 7 errata-xmlrpc 2016-03-01 05:47:23 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0193.html