Bug 1287104

Summary:	Files are not getting demoted when the replica bricks are down
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	RajeshReddy <rmekala>
Component:	tier	Assignee:	Joseph Elwin Fernandes <josferna>
Status:	CLOSED NOTABUG	QA Contact:	Nag Pavan Chilakam <nchilaka>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.1	CC:	josferna, mzywusko, rhs-bugs, sankarshan, storage-qa-internal, vagarwal
Target Milestone:	---	Keywords:	ZStream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-12-08 18:31:42 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1260783

Description RajeshReddy 2015-12-01 14:06:39 UTC

Description of problem:
===============
Files are not getting demoted when the replica bricks are down

Version-Release number of selected component (if applicable):
=====================
glusterfs-server-3.7.5-7.el7rhgs.x86_64

How reproducible:


Steps to Reproduce:
==============
1.Create 1x2 volume and then attach 1x2 hot tier and mount it on client using FUSE 
2. Create a directory and before creation of file on the mount bring down all bricks on rh-client19 server 
3.After creation of file waited for couple of hours but still file in hot tier and not demoted to cold tier 

Actual results:



Expected results:
=========
Files should be demoted 


Additional info:
=======
[root@rhs-client18 data4_big]# gluster vol status afr2x2_tier 
Status of volume: afr2x2_tier
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick rhs-client19.lab.eng.blr.redhat.com:/
rhs/brick6/tier1                            N/A       N/A        N       N/A  
Brick rhs-client18.lab.eng.blr.redhat.com:/
rhs/brick6/tier1                            49169     0          Y       5607 
Cold Bricks:
Brick rhs-client18.lab.eng.blr.redhat.com:/
rhs/brick7/afr2x2_tier                      49167     0          Y       5615 
Brick rhs-client19.lab.eng.blr.redhat.com:/
rhs/brick7/afr2x2_tier                      N/A       N/A        N       N/A  
NFS Server on localhost                     2049      0          Y       19743
Self-heal Daemon on localhost               N/A       N/A        Y       19751
NFS Server on rhs-client19.lab.eng.blr.redh
at.com                                      2049      0          Y       13666
Self-heal Daemon on rhs-client19.lab.eng.bl
r.redhat.com                                N/A       N/A        Y       13674
 
Task Status of Volume afr2x2_tier
------------------------------------------------------------------------------
Task                 : Tier migration      
ID                   : de04b702-8c03-402c-bdba-84e094baaa37
Status               : in progress         

echo "select * from gf_file_tb; select * from gf_flink_tb;" | sqlite3 /rhs/brick6/tier1/.glusterfs/tier1.db|grep c1b3e0a9-fc49-4670-a334-bbf2a902db88;echo "==========+COLD"; echo "select * from gf_file_tb; select * from gf_flink_tb;" | sqlite3 /rhs/brick7/afr2x2_tier/.glusterfs/afr2x2_tier|grep file_10g.txt|grep c1b3e0a9-fc49-4670-a334-bbf2a902db88
 c1b3e0a9-fc49-4670-a334-bbf2a902db88|1448966910|69712|0|0|0|0|0|0|1|1
c1b3e0a9-fc49-4670-a334-bbf2a902db88|963dbdad-af6d-4f3c-9859-eef45663f65e|file_10g.txt|/data4_big/file_10g.txt|0|0



[root@rhs-client18 data4_big]# gluster vol info afr2x2_tier
 
Volume Name: afr2x2_tier
Type: Tier
Volume ID: e8d8466d-4883-465c-868d-fd4330e6049e
Status: Started
Number of Bricks: 4
Transport-type: tcp
Hot Tier :
Hot Tier Type : Replicate
Number of Bricks: 1 x 2 = 2
Brick1: rhs-client19.lab.eng.blr.redhat.com:/rhs/brick6/tier1
Brick2: rhs-client18.lab.eng.blr.redhat.com:/rhs/brick6/tier1
Cold Tier:
Cold Tier Type : Replicate
Number of Bricks: 1 x 2 = 2
Brick3: rhs-client18.lab.eng.blr.redhat.com:/rhs/brick7/afr2x2_tier
Brick4: rhs-client19.lab.eng.blr.redhat.com:/rhs/brick7/afr2x2_tier
Options Reconfigured:
cluster.tier-mode: test
cluster.metadata-self-heal: off
cluster.data-self-heal: off
cluster.self-heal-daemon: on
features.ctr-enabled: on
performance.readdir-ahead: on
cluster.entry-self-heal: off

[root@rhs-client18 data4_big]# pwd
/rhs/brick6/tier1/data4_big
[root@rhs-client18 data4_big]# ls
file_10g.txt

client: (10.70.47.114)

sosreport  are available @/home/repo/sosreports/bug.1287084 on rhsqe-repo.lab.eng.blr.redhat.com

Comment 2 RajeshReddy 2015-12-07 11:03:07 UTC

Tested with glusterfs-api-3.7.5-9, and now demotions are happening though replica brick is down but demotions are not happening even after bring back the replica brick and slefheal

Comment 3 Joseph Elwin Fernandes 2015-12-07 13:22:14 UTC

(In reply to RajeshReddy from comment #2)
> Tested with glusterfs-api-3.7.5-9, and now demotions are happening though
> replica brick is down but demotions are not happening even after bring back
> the replica brick and slefheal

TYPO : Tested with glusterfs-api-3.7.5-9, and now PROMOTION are happening though replica brick is down but demotions are not happening even after bring back the replica brick and slefheal

Please set 

gluster vol set test cluster.watermark-low 1

and let me know

Comment 4 Joseph Elwin Fernandes 2015-12-07 14:27:30 UTC

Please ignore my previous comment. I tested with upstream code. Looks like working. Lets discuss more tomorrow.

Comment 5 RajeshReddy 2015-12-08 10:26:16 UTC

(In reply to RajeshReddy from comment #2)
> Tested with glusterfs-api-3.7.5-9, and now demotions are happening though
> replica brick is down but demotions are not happening even after bring back
> the replica brick and slefheal

Typo

Tested with glusterfs-api-3.7.5-9, and now demotions are happening though
 replica brick is down but promotions are not happening even after bring back
 the replica brick and slefheal

Comment 7 Joseph Elwin Fernandes 2015-12-08 15:25:15 UTC

Checked on QA set, promotion /demotion are working on set where replica are down.
Not a bug.

Comment 8 Joseph Elwin Fernandes 2015-12-08 18:31:42 UTC

As discussed with Rajesh, after having a look at the his setup, Promotion and demotions are happening as expected on a tiered volume where replicas are down.
Closing this bug as not a bug.