Bug 1277631

Summary:	tiering: Error message "E [MSGID: 109037] [tier.c:1488:tier_start] 0-testvol-tier-dht: Demotion failed" being logged, even when there are not files to demote
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Anil Shah <ashah>
Component:	tier	Assignee:	Mohamed Ashiq <mliyazud>
Status:	CLOSED WORKSFORME	QA Contact:	Nag Pavan Chilakam <nchilaka>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.1	CC:	asrivast, mliyazud, rhs-bugs, storage-qa-internal, vagarwal
Target Milestone:	---	Keywords:	ZStream
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-11-27 15:15:00 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1260923

Description Anil Shah 2015-11-03 17:02:31 UTC

Description of problem:

logs file is getting filled up log message "E [MSGID: 109037] [tier.c:1488:tier_start] 0-testvol-tier-dht: Demotion failed", even when this are no files to demote. 

Version-Release number of selected component (if applicable):

[root@localhost ec01]# rpm -qa | grep glusterfs
glusterfs-client-xlators-3.7.5-5.el7rhgs.x86_64
glusterfs-api-3.7.5-5.el7rhgs.x86_64
glusterfs-cli-3.7.5-5.el7rhgs.x86_64
glusterfs-libs-3.7.5-5.el7rhgs.x86_64
glusterfs-3.7.5-5.el7rhgs.x86_64
glusterfs-fuse-3.7.5-5.el7rhgs.x86_64
glusterfs-server-3.7.5-5.el7rhgs.x86_64
glusterfs-geo-replication-3.7.5-5.el7rhgs.x86_64


How reproducible:

1/1

Steps to Reproduce:

1. Created 2*2 distribute replicate volume
2. Fuse mount the volume 
3. Set quota on volume 
4. Create file on the so that disk quota exceeds 
5. Now attach 2*2 distribute replicate tier
6. Modify the quota limit
7. Enable ctr, tier-promote, tier-demote frequency 
7. Create some files from mount point. 
8. Wait for demotion to happen
9. Read one of the file from Mount point
10. Wait for demotion to happen after specified time

Actual results:

Logs messaged being logged with error "Demotion filed" when there are not files to demote. 

==============================================
[2015-11-03 22:27:45.288750] E [MSGID: 109037] [tier.c:463:tier_migrate_using_query_file] 0-testvol-tier-dht: ERROR in current lookup

[2015-11-03 22:27:45.289167] E [MSGID: 109037] [tier.c:1488:tier_start] 0-testvol-tier-dht: Demotion failed
=============================================
Expected results:

There should not be any log messages.

Additional info:

[root@localhost ec01]# gluster v info
 
Volume Name: testvol
Type: Tier
Volume ID: fbee6a2e-39ef-4388-8239-8a148dafdba9
Status: Started
Number of Bricks: 8
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick1: 10.70.47.3:/rhs/brick2/ec04
Brick2: 10.70.47.2:/rhs/brick2/ec03
Brick3: 10.70.47.145:/rhs/brick2/ec02
Brick4: 10.70.47.143:/rhs/brick2/ec01
Cold Tier:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick5: 10.70.47.143:/rhs/brick1/b01
Brick6: 10.70.47.145:/rhs/brick1/b02
Brick7: 10.70.47.2:/rhs/brick1/b03
Brick8: 10.70.47.3:/rhs/brick1/b04
Options Reconfigured:
features.barrier: disable
cluster.tier-promote-frequency: 45
cluster.tier-demote-frequency: 45
cluster.write-freq-threshold: 0
cluster.read-freq-threshold: 0

Comment 2 Anil Shah 2015-11-04 08:06:49 UTC

Sos reports uploaded @
http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1277627/

Comment 4 Mohamed Ashiq 2015-11-23 11:09:32 UTC

I tried to reproduce this bug by following the above steps and couldn't find the logs.

can you please specify the way to reproduce this bug.

Comment 5 Mohamed Ashiq 2015-11-26 06:14:58 UTC

I am able to reproduce this bug in glusterfs-3.7.5-5 with the above steps, but I am not able to reproduce this bug in latest build glusterfs-3.7.5-7. I am trying to find the root cause for the problem and why it is not reproducible in latest.

Comment 6 Mohamed Ashiq 2015-11-27 15:15:00 UTC

(In reply to Mohamed Ashiq from comment #5)
> I am able to reproduce this bug in glusterfs-3.7.5-5 with the above steps,
> but I am not able to reproduce this bug in latest build glusterfs-3.7.5-7. I
> am trying to find the root cause for the problem and why it is not
> reproducible in latest.

I was not able to reproduce the bug in 3.7.5-5. Although, I have mentioned in my previous comment that I am able to reproduce the bug, but after looking at the logs, I realized it is due to

[2015-11-26 09:14:35.087162] W [MSGID: 114031] [client-rpc-fops.c:2262:client3_3_ipc_cbk] 0-vol1-client-4: remote operation failed [Transport endpoint is not connected]

which is because some of my nodes in the cluster went down. After getting nodes up, I am not able to reproduce this bug.

After discussing the same with QE, I am closing the bug now.