Bug 1289071

Summary:	[Tier]: Failed to open "demotequeryfile-master-tier-dht" errors logged on the node having only cold bricks
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Rahul Hinduja <rhinduja>
Component:	tier	Assignee:	Joseph Elwin Fernandes <josferna>
Status:	CLOSED ERRATA	QA Contact:	Rahul Hinduja <rhinduja>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.1	CC:	byarlaga, josferna, rhs-bugs, sankarshan, storage-qa-internal
Target Milestone:	---	Keywords:	ZStream
Target Release:	RHGS 3.1.2
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.7.5-11	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1289578 (view as bug list)		Environment:
Last Closed:	2016-03-01 06:01:23 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1260783

Description Rahul Hinduja 2015-12-07 11:39:13 UTC

Description of problem:
=======================

Observed that the nodes which do not participate in the hot tier bricks do not have "demotequeryfile-master-tier-dht" under ls /var/run/gluster/<volname>-tier-dht/ . This looks expected as these bricks wont log any entries in this binary for demotion. 

But the tier log on these nodes constantly report below ERROR:

[2015-12-06 23:32:00.906688] E [MSGID: 109051] [tier.c:1069:tier_migrate_files_using_qfile] 0-tier: Failed to open /var/run/gluster/master-tier-dht/demotequeryfile-master-tier-dht for migration [No such file or directory]
[2015-12-06 23:32:00.906994] E [MSGID: 109037] [tier.c:1469:tier_start] 0-master-tier-dht: Demotion failed
[2015-12-06 23:34:00.992814] E [MSGID: 109051] [tier.c:1069:tier_migrate_files_using_qfile] 0-tier: Failed to open /var/run/gluster/master-tier-dht/demotequeryfile-master-tier-dht for migration [No such file or directory]
[2015-12-06 23:34:00.993072] E [MSGID: 109037] [tier.c:1469:tier_start] 0-master-tier-dht: Demotion failed

[root@dhcp37-165 glusterfs]# ls /var/run/gluster/master-tier-dht/
promotequeryfile-master-tier-dht
[root@dhcp37-165 glusterfs]# pwd
/var/log/glusterfs
[root@dhcp37-165 glusterfs]# 

This doesn't look to be genuine error and it should be handled gracefully.

Version-Release number of selected component (if applicable):
=============================================================

glusterfs-3.7.5-9.el7rhgs.x86_64


How reproducible:
=================

1/1

Steps to Reproduce:
===================
1. Create a cluster of 6 nodes
2. Create and start a tier volume in which cold tier uses bricks of all 6 nodes and hot tier uses bricks of only 4 nodes
3. Set the tier mode to test
4. Mount the volume and start creating data

Comment 3 Joseph Elwin Fernandes 2015-12-09 14:33:10 UTC

https://code.engineering.redhat.com/gerrit/#/c/63393/

Comment 4 Rahul Hinduja 2015-12-16 09:57:22 UTC

Verified with build: glusterfs-3.7.5-11.el7rhgs.x86_64

[root@dhcp37-165 master-tier-dht]# gluster volume rebalance master tier status
Node                 Promoted files       Demoted files        Status              
---------            ---------            ---------            ---------           
localhost            0                    0                    in progress         
10.70.37.133         0                    0                    in progress         
10.70.37.160         0                    0                    in progress         
10.70.37.158         0                    372                  in progress         
10.70.37.110         0                    0                    in progress         
10.70.37.155         0                    391                  in progress         
Tiering Migration Functionality: master: success
[root@dhcp37-165 master-tier-dht]# ls
promotequeryfile-master-tier-dht
[root@dhcp37-165 master-tier-dht]#

[root@dhcp37-165 glusterfs]# grep -i " E " master-tier.log 
[root@dhcp37-165 glusterfs]# 
[root@dhcp37-165 glusterfs]# grep -i " E " master-*
[root@dhcp37-165 glusterfs]# grep -i "tier_migrate_files_using_qfile" master-*
[root@dhcp37-165 glusterfs]# grep -i "demotequeryfile" master-*
master-tier.log:[2015-12-15 21:40:36.615806] I [MSGID: 109038] [tier.c:1872:tier_init] 0-master-tier-dht: Promote file /var/run/gluster/master-tier-dht/promotequeryfile-master-tier-dht demote file /var/run/gluster/master-tier-dht/demotequeryfile-master-tier-dht
[root@dhcp37-165 glusterfs]# 


No error reported, moving the bug to verified state.

Comment 6 errata-xmlrpc 2016-03-01 06:01:23 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0193.html