Bug 1277659

Summary:	lookup and set xattr fails when bit rot is enabled on a tiered volume.
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	RamaKasturi <knarra>
Component:	tier	Assignee:	Joseph Elwin Fernandes <josferna>
Status:	CLOSED ERRATA	QA Contact:	RamaKasturi <knarra>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.1	CC:	aspandey, dlambrig, nchilaka, rcyriac, rhs-bugs, sankarshan, storage-qa-internal
Target Milestone:	---	Keywords:	ZStream
Target Release:	RHGS 3.1.2
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.7.5-9	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-03-01 05:50:51 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1260783, 1260923

Description RamaKasturi 2015-11-03 18:53:41 UTC

Description of problem:
I see following errors in the tiered logs when bit rot is enabled on a tiered volume because  of lookup and set xattr on a file fails.

[2015-11-03 15:42:00.750401] E [MSGID: 109037] [tier.c:463:tier_migrate_using_query_file] 0-vol_ec-tier-dht: ERROR in current lookup

[2015-11-03 15:42:03.031577] W [MSGID: 114031] [client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol_ec-client-13: remote operation failed [Permissio
n denied]
[2015-11-03 15:48:00.862598] E [MSGID: 109037] [tier.c:463:tier_migrate_using_query_file] 0-vol_ec-tier-dht: ERROR in current lookup

[2015-11-03 15:48:00.895728] E [MSGID: 109037] [tier.c:1488:tier_start] 0-vol_ec-tier-dht: Demotion failed
[2015-11-03 15:48:00.889681] E [MSGID: 109037] [tier.c:463:tier_migrate_using_query_file] 0-vol_ec-tier-dht: ERROR in current lookup

[2015-11-03 15:54:01.244214] E [MSGID: 109037] [tier.c:463:tier_migrate_using_query_file] 0-vol_ec-tier-dht: ERROR in current lookup

[2015-11-03 15:54:01.247754] E [MSGID: 109037] [tier.c:1488:tier_start] 0-vol_ec-tier-dht: Demotion failed
[2015-11-03 15:54:01.870459] W [MSGID: 114031] [client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol_ec-client-13: remote operation failed [Permissio
n denied]
[2015-11-03 15:54:01.870510] W [MSGID: 114031] [client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol_ec-client-12: remote operation failed [Permissio
n denied]
[2015-11-03 15:54:01.870846] W [MSGID: 109023] [dht-rebalance.c:591:__dht_rebalance_create_dst_file] 0-vol_ec-tier-dht: /f2: failed to set xattr on vo
l_ec-hot-dht (Permission denied)
[2015-11-03 15:54:01.941675] W [MSGID: 114031] [client-rpc-fops.c:1085:client3_3_setxattr_cbk] 0-vol_ec-client-13: remote operation failed [Permission
 denied]
[2015-11-03 15:54:01.941779] W [MSGID: 114031] [client-rpc-fops.c:1085:client3_3_setxattr_cbk] 0-vol_ec-client-12: remote operation failed [Permission
 denied]
[2015-11-03 15:54:01.942102] W [MSGID: 109023] [dht-rebalance.c:1462:dht_migrate_file] 0-vol_ec-tier-dht: Migrate file failed:/f2: failed to set xattr
 on vol_ec-hot-dht (Permission denied)
[2015-11-03 15:54:02.186818] W [MSGID: 114031] [client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol_ec-client-12: remote operation failed [Permissio
n denied]
[2015-11-03 15:54:02.187079] W [MSGID: 109023] [dht-rebalance.c:591:__dht_rebalance_create_dst_file] 0-vol_ec-tier-dht: /f1: failed to set xattr on vo
l_ec-hot-dht (Permission denied)
[2015-11-03 15:54:02.261421] W [MSGID: 114031] [client-rpc-fops.c:1085:client3_3_setxattr_cbk] 0-vol_ec-client-13: remote operation failed [Permission
 denied]
[2015-11-03 15:54:02.261534] W [MSGID: 114031] [client-rpc-fops.c:1085:client3_3_setxattr_cbk] 0-vol_ec-client-12: remote operation failed [Permission
 denied]
[2015-11-03 15:54:02.261842] W [MSGID: 109023] [dht-rebalance.c:1462:dht_migrate_file] 0-vol_ec-tier-dht: Migrate file failed:/f1: failed to set xattr
 on vol_ec-hot-dht (Permission denied)


Version-Release number of selected component (if applicable):
glusterfs-3.7.5-5.el6rhs.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Create a tiered volume
2. Now enable bit rot on the volume
3. Mount the volume using fuse mount and create some file.
4. Wait for the bit rot to sign the files and promotion / demotion to happen.

Actual results:
I see lot of permission denied, ERROR in current lookup, failed to set xattr , remote operation failed errors in the logs.

Expected results:
All the above error should not be seen and lookup and set xattr should be successful.

Additional info:

Comment 2 RamaKasturi 2015-11-03 18:55:09 UTC

sos reports can be found at http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1277368/

Comment 4 Ashish Pandey 2015-12-07 09:37:48 UTC

https://code.engineering.redhat.com/gerrit/#/c/62104/

Comment 5 RamaKasturi 2015-12-08 12:06:13 UTC

verified and works fine with build glusterfs-3.7.5-9.el7rhgs.x86_64.

Followed the steps below and do not see any errors related to set xattr failing / permission denied in tier logs.
1. Create a tiered volume
2. Now enable bit rot on the volume
3. Mount the volume using fuse mount and create some file.
4. Wait for the bit rot to sign the files and promotion / demotion to happen.

Attaching the logs here which have been used to verify this bug.

http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1277659/

Comment 7 errata-xmlrpc 2016-03-01 05:50:51 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0193.html