Description of problem: I see following errors in the tiered logs when bit rot is enabled on a tiered volume because of lookup and set xattr on a file fails. [2015-11-03 15:42:00.750401] E [MSGID: 109037] [tier.c:463:tier_migrate_using_query_file] 0-vol_ec-tier-dht: ERROR in current lookup [2015-11-03 15:42:03.031577] W [MSGID: 114031] [client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol_ec-client-13: remote operation failed [Permissio n denied] [2015-11-03 15:48:00.862598] E [MSGID: 109037] [tier.c:463:tier_migrate_using_query_file] 0-vol_ec-tier-dht: ERROR in current lookup [2015-11-03 15:48:00.895728] E [MSGID: 109037] [tier.c:1488:tier_start] 0-vol_ec-tier-dht: Demotion failed [2015-11-03 15:48:00.889681] E [MSGID: 109037] [tier.c:463:tier_migrate_using_query_file] 0-vol_ec-tier-dht: ERROR in current lookup [2015-11-03 15:54:01.244214] E [MSGID: 109037] [tier.c:463:tier_migrate_using_query_file] 0-vol_ec-tier-dht: ERROR in current lookup [2015-11-03 15:54:01.247754] E [MSGID: 109037] [tier.c:1488:tier_start] 0-vol_ec-tier-dht: Demotion failed [2015-11-03 15:54:01.870459] W [MSGID: 114031] [client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol_ec-client-13: remote operation failed [Permissio n denied] [2015-11-03 15:54:01.870510] W [MSGID: 114031] [client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol_ec-client-12: remote operation failed [Permissio n denied] [2015-11-03 15:54:01.870846] W [MSGID: 109023] [dht-rebalance.c:591:__dht_rebalance_create_dst_file] 0-vol_ec-tier-dht: /f2: failed to set xattr on vo l_ec-hot-dht (Permission denied) [2015-11-03 15:54:01.941675] W [MSGID: 114031] [client-rpc-fops.c:1085:client3_3_setxattr_cbk] 0-vol_ec-client-13: remote operation failed [Permission denied] [2015-11-03 15:54:01.941779] W [MSGID: 114031] [client-rpc-fops.c:1085:client3_3_setxattr_cbk] 0-vol_ec-client-12: remote operation failed [Permission denied] [2015-11-03 15:54:01.942102] W [MSGID: 109023] [dht-rebalance.c:1462:dht_migrate_file] 0-vol_ec-tier-dht: Migrate file failed:/f2: failed to set xattr on vol_ec-hot-dht (Permission denied) [2015-11-03 15:54:02.186818] W [MSGID: 114031] [client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol_ec-client-12: remote operation failed [Permissio n denied] [2015-11-03 15:54:02.187079] W [MSGID: 109023] [dht-rebalance.c:591:__dht_rebalance_create_dst_file] 0-vol_ec-tier-dht: /f1: failed to set xattr on vo l_ec-hot-dht (Permission denied) [2015-11-03 15:54:02.261421] W [MSGID: 114031] [client-rpc-fops.c:1085:client3_3_setxattr_cbk] 0-vol_ec-client-13: remote operation failed [Permission denied] [2015-11-03 15:54:02.261534] W [MSGID: 114031] [client-rpc-fops.c:1085:client3_3_setxattr_cbk] 0-vol_ec-client-12: remote operation failed [Permission denied] [2015-11-03 15:54:02.261842] W [MSGID: 109023] [dht-rebalance.c:1462:dht_migrate_file] 0-vol_ec-tier-dht: Migrate file failed:/f1: failed to set xattr on vol_ec-hot-dht (Permission denied) Version-Release number of selected component (if applicable): glusterfs-3.7.5-5.el6rhs.x86_64 How reproducible: Always Steps to Reproduce: 1. Create a tiered volume 2. Now enable bit rot on the volume 3. Mount the volume using fuse mount and create some file. 4. Wait for the bit rot to sign the files and promotion / demotion to happen. Actual results: I see lot of permission denied, ERROR in current lookup, failed to set xattr , remote operation failed errors in the logs. Expected results: All the above error should not be seen and lookup and set xattr should be successful. Additional info:
sos reports can be found at http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1277368/
https://code.engineering.redhat.com/gerrit/#/c/62104/
verified and works fine with build glusterfs-3.7.5-9.el7rhgs.x86_64. Followed the steps below and do not see any errors related to set xattr failing / permission denied in tier logs. 1. Create a tiered volume 2. Now enable bit rot on the volume 3. Mount the volume using fuse mount and create some file. 4. Wait for the bit rot to sign the files and promotion / demotion to happen. Attaching the logs here which have been used to verify this bug. http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1277659/
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0193.html