Description of problem: ======================= Upon demotion of the file one of nodes from each replica set continuous logs following errors; [2015-11-26 07:48:40.761146] E [MSGID: 101044] [ec.c:1041:ec_gf_fsetxattr] 0-tiervolume-disperse-0: attempt to set internal xattr: trusted.ec.* [Operation not permitted] [2015-11-26 07:48:40.761480] W [MSGID: 109023] [dht-rebalance.c:592:__dht_rebalance_create_dst_file] 0-tiervolume-tier-dht: //dd.150: failed to set xattr on tiervolume-cold-dht (Operation not permitted) [2015-11-26 07:48:44.972013] E [MSGID: 101044] [ec.c:1024:ec_gf_setxattr] 0-tiervolume-disperse-0: attempt to set internal xattr: trusted.ec.* [Operation not permitted] [2015-11-26 07:48:44.972370] W [MSGID: 109023] [dht-rebalance.c:1464:dht_migrate_file] 0-tiervolume-tier-dht: Migrate file failed://dd.150: failed to set xattr on tiervolume-cold-dht (Operation not permitted) [2015-11-26 07:48:45.182972] E [MSGID: 101044] [ec.c:1041:ec_gf_fsetxattr] 0-tiervolume-disperse-1: attempt to set internal xattr: trusted.ec.* [Operation not permitted] [2015-11-26 07:48:45.183331] W [MSGID: 109023] [dht-rebalance.c:592:__dht_rebalance_create_dst_file] 0-tiervolume-tier-dht: /new_dir/file.326: failed to set xattr on tiervolume-cold-dht (Operation not permitted) [2015-11-26 07:48:47.720659] E [MSGID: 101044] [ec.c:1024:ec_gf_setxattr] 0-tiervolume-disperse-1: attempt to set internal xattr: trusted.ec.* [Operation not permitted] [2015-11-26 07:48:47.720924] W [MSGID: 109023] [dht-rebalance.c:1464:dht_migrate_file] 0-tiervolume-tier-dht: Migrate file failed:/new_dir/file.326: failed to set xattr on tiervolume-cold-dht (Operation not permitted) [2015-11-26 07:48:47.979652] E [MSGID: 101044] [ec.c:1041:ec_gf_fsetxattr] 0-tiervolume-disperse-0: attempt to set internal xattr: trusted.ec.* [Operation not permitted] [2015-11-26 07:48:47.980141] W [MSGID: 109023] [dht-rebalance.c:592:__dht_rebalance_create_dst_file] 0-tiervolume-tier-dht: /etc.5/selinux/targeted/modules/active/modules/pads.pp: failed to set xattr on tiervolume-cold-dht (Operation not permitted) [2015-11-26 07:48:48.103950] E [MSGID: 101044] [ec.c:1024:ec_gf_setxattr] 0-tiervolume-disperse-0: attempt to set internal xattr: trusted.ec.* [Operation not permitted] [2015-11-26 07:48:48.104223] W [MSGID: 109023] [dht-rebalance.c:1464:dht_migrate_file] 0-tiervolume-tier-dht: Migrate file failed:/etc.5/selinux/targeted/modules/active/modules/pads.pp: failed to set xattr on tiervolume-cold-dht (Operation not permitted) [2015-11-26 07:48:48.166354] E [MSGID: 101044] [ec.c:1041:ec_gf_fsetxattr] 0-tiervolume-disperse-0: attempt to set internal xattr: trusted.ec.* [Operation not permitted] [2015-11-26 07:48:48.166648] W [MSGID: 109023] [dht-rebalance.c:592:__dht_rebalance_create_dst_file] 0-tiervolume-tier-dht: /etc.8/selinux/targeted/modules/active/modules/udev.pp: failed to set xattr on tiervolume-cold-dht (Operation not permitted) [2015-11-26 07:48:48.341027] E [MSGID: 101044] [ec.c:1024:ec_gf_setxattr] 0-tiervolume-disperse-0: attempt to set internal xattr: trusted.ec.* [Operation not permitted] [2015-11-26 07:48:48.341379] W [MSGID: 109023] [dht-rebalance.c:1464:dht_migrate_file] 0-tiervolume-tier-dht: Migrate file failed:/etc.8/selinux/targeted/modules/active/modules/udev.pp: failed to set xattr on tiervolume-cold-dht (Operation not permitted) [2015-11-26 07:48:49.168693] E [MSGID: 101044] [ec.c:1041:ec_gf_fsetxattr] 0-tiervolume-disperse-0: attempt to set internal xattr: trusted.ec.* [Operation not permitted] [2015-11-26 07:48:49.169633] W [MSGID: 109023] [dht-rebalance.c:592:__dht_rebalance_create_dst_file] 0-tiervolume-tier-dht: /etc.5/selinux/targeted/modules/active/modules/ipa.pp: failed to set xattr on tiervolume-cold-dht (Operation not permitted) While, the file is actually moved to cold tier and extended attributes are set as follows: [root@dhcp37-165 tiervolume_ct-disp1]# ls etc.8/selinux/targeted/modules/active/modules/udev.pp etc.8/selinux/targeted/modules/active/modules/udev.pp [root@dhcp37-165 tiervolume_ct-disp1]# getfattr -d -e hex -m . etc.8/selinux/targeted/modules/active/modules/udev.pp # file: etc.8/selinux/targeted/modules/active/modules/udev.pp security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000 trusted.bit-rot.version=0x02000000000000005656aa28000303c3 trusted.ec.config=0x0000080602000200 trusted.ec.size=0x0000000000004596 trusted.ec.version=0x00000000000000020000000000000007 trusted.gfid=0x28703cae38b6472d972b9cd2197c883c trusted.glusterfs.quota.f7d67fe6-b460-4023-aa07-c6b24968087d.contri=0x00000000000012000000000000000001 trusted.pgfid.f7d67fe6-b460-4023-aa07-c6b24968087d=0x00000001 [root@dhcp37-165 tiervolume_ct-disp1]# No Failures: ============ [root@dhcp37-165 tiervolume_ct-disp1]# gluster volume rebal tiervolume status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 8 0Bytes 63801 0 0 in progress 4067.00 10.70.37.133 838 0Bytes 26591 0 0 in progress 4067.00 10.70.37.160 7 0Bytes 64706 0 0 in progress 4067.00 10.70.37.158 799 0Bytes 28082 0 0 in progress 4067.00 10.70.37.110 24 0Bytes 64762 0 0 in progress 4067.00 10.70.37.155 849 0Bytes 27920 0 0 in progress 4067.00 10.70.37.99 8 0Bytes 65781 0 0 in progress 4067.00 10.70.37.88 893 0Bytes 28790 0 0 in progress 4067.00 10.70.37.112 18 0Bytes 62581 0 0 in progress 4067.00 10.70.37.199 786 0Bytes 27834 0 0 in progress 4067.00 10.70.37.162 16 0Bytes 65608 0 0 in progress 4067.00 10.70.37.87 950 0Bytes 28716 0 0 in progress 4067.00 volume rebalance: tiervolume: success [root@dhcp37-165 tiervolume_ct-disp1]# gluster volume rebal tiervolume tier status Node Promoted files Demoted files Status --------- --------- --------- --------- localhost 0 0 in progress 10.70.37.133 0 839 in progress 10.70.37.160 0 0 in progress 10.70.37.158 0 801 in progress 10.70.37.110 0 0 in progress 10.70.37.155 0 852 in progress 10.70.37.99 0 0 in progress 10.70.37.88 0 900 in progress 10.70.37.112 0 0 in progress 10.70.37.199 0 788 in progress 10.70.37.162 0 0 in progress 10.70.37.87 0 955 in progress Tiering Migration Functionality: tiervolume: success [root@dhcp37-165 tiervolume_ct-disp1]# Version-Release number of selected component (if applicable): ============================================================= glusterfs-3.7.5-7.el7rhgs.x86_64 How reproducible: ================= 1/1 Steps to Reproduce: =================== 1. Created 12 node cluster 2. Create tiered volume with Hot tier as (6 x 2) and Cold tier as (2 x (4 + 2) = 16). Set the watermark level low and hi to 1 and 2 respectively 3. Fuse Mount the volume on 3 clients RHEL7.2,RHEl7.1 and RHEL6.7 4. Start creating data from each client: Client 1: ========= [root@dj ~]# crefi --multi -n 10 -b 10 -d 10 --max=1024k --min=5k --random -T 5 -t text -I 5 --fop=create /mnt/fuse/ Client 2: ========= [root@mia ~]# cd /mnt/fuse/ [root@mia fuse]# for i in {1..10}; do cp -rf /etc etc.$i ; sleep 100 ; done Client 3: ========= [root@wingo fuse]# for i in {1..999}; do dd if=/dev/zero of=dd.$i bs=1M count=1 ; sleep 10 ; done Actual results: =============== Migration should succeed without logging error Expected results: ================= Migration failure and setting xattr errors are reported for each file attempted to migrate.
Verified with build: glusterfs-3.7.5-9.el7rhgs.x86_64 No related errors reported. Moving this bug to verified state. [root@dhcp37-165 scripts]# gluster volume rebal tiervolume tier status Node Promoted files Demoted files Status --------- --------- --------- --------- localhost 42 0 in progress 10.70.37.133 0 1085 in progress 10.70.37.160 0 0 in progress 10.70.37.158 0 1127 in progress 10.70.37.110 0 0 in progress 10.70.37.155 0 1116 in progress 10.70.37.99 38 0 in progress 10.70.37.88 0 1041 in progress 10.70.37.112 0 0 in progress 10.70.37.199 0 984 in progress 10.70.37.162 0 0 in progress 10.70.37.87 0 1120 in progress Tiering Migration Functionality: tiervolume: success [root@dhcp37-165 scripts]# [root@dhcp37-165 glusterfs]# grep -i "ec_gf_fsetxattr" tiervolume-tier.log [root@dhcp37-165 glusterfs]# grep -i "dht_rebalance_create_dst_file" tiervolume-tier.log [root@dhcp37-165 glusterfs]# [root@dhcp37-133 glusterfs]# grep -i "ec_gf_fsetxattr" tiervolume-tier.log [root@dhcp37-133 glusterfs]# grep -i "dht_rebalance_create_dst_file" tiervolume-tier.log [root@dhcp37-133 glusterfs]# [root@dhcp37-165 glusterfs]# grep -i " E " tiervolume-tier.log [root@dhcp37-165 glusterfs]# [root@dhcp37-133 glusterfs]# grep -i " E " tiervolume-tier.log [root@dhcp37-133 glusterfs]#
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0193.html