Bug 1285651
| Summary: | [Tier]: Error: attempt to set internal xattr: trusted.ec.* [Operation not permitted] | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Rahul Hinduja <rhinduja> |
| Component: | tier | Assignee: | Ashish Pandey <aspandey> |
| Status: | CLOSED ERRATA | QA Contact: | Rahul Hinduja <rhinduja> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | rhgs-3.1 | CC: | nbalacha, rhs-bugs, sankarshan, storage-qa-internal |
| Target Milestone: | --- | Keywords: | ZStream |
| Target Release: | RHGS 3.1.2 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | glusterfs-3.7.5-9 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-03-01 05:58:36 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1260783 | ||
Verified with build: glusterfs-3.7.5-9.el7rhgs.x86_64 No related errors reported. Moving this bug to verified state. [root@dhcp37-165 scripts]# gluster volume rebal tiervolume tier status Node Promoted files Demoted files Status --------- --------- --------- --------- localhost 42 0 in progress 10.70.37.133 0 1085 in progress 10.70.37.160 0 0 in progress 10.70.37.158 0 1127 in progress 10.70.37.110 0 0 in progress 10.70.37.155 0 1116 in progress 10.70.37.99 38 0 in progress 10.70.37.88 0 1041 in progress 10.70.37.112 0 0 in progress 10.70.37.199 0 984 in progress 10.70.37.162 0 0 in progress 10.70.37.87 0 1120 in progress Tiering Migration Functionality: tiervolume: success [root@dhcp37-165 scripts]# [root@dhcp37-165 glusterfs]# grep -i "ec_gf_fsetxattr" tiervolume-tier.log [root@dhcp37-165 glusterfs]# grep -i "dht_rebalance_create_dst_file" tiervolume-tier.log [root@dhcp37-165 glusterfs]# [root@dhcp37-133 glusterfs]# grep -i "ec_gf_fsetxattr" tiervolume-tier.log [root@dhcp37-133 glusterfs]# grep -i "dht_rebalance_create_dst_file" tiervolume-tier.log [root@dhcp37-133 glusterfs]# [root@dhcp37-165 glusterfs]# grep -i " E " tiervolume-tier.log [root@dhcp37-165 glusterfs]# [root@dhcp37-133 glusterfs]# grep -i " E " tiervolume-tier.log [root@dhcp37-133 glusterfs]# Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0193.html |
Description of problem: ======================= Upon demotion of the file one of nodes from each replica set continuous logs following errors; [2015-11-26 07:48:40.761146] E [MSGID: 101044] [ec.c:1041:ec_gf_fsetxattr] 0-tiervolume-disperse-0: attempt to set internal xattr: trusted.ec.* [Operation not permitted] [2015-11-26 07:48:40.761480] W [MSGID: 109023] [dht-rebalance.c:592:__dht_rebalance_create_dst_file] 0-tiervolume-tier-dht: //dd.150: failed to set xattr on tiervolume-cold-dht (Operation not permitted) [2015-11-26 07:48:44.972013] E [MSGID: 101044] [ec.c:1024:ec_gf_setxattr] 0-tiervolume-disperse-0: attempt to set internal xattr: trusted.ec.* [Operation not permitted] [2015-11-26 07:48:44.972370] W [MSGID: 109023] [dht-rebalance.c:1464:dht_migrate_file] 0-tiervolume-tier-dht: Migrate file failed://dd.150: failed to set xattr on tiervolume-cold-dht (Operation not permitted) [2015-11-26 07:48:45.182972] E [MSGID: 101044] [ec.c:1041:ec_gf_fsetxattr] 0-tiervolume-disperse-1: attempt to set internal xattr: trusted.ec.* [Operation not permitted] [2015-11-26 07:48:45.183331] W [MSGID: 109023] [dht-rebalance.c:592:__dht_rebalance_create_dst_file] 0-tiervolume-tier-dht: /new_dir/file.326: failed to set xattr on tiervolume-cold-dht (Operation not permitted) [2015-11-26 07:48:47.720659] E [MSGID: 101044] [ec.c:1024:ec_gf_setxattr] 0-tiervolume-disperse-1: attempt to set internal xattr: trusted.ec.* [Operation not permitted] [2015-11-26 07:48:47.720924] W [MSGID: 109023] [dht-rebalance.c:1464:dht_migrate_file] 0-tiervolume-tier-dht: Migrate file failed:/new_dir/file.326: failed to set xattr on tiervolume-cold-dht (Operation not permitted) [2015-11-26 07:48:47.979652] E [MSGID: 101044] [ec.c:1041:ec_gf_fsetxattr] 0-tiervolume-disperse-0: attempt to set internal xattr: trusted.ec.* [Operation not permitted] [2015-11-26 07:48:47.980141] W [MSGID: 109023] [dht-rebalance.c:592:__dht_rebalance_create_dst_file] 0-tiervolume-tier-dht: /etc.5/selinux/targeted/modules/active/modules/pads.pp: failed to set xattr on tiervolume-cold-dht (Operation not permitted) [2015-11-26 07:48:48.103950] E [MSGID: 101044] [ec.c:1024:ec_gf_setxattr] 0-tiervolume-disperse-0: attempt to set internal xattr: trusted.ec.* [Operation not permitted] [2015-11-26 07:48:48.104223] W [MSGID: 109023] [dht-rebalance.c:1464:dht_migrate_file] 0-tiervolume-tier-dht: Migrate file failed:/etc.5/selinux/targeted/modules/active/modules/pads.pp: failed to set xattr on tiervolume-cold-dht (Operation not permitted) [2015-11-26 07:48:48.166354] E [MSGID: 101044] [ec.c:1041:ec_gf_fsetxattr] 0-tiervolume-disperse-0: attempt to set internal xattr: trusted.ec.* [Operation not permitted] [2015-11-26 07:48:48.166648] W [MSGID: 109023] [dht-rebalance.c:592:__dht_rebalance_create_dst_file] 0-tiervolume-tier-dht: /etc.8/selinux/targeted/modules/active/modules/udev.pp: failed to set xattr on tiervolume-cold-dht (Operation not permitted) [2015-11-26 07:48:48.341027] E [MSGID: 101044] [ec.c:1024:ec_gf_setxattr] 0-tiervolume-disperse-0: attempt to set internal xattr: trusted.ec.* [Operation not permitted] [2015-11-26 07:48:48.341379] W [MSGID: 109023] [dht-rebalance.c:1464:dht_migrate_file] 0-tiervolume-tier-dht: Migrate file failed:/etc.8/selinux/targeted/modules/active/modules/udev.pp: failed to set xattr on tiervolume-cold-dht (Operation not permitted) [2015-11-26 07:48:49.168693] E [MSGID: 101044] [ec.c:1041:ec_gf_fsetxattr] 0-tiervolume-disperse-0: attempt to set internal xattr: trusted.ec.* [Operation not permitted] [2015-11-26 07:48:49.169633] W [MSGID: 109023] [dht-rebalance.c:592:__dht_rebalance_create_dst_file] 0-tiervolume-tier-dht: /etc.5/selinux/targeted/modules/active/modules/ipa.pp: failed to set xattr on tiervolume-cold-dht (Operation not permitted) While, the file is actually moved to cold tier and extended attributes are set as follows: [root@dhcp37-165 tiervolume_ct-disp1]# ls etc.8/selinux/targeted/modules/active/modules/udev.pp etc.8/selinux/targeted/modules/active/modules/udev.pp [root@dhcp37-165 tiervolume_ct-disp1]# getfattr -d -e hex -m . etc.8/selinux/targeted/modules/active/modules/udev.pp # file: etc.8/selinux/targeted/modules/active/modules/udev.pp security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000 trusted.bit-rot.version=0x02000000000000005656aa28000303c3 trusted.ec.config=0x0000080602000200 trusted.ec.size=0x0000000000004596 trusted.ec.version=0x00000000000000020000000000000007 trusted.gfid=0x28703cae38b6472d972b9cd2197c883c trusted.glusterfs.quota.f7d67fe6-b460-4023-aa07-c6b24968087d.contri=0x00000000000012000000000000000001 trusted.pgfid.f7d67fe6-b460-4023-aa07-c6b24968087d=0x00000001 [root@dhcp37-165 tiervolume_ct-disp1]# No Failures: ============ [root@dhcp37-165 tiervolume_ct-disp1]# gluster volume rebal tiervolume status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 8 0Bytes 63801 0 0 in progress 4067.00 10.70.37.133 838 0Bytes 26591 0 0 in progress 4067.00 10.70.37.160 7 0Bytes 64706 0 0 in progress 4067.00 10.70.37.158 799 0Bytes 28082 0 0 in progress 4067.00 10.70.37.110 24 0Bytes 64762 0 0 in progress 4067.00 10.70.37.155 849 0Bytes 27920 0 0 in progress 4067.00 10.70.37.99 8 0Bytes 65781 0 0 in progress 4067.00 10.70.37.88 893 0Bytes 28790 0 0 in progress 4067.00 10.70.37.112 18 0Bytes 62581 0 0 in progress 4067.00 10.70.37.199 786 0Bytes 27834 0 0 in progress 4067.00 10.70.37.162 16 0Bytes 65608 0 0 in progress 4067.00 10.70.37.87 950 0Bytes 28716 0 0 in progress 4067.00 volume rebalance: tiervolume: success [root@dhcp37-165 tiervolume_ct-disp1]# gluster volume rebal tiervolume tier status Node Promoted files Demoted files Status --------- --------- --------- --------- localhost 0 0 in progress 10.70.37.133 0 839 in progress 10.70.37.160 0 0 in progress 10.70.37.158 0 801 in progress 10.70.37.110 0 0 in progress 10.70.37.155 0 852 in progress 10.70.37.99 0 0 in progress 10.70.37.88 0 900 in progress 10.70.37.112 0 0 in progress 10.70.37.199 0 788 in progress 10.70.37.162 0 0 in progress 10.70.37.87 0 955 in progress Tiering Migration Functionality: tiervolume: success [root@dhcp37-165 tiervolume_ct-disp1]# Version-Release number of selected component (if applicable): ============================================================= glusterfs-3.7.5-7.el7rhgs.x86_64 How reproducible: ================= 1/1 Steps to Reproduce: =================== 1. Created 12 node cluster 2. Create tiered volume with Hot tier as (6 x 2) and Cold tier as (2 x (4 + 2) = 16). Set the watermark level low and hi to 1 and 2 respectively 3. Fuse Mount the volume on 3 clients RHEL7.2,RHEl7.1 and RHEL6.7 4. Start creating data from each client: Client 1: ========= [root@dj ~]# crefi --multi -n 10 -b 10 -d 10 --max=1024k --min=5k --random -T 5 -t text -I 5 --fop=create /mnt/fuse/ Client 2: ========= [root@mia ~]# cd /mnt/fuse/ [root@mia fuse]# for i in {1..10}; do cp -rf /etc etc.$i ; sleep 100 ; done Client 3: ========= [root@wingo fuse]# for i in {1..999}; do dd if=/dev/zero of=dd.$i bs=1M count=1 ; sleep 10 ; done Actual results: =============== Migration should succeed without logging error Expected results: ================= Migration failure and setting xattr errors are reported for each file attempted to migrate.