Bug 1285651 - [Tier]: Error: attempt to set internal xattr: trusted.ec.* [Operation not permitted]
[Tier]: Error: attempt to set internal xattr: trusted.ec.* [Operation not per...
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: tier (Show other bugs)
3.1
x86_64 Linux
unspecified Severity high
: ---
: RHGS 3.1.2
Assigned To: Ashish Pandey
Rahul Hinduja
: ZStream
Depends On:
Blocks: 1260783
  Show dependency treegraph
 
Reported: 2015-11-26 02:57 EST by Rahul Hinduja
Modified: 2016-09-17 11:43 EDT (History)
4 users (show)

See Also:
Fixed In Version: glusterfs-3.7.5-9
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-03-01 00:58:36 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Rahul Hinduja 2015-11-26 02:57:29 EST
Description of problem:
=======================

Upon demotion of the file one of nodes from each replica set continuous logs following errors;

[2015-11-26 07:48:40.761146] E [MSGID: 101044] [ec.c:1041:ec_gf_fsetxattr] 0-tiervolume-disperse-0: attempt to set internal xattr: trusted.ec.* [Operation not permitted]
[2015-11-26 07:48:40.761480] W [MSGID: 109023] [dht-rebalance.c:592:__dht_rebalance_create_dst_file] 0-tiervolume-tier-dht: //dd.150: failed to set xattr on tiervolume-cold-dht (Operation not permitted)
[2015-11-26 07:48:44.972013] E [MSGID: 101044] [ec.c:1024:ec_gf_setxattr] 0-tiervolume-disperse-0: attempt to set internal xattr: trusted.ec.* [Operation not permitted]
[2015-11-26 07:48:44.972370] W [MSGID: 109023] [dht-rebalance.c:1464:dht_migrate_file] 0-tiervolume-tier-dht: Migrate file failed://dd.150: failed to set xattr on tiervolume-cold-dht (Operation not permitted)
[2015-11-26 07:48:45.182972] E [MSGID: 101044] [ec.c:1041:ec_gf_fsetxattr] 0-tiervolume-disperse-1: attempt to set internal xattr: trusted.ec.* [Operation not permitted]
[2015-11-26 07:48:45.183331] W [MSGID: 109023] [dht-rebalance.c:592:__dht_rebalance_create_dst_file] 0-tiervolume-tier-dht: /new_dir/file.326: failed to set xattr on tiervolume-cold-dht (Operation not permitted)
[2015-11-26 07:48:47.720659] E [MSGID: 101044] [ec.c:1024:ec_gf_setxattr] 0-tiervolume-disperse-1: attempt to set internal xattr: trusted.ec.* [Operation not permitted]
[2015-11-26 07:48:47.720924] W [MSGID: 109023] [dht-rebalance.c:1464:dht_migrate_file] 0-tiervolume-tier-dht: Migrate file failed:/new_dir/file.326: failed to set xattr on tiervolume-cold-dht (Operation not permitted)
[2015-11-26 07:48:47.979652] E [MSGID: 101044] [ec.c:1041:ec_gf_fsetxattr] 0-tiervolume-disperse-0: attempt to set internal xattr: trusted.ec.* [Operation not permitted]
[2015-11-26 07:48:47.980141] W [MSGID: 109023] [dht-rebalance.c:592:__dht_rebalance_create_dst_file] 0-tiervolume-tier-dht: /etc.5/selinux/targeted/modules/active/modules/pads.pp: failed to set xattr on tiervolume-cold-dht (Operation not permitted)
[2015-11-26 07:48:48.103950] E [MSGID: 101044] [ec.c:1024:ec_gf_setxattr] 0-tiervolume-disperse-0: attempt to set internal xattr: trusted.ec.* [Operation not permitted]
[2015-11-26 07:48:48.104223] W [MSGID: 109023] [dht-rebalance.c:1464:dht_migrate_file] 0-tiervolume-tier-dht: Migrate file failed:/etc.5/selinux/targeted/modules/active/modules/pads.pp: failed to set xattr on tiervolume-cold-dht (Operation not permitted)
[2015-11-26 07:48:48.166354] E [MSGID: 101044] [ec.c:1041:ec_gf_fsetxattr] 0-tiervolume-disperse-0: attempt to set internal xattr: trusted.ec.* [Operation not permitted]
[2015-11-26 07:48:48.166648] W [MSGID: 109023] [dht-rebalance.c:592:__dht_rebalance_create_dst_file] 0-tiervolume-tier-dht: /etc.8/selinux/targeted/modules/active/modules/udev.pp: failed to set xattr on tiervolume-cold-dht (Operation not permitted)
[2015-11-26 07:48:48.341027] E [MSGID: 101044] [ec.c:1024:ec_gf_setxattr] 0-tiervolume-disperse-0: attempt to set internal xattr: trusted.ec.* [Operation not permitted]
[2015-11-26 07:48:48.341379] W [MSGID: 109023] [dht-rebalance.c:1464:dht_migrate_file] 0-tiervolume-tier-dht: Migrate file failed:/etc.8/selinux/targeted/modules/active/modules/udev.pp: failed to set xattr on tiervolume-cold-dht (Operation not permitted)
[2015-11-26 07:48:49.168693] E [MSGID: 101044] [ec.c:1041:ec_gf_fsetxattr] 0-tiervolume-disperse-0: attempt to set internal xattr: trusted.ec.* [Operation not permitted]
[2015-11-26 07:48:49.169633] W [MSGID: 109023] [dht-rebalance.c:592:__dht_rebalance_create_dst_file] 0-tiervolume-tier-dht: /etc.5/selinux/targeted/modules/active/modules/ipa.pp: failed to set xattr on tiervolume-cold-dht (Operation not permitted)


While, the file is actually moved to cold tier and extended attributes are set as follows:

[root@dhcp37-165 tiervolume_ct-disp1]# ls etc.8/selinux/targeted/modules/active/modules/udev.pp
etc.8/selinux/targeted/modules/active/modules/udev.pp
[root@dhcp37-165 tiervolume_ct-disp1]# getfattr -d -e hex -m . etc.8/selinux/targeted/modules/active/modules/udev.pp
# file: etc.8/selinux/targeted/modules/active/modules/udev.pp
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.bit-rot.version=0x02000000000000005656aa28000303c3
trusted.ec.config=0x0000080602000200
trusted.ec.size=0x0000000000004596
trusted.ec.version=0x00000000000000020000000000000007
trusted.gfid=0x28703cae38b6472d972b9cd2197c883c
trusted.glusterfs.quota.f7d67fe6-b460-4023-aa07-c6b24968087d.contri=0x00000000000012000000000000000001
trusted.pgfid.f7d67fe6-b460-4023-aa07-c6b24968087d=0x00000001

[root@dhcp37-165 tiervolume_ct-disp1]# 



No Failures:
============


[root@dhcp37-165 tiervolume_ct-disp1]# gluster volume rebal tiervolume status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                8        0Bytes         63801             0             0          in progress            4067.00
                            10.70.37.133              838        0Bytes         26591             0             0          in progress            4067.00
                            10.70.37.160                7        0Bytes         64706             0             0          in progress            4067.00
                            10.70.37.158              799        0Bytes         28082             0             0          in progress            4067.00
                            10.70.37.110               24        0Bytes         64762             0             0          in progress            4067.00
                            10.70.37.155              849        0Bytes         27920             0             0          in progress            4067.00
                             10.70.37.99                8        0Bytes         65781             0             0          in progress            4067.00
                             10.70.37.88              893        0Bytes         28790             0             0          in progress            4067.00
                            10.70.37.112               18        0Bytes         62581             0             0          in progress            4067.00
                            10.70.37.199              786        0Bytes         27834             0             0          in progress            4067.00
                            10.70.37.162               16        0Bytes         65608             0             0          in progress            4067.00
                             10.70.37.87              950        0Bytes         28716             0             0          in progress            4067.00
volume rebalance: tiervolume: success
[root@dhcp37-165 tiervolume_ct-disp1]# gluster volume rebal tiervolume tier status
Node                 Promoted files       Demoted files        Status              
---------            ---------            ---------            ---------           
localhost            0                    0                    in progress         
10.70.37.133         0                    839                  in progress         
10.70.37.160         0                    0                    in progress         
10.70.37.158         0                    801                  in progress         
10.70.37.110         0                    0                    in progress         
10.70.37.155         0                    852                  in progress         
10.70.37.99          0                    0                    in progress         
10.70.37.88          0                    900                  in progress         
10.70.37.112         0                    0                    in progress         
10.70.37.199         0                    788                  in progress         
10.70.37.162         0                    0                    in progress         
10.70.37.87          0                    955                  in progress         
Tiering Migration Functionality: tiervolume: success
[root@dhcp37-165 tiervolume_ct-disp1]# 


Version-Release number of selected component (if applicable):
=============================================================

glusterfs-3.7.5-7.el7rhgs.x86_64

How reproducible:
=================

1/1


Steps to Reproduce:
===================
1. Created 12 node cluster
2. Create tiered volume with Hot tier as (6 x 2) and Cold tier as (2 x (4 + 2) = 16). Set the watermark level low and hi to 1 and 2 respectively
3. Fuse Mount the volume on 3 clients RHEL7.2,RHEl7.1 and RHEL6.7
4. Start creating data from each client:

Client 1:
=========
[root@dj ~]# crefi --multi -n 10 -b 10 -d 10 --max=1024k --min=5k --random -T 5 -t text -I 5 --fop=create /mnt/fuse/

Client 2:
=========
[root@mia ~]# cd /mnt/fuse/
[root@mia fuse]# for i in {1..10}; do cp -rf /etc etc.$i ; sleep 100 ; done

Client 3:
=========
[root@wingo fuse]# for i in {1..999}; do dd if=/dev/zero of=dd.$i bs=1M count=1 ; sleep 10 ; done

Actual results:
===============

Migration should succeed without logging error

Expected results:
=================

Migration failure and setting xattr errors are reported for each file attempted to migrate.
Comment 6 Rahul Hinduja 2015-12-04 06:33:17 EST
Verified with build: glusterfs-3.7.5-9.el7rhgs.x86_64

No related errors reported. Moving this bug to verified state. 

[root@dhcp37-165 scripts]# gluster volume rebal tiervolume tier status
Node                 Promoted files       Demoted files        Status              
---------            ---------            ---------            ---------           
localhost            42                   0                    in progress         
10.70.37.133         0                    1085                 in progress         
10.70.37.160         0                    0                    in progress         
10.70.37.158         0                    1127                 in progress         
10.70.37.110         0                    0                    in progress         
10.70.37.155         0                    1116                 in progress         
10.70.37.99          38                   0                    in progress         
10.70.37.88          0                    1041                 in progress         
10.70.37.112         0                    0                    in progress         
10.70.37.199         0                    984                  in progress         
10.70.37.162         0                    0                    in progress         
10.70.37.87          0                    1120                 in progress         
Tiering Migration Functionality: tiervolume: success
[root@dhcp37-165 scripts]# 

[root@dhcp37-165 glusterfs]# grep -i "ec_gf_fsetxattr" tiervolume-tier.log
[root@dhcp37-165 glusterfs]# grep -i "dht_rebalance_create_dst_file" tiervolume-tier.log
[root@dhcp37-165 glusterfs]# 

[root@dhcp37-133 glusterfs]# grep -i "ec_gf_fsetxattr" tiervolume-tier.log
[root@dhcp37-133 glusterfs]# grep -i "dht_rebalance_create_dst_file" tiervolume-tier.log
[root@dhcp37-133 glusterfs]# 

[root@dhcp37-165 glusterfs]# grep -i " E " tiervolume-tier.log
[root@dhcp37-165 glusterfs]# 

[root@dhcp37-133 glusterfs]# grep -i " E " tiervolume-tier.log
[root@dhcp37-133 glusterfs]#
Comment 8 errata-xmlrpc 2016-03-01 00:58:36 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0193.html

Note You need to log in before you can comment on or make changes to this bug.