Bug 1399504 - Permission denied errors on server during rebalance
Summary: Permission denied errors on server during rebalance
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: distribute
Version: rhgs-3.2
Hardware: All
OS: Linux
Target Milestone: ---
: RHGS 3.2.0
Assignee: Susant Kumar Palai
QA Contact: Prasad Desala
Depends On:
TreeView+ depends on / blocked
Reported: 2016-11-29 08:38 UTC by Karan Sandha
Modified: 2017-01-30 10:20 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2017-01-30 10:20:37 UTC

Attachments (Terms of Use)
Permission Denied (722.36 KB, image/png)
2016-11-29 08:38 UTC, Karan Sandha
no flags Details

Description Karan Sandha 2016-11-29 08:38:18 UTC
Created attachment 1225712 [details]
Permission Denied

Description of problem:
Getting permission denied on migration of data warnigns on the server after rebalance . 

Version-Release number of selected component (if applicable):
[root@dhcp47-116 testmount]# rpm -qa | grep gluster
[root@dhcp47-116 testmount]# 

How reproducible:
Logs are placed at rhsqe-repo.lab.eng.blr.redhat.com:/var/www/html/sosreports/<bug>

Steps to Reproduce:
1. Create a 1x(2+1) arbiter volume and mount it to a client
2. Create 1000 directories and 1000 files using mkdir dir{1..1000} and touch abc{1..1000}
3. Now replace a brick from server1 with a new brick on the same server.
4. let the heal be completed and now add 3 bricks to form a new subvolume in the same volume. 
5. After the adding of bricks is successful, now trigger rebalance ; gluster volume rebalance testvol start

Actual results:
W [MSGID: 114031] [client-rpc-fops.c:1893:client3_3_fsetxattr_cbk] 0-testvol-client-3: remote operation failed [Permission denied]
[2016-11-29 06:35:01.819231] W [MSGID: 114031] [client-rpc-fops.c:1893:client3_3_fsetxattr_cbk] 0-testvol-client-5: remote operation failed [Permission denied]
[2016-11-29 06:35:01.819293] W [MSGID: 114031] [client-rpc-fops.c:1893:client3_3_fsetxattr_cbk] 0-testvol-client-4: remote operation failed [Permission denied]
[2016-11-29 06:35:01.820177] W [MSGID: 109023] [dht-rebalance.c:693:__dht_rebalance_create_dst_file] 0-testvol-dht: /abc1664: failed to set xattr on testvol-replicate-1 (Permission denied)

Expected results:

There should be no errors/warnings during rebalance.

Additional info:
tested on RHEL 6.8

Comment 5 Karan Sandha 2016-12-01 06:56:40 UTC

Yes everything was being run as root user. This test didn't involve non-root user.

Thanks & regards

Comment 7 Susant Kumar Palai 2016-12-01 09:57:34 UTC
 From brick logs I could see setxattr for key: security.selinux*

Here is log for few setxattr failures:
[2016-11-29 07:37:43.046552] E [MSGID: 113001] [posix-helpers.c:1160:posix_handle_pair] 0-testvol-posix: /bricks/brick0/testvol_brick0/.glusterfs/6e/98/6e9812a2-711e-4cb5-b452-1f11a2d08e77: key:security.selinuxflags: 0 length:35 [Permission denied]
[2016-11-29 07:37:43.046593] I [MSGID: 115060] [server-rpc-fops.c:865:_gf_server_log_setxattr_failure] 0-testvol-server: 210941: SETXATTR /abc9998 (6e9812a2-711e-4cb5-b452-1f11a2d08e77) ==> security.selinux
[2016-11-29 07:37:43.046610] I [MSGID: 115060] [server-rpc-fops.c:865:_gf_server_log_setxattr_failure] 0-testvol-server: 210941: SETXATTR /abc9998 (6e9812a2-711e-4cb5-b452-1f11a2d08e77) ==> trusted.glusterfs.dht.linkto

I could not reproduce the issue so far. Even Karan said, it is intermittent.

Will update after more debugging.

Note: Rebalance sets uid and gid to root for all rebalance ops.

Comment 8 Susant Kumar Palai 2016-12-01 11:29:07 UTC
Added logs to capture the uid, gid at the time of setxattr. Have requested Karan to reproduce the issue. Will keep updating.


Comment 9 Susant Kumar Palai 2016-12-01 11:34:55 UTC
setting needinfo on Karan for the above info.

Note You need to log in before you can comment on or make changes to this bug.