Bug 1399504
| Summary: | Permission denied errors on server during rebalance | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Karan Sandha <ksandha> | ||||
| Component: | distribute | Assignee: | Susant Kumar Palai <spalai> | ||||
| Status: | CLOSED WORKSFORME | QA Contact: | Prasad Desala <tdesala> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | rhgs-3.2 | CC: | amukherj, ksandha, nbalacha, rhinduja, rhs-bugs, storage-qa-internal | ||||
| Target Milestone: | --- | ||||||
| Target Release: | RHGS 3.2.0 | ||||||
| Hardware: | All | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2017-01-30 10:20:37 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Nithya, Yes everything was being run as root user. This test didn't involve non-root user. Thanks & regards Karan Update: From brick logs I could see setxattr for key: security.selinux* Here is log for few setxattr failures: [2016-11-29 07:37:43.046552] E [MSGID: 113001] [posix-helpers.c:1160:posix_handle_pair] 0-testvol-posix: /bricks/brick0/testvol_brick0/.glusterfs/6e/98/6e9812a2-711e-4cb5-b452-1f11a2d08e77: key:security.selinuxflags: 0 length:35 [Permission denied] [2016-11-29 07:37:43.046593] I [MSGID: 115060] [server-rpc-fops.c:865:_gf_server_log_setxattr_failure] 0-testvol-server: 210941: SETXATTR /abc9998 (6e9812a2-711e-4cb5-b452-1f11a2d08e77) ==> security.selinux [2016-11-29 07:37:43.046610] I [MSGID: 115060] [server-rpc-fops.c:865:_gf_server_log_setxattr_failure] 0-testvol-server: 210941: SETXATTR /abc9998 (6e9812a2-711e-4cb5-b452-1f11a2d08e77) ==> trusted.glusterfs.dht.linkto I could not reproduce the issue so far. Even Karan said, it is intermittent. Will update after more debugging. Note: Rebalance sets uid and gid to root for all rebalance ops. Added logs to capture the uid, gid at the time of setxattr. Have requested Karan to reproduce the issue. Will keep updating. Susant- setting needinfo on Karan for the above info. |
Created attachment 1225712 [details] Permission Denied Description of problem: Getting permission denied on migration of data warnigns on the server after rebalance . Version-Release number of selected component (if applicable): [root@dhcp47-116 testmount]# rpm -qa | grep gluster glusterfs-client-xlators-3.8.4-5.el7.x86_64 glusterfs-3.8.4-5.el7.x86_64 glusterfs-fuse-3.8.4-5.el7.x86_64 glusterfs-libs-3.8.4-5.el7.x86_64 [root@dhcp47-116 testmount]# How reproducible: 2/3 Logs are placed at rhsqe-repo.lab.eng.blr.redhat.com:/var/www/html/sosreports/<bug> Steps to Reproduce: 1. Create a 1x(2+1) arbiter volume and mount it to a client 2. Create 1000 directories and 1000 files using mkdir dir{1..1000} and touch abc{1..1000} 3. Now replace a brick from server1 with a new brick on the same server. 4. let the heal be completed and now add 3 bricks to form a new subvolume in the same volume. 5. After the adding of bricks is successful, now trigger rebalance ; gluster volume rebalance testvol start Actual results: W [MSGID: 114031] [client-rpc-fops.c:1893:client3_3_fsetxattr_cbk] 0-testvol-client-3: remote operation failed [Permission denied] [2016-11-29 06:35:01.819231] W [MSGID: 114031] [client-rpc-fops.c:1893:client3_3_fsetxattr_cbk] 0-testvol-client-5: remote operation failed [Permission denied] [2016-11-29 06:35:01.819293] W [MSGID: 114031] [client-rpc-fops.c:1893:client3_3_fsetxattr_cbk] 0-testvol-client-4: remote operation failed [Permission denied] [2016-11-29 06:35:01.820177] W [MSGID: 109023] [dht-rebalance.c:693:__dht_rebalance_create_dst_file] 0-testvol-dht: /abc1664: failed to set xattr on testvol-replicate-1 (Permission denied) Expected results: There should be no errors/warnings during rebalance. Additional info: tested on RHEL 6.8