Bug 1131418
| Summary: | remove-brick: logs display the error related to "Operation not permitted" | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Saurabh <saujain> |
| Component: | distribute | Assignee: | Susant Kumar Palai <spalai> |
| Status: | CLOSED ERRATA | QA Contact: | Triveni Rao <trao> |
| Severity: | medium | Docs Contact: | |
| Priority: | high | ||
| Version: | rhgs-3.0 | CC: | annair, bmohanra, mzywusko, nbalacha, nsathyan, sashinde, spalai, vagarwal |
| Target Milestone: | --- | ||
| Target Release: | RHGS 3.1.0 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | glusterfs-3.7.1-1 | Doc Type: | Bug Fix |
| Doc Text: |
Previously, when the gf_defrag_handle_hardlink function was executed, the setxattr was performed on the internal afr keys too. This lead to afr aborting the operation with the following error, which resulted in hard link migration failures:
"operation not supported".
With this fix, setxattr is performed only on the required keys.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-07-29 04:35:15 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1202842 | ||
Upstream patch: http://review.gluster.org/#/c/7943/ Downstream patch: https://code.engineering.redhat.com/gerrit/#/c/26486/ [root@casino-vm1 ~]# gluster v info kit
Volume Name: kit
Type: Distributed-Replicate
Volume ID: ae805fc4-45c2-4d80-94e8-ce50336bc3c4
Status: Started
Number of Bricks: 5 x 2 = 10
Transport-type: tcp
Bricks:
Brick1: 10.70.35.57:/rhs/brick1/s0
Brick2: 10.70.35.136:/rhs/brick1/s0
Brick3: 10.70.35.57:/rhs/brick2/s0
Brick4: 10.70.35.136:/rhs/brick2/s0
Brick5: 10.70.35.57:/rhs/brick4/s0
Brick6: 10.70.35.136:/rhs/brick4/s0
Brick7: 10.70.35.57:/rhs/brick3/s0
Brick8: 10.70.35.136:/rhs/brick3/s0
Brick9: 10.70.35.57:/rhs/brick3/s1
Brick10: 10.70.35.136:/rhs/brick3/s1
Options Reconfigured:
performance.readdir-ahead: on
[root@casino-vm1 ~]#
[root@casino-vm1 ~]# gluster v remove-brick kit 10.70.35.57:/rhs/brick3/s1 10.70.35.136:/rhs/brick3/s1 start
volume remove-brick start: success
ID: 11cfca87-0737-42fe-88ab-ec698b77914d
[root@casino-vm1 ~]#
[root@casino-vm1 ~]# gluster v remove-brick kit 10.70.35.57:/rhs/brick3/s1 10.70.35.136:/rhs/brick3/s1 status
Node Rebalanced-files size scanned failures skipped status run time in s
ecs
--------- ----------- ----------- ----------- ----------- ----------- ------------ -----------
---
localhost 121 0Bytes 987 0 0 in progress 6
.00
10.70.35.136 0 0Bytes 0 0 0 in progress 6.00
[root@casino-vm1 ~]# gluster v remove-brick kit 10.70.35.57:/rhs/brick3/s1 10.70.35.136:/rhs/brick3/s1 status
Node Rebalanced-files size scanned failures skipped status run time in secs
--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------
localhost 800 0Bytes 2492 0 0 in progress 26.00
10.70.35.136 0 0Bytes 0 0 0 completed 8.00
[root@casino-vm1 ~]#
[root@casino-vm1 ~]# gluster v remove-brick kit 10.70.35.57:/rhs/brick3/s1 10.70.35.136:/rhs/brick3/s1 status
Node Rebalanced-files size scanned failures skipped status run time in secs
--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------
localhost 14528 0Bytes 31863 0 0 in progress 425.00
10.70.35.136 0 0Bytes 0 0 0 completed 8.00
[root@casino-vm1 ~]# gluster v remove-brick kit 10.70.35.57:/rhs/brick3/s1 10.70.35.136:/rhs/brick3/s1 status
Node Rebalanced-files size scanned failures skipped status run time in secs
--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------
localhost 16596 0Bytes 35305 0 0 completed 479.00
10.70.35.136 0 0Bytes 0 0 0 completed 8.00
[root@casino-vm1 ~]#
[root@casino-vm2 ~]# less /var/log/glusterfs/kit-rebalance.log | grep "\sE\s"
[root@casino-vm2 ~]# less /var/log/glusterfs/kit-rebalance.log | grep Operation
[root@casino-vm2 ~]#
[root@casino-vm2 ~]# less /var/log/glusterfs/kit-rebalance.log | grep permitted
[root@casino-vm2 ~]#
this bug is verified and found no issues.
Updated doc text. Can you please review the same and sign off. Doc looks good. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-1495.html |
Description of problem: Started remove brick for a volume of distributed repilcated volume. I was already having some data inside the volume. [root@nfs2 ~]# gluster volume info dist-rep1 Volume Name: dist-rep1 Type: Distributed-Replicate Volume ID: dab9f592-39b4-428d-bc6d-01a0b7185743 Status: Started Snap Volume: no Number of Bricks: 8 x 2 = 16 Transport-type: tcp Bricks: Brick1: 10.70.37.62:/bricks/d1r11 Brick2: 10.70.37.215:/bricks/d1r21 Brick3: 10.70.37.44:/bricks/d2r11 Brick4: 10.70.37.201:/bricks/dr2r21 Brick5: 10.70.37.62:/bricks/d3r11 Brick6: 10.70.37.215:/bricks/d3r21 Brick7: 10.70.37.44:/bricks/d4r11 Brick8: 10.70.37.201:/bricks/dr4r21 Brick9: 10.70.37.62:/bricks/d5r11 Brick10: 10.70.37.215:/bricks/d5r21 Brick11: 10.70.37.44:/bricks/d6r11 Brick12: 10.70.37.201:/bricks/dr6r21 Brick13: 10.70.37.62:/bricks/d1r12-add-n Brick14: 10.70.37.215:/bricks/d1r22-add-n Brick15: 10.70.37.62:/bricks/d1r12-add-n1 Brick16: 10.70.37.215:/bricks/d1r22-add-n1 Options Reconfigured: features.quota-deem-statfs: on features.quota: on diagnostics.client-log-level: INFO nfs.log-level: info cluster.self-heal-daemon: on cluster.data-self-heal: off cluster.entry-self-heal: off cluster.metadata-self-heal: off server.root-squash: off nfs.disable: off nfs.volume-access: read-write nfs.rpc-auth-reject: 10.70.35.33 nfs.rpc-auth-allow: rhsauto034.lab.eng.blr.redhat.com,rhsauto002.lab.eng.blr.redhat.com nfs.addr-namelookup: on nfs.export-dirs: on nfs.export-volumes: enable nfs.nlm: on nfs.acl: on performance.readdir-ahead: on nfs.read-size: 1048576 nfs.write-size: 1048576 nfs.mount-udp: off snap-max-hard-limit: 256 snap-max-soft-limit: 90 auto-delete: disable Version-Release number of selected component (if applicable): glusterfs-3.6.0.27-1.el6rhs.x86_64 How reproducible: Steps to Reproduce: 1. start remove brick, command used is this, gluster volume remove-brick dist-rep1 10.70.37.62:/bricks/d1r12-add-n 10.70.37.215:/bricks/d1r22-add-n start 2. take remove-brick status, Actual results: result of step2, [root@nfs1 ~]# gluster volume remove-brick dist-rep1 10.70.37.62:/bricks/d1r12-add-n 10.70.37.215:/bricks/d1r22-add-n status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 9527 11.3MB 29868 2 0 in progress 7729.00 10.70.37.215 0 0Bytes 59968 0 0 in progress 7729.00 logs from rebalance.log [root@nfs1 ~]# less /var/log/glusterfs/dist-rep1-rebalance.log | grep "\sE\s" [2014-08-19 06:28:57.837338] E [afr-inode-write.c:1526:afr_setxattr] 0-dist-rep1-replicate-4: attempt to set internal xattr: trusted.afr.*: Operation not permitted [2014-08-19 06:28:57.837370] E [MSGID: 109023] [dht-rebalance.c:216:gf_defrag_handle_hardlink] 0-dist-rep1-dht: Migrate file failed :Linkto setxattr failed dist-rep1-replicate-4 -> hardlink (Operation not permitted) [2014-08-19 06:28:57.892556] E [afr-inode-write.c:1526:afr_setxattr] 0-dist-rep1-replicate-4: attempt to set internal xattr: trusted.afr.*: Operation not permitted [2014-08-19 06:28:57.892571] E [MSGID: 109023] [dht-rebalance.c:216:gf_defrag_handle_hardlink] 0-dist-rep1-dht: Migrate file failed :Linkto setxattr failed dist-rep1-replicate-4 -> file (Operation not permitted) [2014-08-19 06:28:57.892854] E [MSGID: 109023] [dht-rebalance.c:1546:gf_defrag_migrate_data] 0-dist-rep1-dht: migrate-data failed for /run8681/coverage/dir/file Expected results: No, "operation not permitted" issues expected Additional info: