Description of problem: Started remove brick for a volume of distributed repilcated volume. I was already having some data inside the volume. [root@nfs2 ~]# gluster volume info dist-rep1 Volume Name: dist-rep1 Type: Distributed-Replicate Volume ID: dab9f592-39b4-428d-bc6d-01a0b7185743 Status: Started Snap Volume: no Number of Bricks: 8 x 2 = 16 Transport-type: tcp Bricks: Brick1: 10.70.37.62:/bricks/d1r11 Brick2: 10.70.37.215:/bricks/d1r21 Brick3: 10.70.37.44:/bricks/d2r11 Brick4: 10.70.37.201:/bricks/dr2r21 Brick5: 10.70.37.62:/bricks/d3r11 Brick6: 10.70.37.215:/bricks/d3r21 Brick7: 10.70.37.44:/bricks/d4r11 Brick8: 10.70.37.201:/bricks/dr4r21 Brick9: 10.70.37.62:/bricks/d5r11 Brick10: 10.70.37.215:/bricks/d5r21 Brick11: 10.70.37.44:/bricks/d6r11 Brick12: 10.70.37.201:/bricks/dr6r21 Brick13: 10.70.37.62:/bricks/d1r12-add-n Brick14: 10.70.37.215:/bricks/d1r22-add-n Brick15: 10.70.37.62:/bricks/d1r12-add-n1 Brick16: 10.70.37.215:/bricks/d1r22-add-n1 Options Reconfigured: features.quota-deem-statfs: on features.quota: on diagnostics.client-log-level: INFO nfs.log-level: info cluster.self-heal-daemon: on cluster.data-self-heal: off cluster.entry-self-heal: off cluster.metadata-self-heal: off server.root-squash: off nfs.disable: off nfs.volume-access: read-write nfs.rpc-auth-reject: 10.70.35.33 nfs.rpc-auth-allow: rhsauto034.lab.eng.blr.redhat.com,rhsauto002.lab.eng.blr.redhat.com nfs.addr-namelookup: on nfs.export-dirs: on nfs.export-volumes: enable nfs.nlm: on nfs.acl: on performance.readdir-ahead: on nfs.read-size: 1048576 nfs.write-size: 1048576 nfs.mount-udp: off snap-max-hard-limit: 256 snap-max-soft-limit: 90 auto-delete: disable Version-Release number of selected component (if applicable): glusterfs-3.6.0.27-1.el6rhs.x86_64 How reproducible: Steps to Reproduce: 1. start remove brick, command used is this, gluster volume remove-brick dist-rep1 10.70.37.62:/bricks/d1r12-add-n 10.70.37.215:/bricks/d1r22-add-n start 2. take remove-brick status, Actual results: result of step2, [root@nfs1 ~]# gluster volume remove-brick dist-rep1 10.70.37.62:/bricks/d1r12-add-n 10.70.37.215:/bricks/d1r22-add-n status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 9527 11.3MB 29868 2 0 in progress 7729.00 10.70.37.215 0 0Bytes 59968 0 0 in progress 7729.00 logs from rebalance.log [root@nfs1 ~]# less /var/log/glusterfs/dist-rep1-rebalance.log | grep "\sE\s" [2014-08-19 06:28:57.837338] E [afr-inode-write.c:1526:afr_setxattr] 0-dist-rep1-replicate-4: attempt to set internal xattr: trusted.afr.*: Operation not permitted [2014-08-19 06:28:57.837370] E [MSGID: 109023] [dht-rebalance.c:216:gf_defrag_handle_hardlink] 0-dist-rep1-dht: Migrate file failed :Linkto setxattr failed dist-rep1-replicate-4 -> hardlink (Operation not permitted) [2014-08-19 06:28:57.892556] E [afr-inode-write.c:1526:afr_setxattr] 0-dist-rep1-replicate-4: attempt to set internal xattr: trusted.afr.*: Operation not permitted [2014-08-19 06:28:57.892571] E [MSGID: 109023] [dht-rebalance.c:216:gf_defrag_handle_hardlink] 0-dist-rep1-dht: Migrate file failed :Linkto setxattr failed dist-rep1-replicate-4 -> file (Operation not permitted) [2014-08-19 06:28:57.892854] E [MSGID: 109023] [dht-rebalance.c:1546:gf_defrag_migrate_data] 0-dist-rep1-dht: migrate-data failed for /run8681/coverage/dir/file Expected results: No, "operation not permitted" issues expected Additional info:
Upstream patch: http://review.gluster.org/#/c/7943/ Downstream patch: https://code.engineering.redhat.com/gerrit/#/c/26486/
[root@casino-vm1 ~]# gluster v info kit Volume Name: kit Type: Distributed-Replicate Volume ID: ae805fc4-45c2-4d80-94e8-ce50336bc3c4 Status: Started Number of Bricks: 5 x 2 = 10 Transport-type: tcp Bricks: Brick1: 10.70.35.57:/rhs/brick1/s0 Brick2: 10.70.35.136:/rhs/brick1/s0 Brick3: 10.70.35.57:/rhs/brick2/s0 Brick4: 10.70.35.136:/rhs/brick2/s0 Brick5: 10.70.35.57:/rhs/brick4/s0 Brick6: 10.70.35.136:/rhs/brick4/s0 Brick7: 10.70.35.57:/rhs/brick3/s0 Brick8: 10.70.35.136:/rhs/brick3/s0 Brick9: 10.70.35.57:/rhs/brick3/s1 Brick10: 10.70.35.136:/rhs/brick3/s1 Options Reconfigured: performance.readdir-ahead: on [root@casino-vm1 ~]# [root@casino-vm1 ~]# gluster v remove-brick kit 10.70.35.57:/rhs/brick3/s1 10.70.35.136:/rhs/brick3/s1 start volume remove-brick start: success ID: 11cfca87-0737-42fe-88ab-ec698b77914d [root@casino-vm1 ~]# [root@casino-vm1 ~]# gluster v remove-brick kit 10.70.35.57:/rhs/brick3/s1 10.70.35.136:/rhs/brick3/s1 status Node Rebalanced-files size scanned failures skipped status run time in s ecs --------- ----------- ----------- ----------- ----------- ----------- ------------ ----------- --- localhost 121 0Bytes 987 0 0 in progress 6 .00 10.70.35.136 0 0Bytes 0 0 0 in progress 6.00 [root@casino-vm1 ~]# gluster v remove-brick kit 10.70.35.57:/rhs/brick3/s1 10.70.35.136:/rhs/brick3/s1 status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 800 0Bytes 2492 0 0 in progress 26.00 10.70.35.136 0 0Bytes 0 0 0 completed 8.00 [root@casino-vm1 ~]# [root@casino-vm1 ~]# gluster v remove-brick kit 10.70.35.57:/rhs/brick3/s1 10.70.35.136:/rhs/brick3/s1 status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 14528 0Bytes 31863 0 0 in progress 425.00 10.70.35.136 0 0Bytes 0 0 0 completed 8.00 [root@casino-vm1 ~]# gluster v remove-brick kit 10.70.35.57:/rhs/brick3/s1 10.70.35.136:/rhs/brick3/s1 status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 16596 0Bytes 35305 0 0 completed 479.00 10.70.35.136 0 0Bytes 0 0 0 completed 8.00 [root@casino-vm1 ~]# [root@casino-vm2 ~]# less /var/log/glusterfs/kit-rebalance.log | grep "\sE\s" [root@casino-vm2 ~]# less /var/log/glusterfs/kit-rebalance.log | grep Operation [root@casino-vm2 ~]# [root@casino-vm2 ~]# less /var/log/glusterfs/kit-rebalance.log | grep permitted [root@casino-vm2 ~]# this bug is verified and found no issues.
Updated doc text. Can you please review the same and sign off.
Doc looks good.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-1495.html