Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: After add-brick, rebalance on volume failed on two storage nodes. Version-Release number of selected component (if applicable): [root@darkknight glusterfs]# rpm -qa | grep glusterfs glusterfs-3.7.0-3.el6rhs.x86_64 glusterfs-server-3.7.0-3.el6rhs.x86_64 glusterfs-api-3.7.0-3.el6rhs.x86_64 glusterfs-geo-replication-3.7.0-3.el6rhs.x86_64 glusterfs-libs-3.7.0-3.el6rhs.x86_64 glusterfs-fuse-3.7.0-3.el6rhs.x86_64 glusterfs-cli-3.7.0-3.el6rhs.x86_64 glusterfs-client-xlators-3.7.0-3.el6rhs.x86_64 How reproducible: 1/1 Steps to Reproduce: 1. Create 4*2 Distribute replicate volume 2. Mount volume as fuse mount on client 3. Enable quota and set quota on root of the volume 4. Create files from mount point so that disk quota exceeds. 5. Add brick to the volume 6. Start rebalance Actual results: Rebalance failed on two storage node. rebalance log ================================= [2015-06-05 10:05:05.912895] I [MSGID: 109036] [dht-common.c:6690:dht_log_new_layout_for_dir_selfheal] 0-vol0-dht: Setting layout of /.trashcan/internal_op with [Subvol_name: vol0-replicate-0, Err: -1 , Start: 11566080 , Stop: 4282015823 , Hash: 1 ], [Subvol_name: vol0-replicate-1, Err: -1 , Start: 4282015824 , Stop: 4294967295 , Hash: 1 ], [Subvol_name: vol0-replicate-2, Err: -1 , Start: 0 , Stop: 11566079 , Hash: 1 ], [2015-06-05 10:05:05.926494] I [dht-rebalance.c:2112:gf_defrag_process_dir] 0-vol0-dht: migrate data called on /.trashcan/internal_op [2015-06-05 10:05:05.943272] I [dht-common.c:3539:dht_setxattr] 0-vol0-dht: fixing the layout of /.trashcan/internal_op [2015-06-05 10:05:05.943757] E [dht-rebalance.c:2368:gf_defrag_settle_hash] 0-vol0-dht: fix layout on /.trashcan/internal_op failed [2015-06-05 10:05:05.944014] E [MSGID: 109016] [dht-rebalance.c:2528:gf_defrag_fix_layout] 0-vol0-dht: Fix layout failed for /.trashcan [2015-06-05 10:05:05.944367] I [dht-rebalance.c:1758:gf_defrag_task] 0-DHT: Thread wokeup. defrag->current_thread_count: 11 [2015-06-05 10:05:05.944664] I [dht-rebalance.c:1758:gf_defrag_task] 0-DHT: Thread wokeup. defrag->current_thread_count: 12 [2015-06-05 10:05:05.944733] I [dht-rebalance.c:1758:gf_defrag_task] 0-DHT: Thread wokeup. defrag->current_thread_count: 13 [2015-06-05 10:05:05.944798] I [dht-rebalance.c:1758:gf_defrag_task] 0-DHT: Thread wokeup. defrag->current_thread_count: 14 [2015-06-05 10:05:05.944877] I [dht-rebalance.c:1758:gf_defrag_task] 0-DHT: Thread wokeup. defrag->current_thread_count: 15 [2015-06-05 10:05:05.944943] I [dht-rebalance.c:1758:gf_defrag_task] 0-DHT: Thread wokeup. defrag->current_thread_count: 16 [2015-06-05 10:05:05.944990] I [dht-rebalance.c:1758:gf_defrag_task] 0-DHT: Thread wokeup. defrag->current_thread_count: 17 [2015-06-05 10:05:05.945031] I [dht-rebalance.c:1758:gf_defrag_task] 0-DHT: Thread wokeup. defrag->current_thread_count: 18 [2015-06-05 10:05:05.945113] I [dht-rebalance.c:1758:gf_defrag_task] 0-DHT: Thread wokeup. defrag->current_thread_count: 19 [2015-06-05 10:05:05.945189] I [dht-rebalance.c:1758:gf_defrag_task] 0-DHT: Thread wokeup. defrag->current_thread_count: 20 [2015-06-05 10:05:05.945829] I [MSGID: 109028] [dht-rebalance.c:3012:gf_defrag_status_get] 0-vol0-dht: Rebalance is failed. Time taken is 0.00 secs [2015-06-05 10:05:05.945882] I [MSGID: 109028] [dht-rebalance.c:3016:gf_defrag_status_get] 0-vol0-dht: Files migrated: 0, size: 0, lookups: 0, failures: 3, skipped: 0 [2015-06-05 10:05:05.946238] W [glusterfsd.c:1219:cleanup_and_exit] (--> 0-: received signum (15), shutting down Expected results: Rebalance should not fail. Additional info: [root@darkknight glusterfs]# gluster v info vol0 Volume Name: vol0 Type: Distributed-Replicate Volume ID: 5f93429a-056a-42d4-9d52-199df0ff1187 Status: Started Number of Bricks: 3 x 2 = 6 Transport-type: tcp Bricks: Brick1: 10.70.33.214:/rhs/brick1/b1 Brick2: 10.70.33.219:/rhs/brick1/b2 Brick3: 10.70.33.225:/rhs/brick1/b3 Brick4: 10.70.44.13:/rhs/brick1/b4 Brick5: 10.70.33.214:/rhs/brick1/b5 Brick6: 10.70.33.219:/rhs/brick1/b6 Options Reconfigured: features.quota-deem-statfs: on features.scrub: Active features.bitrot: on features.barrier: disable features.inode-quota: on features.quota: on features.uss: enable performance.readdir-ahead: on auto-delete: enable snap-activate-on-create: enable ===================================================== [root@darkknight glusterfs]# gluster v rebalance vol0 status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 0 0Bytes 3 0 0 completed 1.00 10.70.33.219 0 0Bytes 0 3 0 failed 1.00 10.70.33.225 2 2.0GB 2 0 0 completed 132.00 10.70.44.13 0 0Bytes 0 3 0 failed 0.00 volume rebalance: vol0: success: =============================================================== [root@darkknight glusterfs]# gluster v status vol0 Status of volume: vol0 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.33.214:/rhs/brick1/b1 49152 0 Y 32213 Brick 10.70.33.219:/rhs/brick1/b2 49152 0 Y 22633 Brick 10.70.33.225:/rhs/brick1/b3 49152 0 Y 21394 Brick 10.70.44.13:/rhs/brick1/b4 49152 0 Y 2874 Brick 10.70.33.214:/rhs/brick1/b5 49388 0 Y 7817 Brick 10.70.33.219:/rhs/brick1/b6 49388 0 Y 10133 Snapshot Daemon on localhost 49153 0 Y 32290 NFS Server on localhost 2049 0 Y 7836 Self-heal Daemon on localhost N/A N/A Y 7844 Quota Daemon on localhost N/A N/A Y 7852 Bitrot Daemon on localhost N/A N/A Y 7860 Scrubber Daemon on localhost N/A N/A Y 7869 Snapshot Daemon on 10.70.44.13 49153 0 Y 19275 NFS Server on 10.70.44.13 2049 0 Y 10231 Self-heal Daemon on 10.70.44.13 N/A N/A Y 10239 Quota Daemon on 10.70.44.13 N/A N/A Y 10291 Bitrot Daemon on 10.70.44.13 N/A N/A Y 10299 Scrubber Daemon on 10.70.44.13 N/A N/A Y 10308 Snapshot Daemon on 10.70.33.225 49153 0 Y 21451 NFS Server on 10.70.33.225 2049 0 Y 11488 Self-heal Daemon on 10.70.33.225 N/A N/A Y 11496 Quota Daemon on 10.70.33.225 N/A N/A Y 11526 Bitrot Daemon on 10.70.33.225 N/A N/A Y 11534 Scrubber Daemon on 10.70.33.225 N/A N/A Y 11543 Snapshot Daemon on 10.70.33.219 49153 0 Y 6536 NFS Server on 10.70.33.219 2049 0 Y 10153 Self-heal Daemon on 10.70.33.219 N/A N/A Y 10161 Quota Daemon on 10.70.33.219 N/A N/A Y 10169 Bitrot Daemon on 10.70.33.219 N/A N/A Y 10177 Scrubber Daemon on 10.70.33.219 N/A N/A Y 10188 Task Status of Volume vol0 ------------------------------------------------------------------------------ Task : Rebalance ID : 1d900b21-59ff-4509-8795-25ed6e48ad7a Status : failed