| Summary: | rebalace fails | ||
|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Saurabh <saurabh> |
| Component: | core | Assignee: | Amar Tumballi <amarts> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | mainline | CC: | amarts, gluster-bugs, vraman |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | Type: | --- | |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
|
Description
Saurabh
2011-07-22 06:20:40 UTC
the git repo is synced to this id,
commit 37b4b7ccad5a479ed2ce96598e902a12fd457bd5
Author: Raghavendra Bhat <raghavendrabhat>
Date: Wed Jul 20 12:21:12 2011 +0530
debug/io-stats: allow log-level to be set to INFO
Change-Id: Ieba6591b0641dcb7dad724c1d8199dc5e91b6bfd
Hi Saurabh, Can you check if rebalance works fine now? because after http://review.gluster.com/90 most of the things should now work fine. Regards, Amar Yeah Amar, the issue is still happening, latest sync points to this change number, Change-Id: I85cf4afc1f534f5f51018449d5d84baef18fce23 rebalance fails after quota disable and adding new bricks, [root@Centos1 ~]# /opt/glusterfs/3.3.0//sbin/gluster volume quota drep disable Disabling quota will delete all the quota configuration. Do you want to continue? (y/n) y Disabling quota has been successful [root@Centos1 ~]# /opt/glusterfs/3.3.0//sbin/gluster volume add-brick drep 10.1.12.134:/mnt/d-d3 10.1.12.135:/mnt/r-r3 Add Brick successful [root@Centos1 ~]# /opt/glusterfs/3.3.0//sbin/gluster volume rebalance drep start starting rebalance on volume drep has been successful [root@Centos1 ~]# /opt/glusterfs/3.3.0//sbin/gluster volume rebalance drep status rebalance failed [2011-08-19 08:42:27.649555] I [dht-rebalance.c:317:dht_migrate_file] 0-drep-dht: /d0/f.26: attempting to move from drep-replicate-0 to d rep-replicate-4 [2011-08-19 08:42:27.650134] W [dict.c:356:dict_del] (-->/opt/glusterfs/3.3.0//lib/glusterfs/3.3.0qa3/xlator/cluster/distribute.so(dht_mi grate_file+0x54e) [0x2aaaada2e7be] (-->/opt/glusterfs/3.3.0//lib/libglusterfs.so.0(syncop_lookup+0x148) [0x2b540f63f508] (-->/opt/gluster fs/3.3.0//lib/glusterfs/3.3.0qa3/xlator/cluster/replicate.so(afr_lookup+0x403) [0x2aaaad8105f3]))) 0-dict: !this || key=gfid-req [2011-08-19 08:42:27.652039] W [dht-rebalance.c:244:__dht_check_free_space] 0-drep-dht: data movement attempted from node (drep-replicate -0) with higher disk space to a node (drep-replicate-4) with lesser disk space (/d0/f.26) [2011-08-19 08:42:27.652098] C [fuse-bridge.c:876:fuse_setxattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage [2011-08-19 08:42:27.652233] W [fuse-bridge.c:847:fuse_err_cbk] 0-glusterfs-fuse: 142: SETXATTR() /d0/f.26 => -1 (Operation not supported) pending frames: frame : type(1) op(SETXATTR) frame : type(1) op(SETXATTR) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2011-08-19 08:42:27 configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 [2011-08-19 08:42:27.649555] I [dht-rebalance.c:317:dht_migrate_file] 0-drep-dht: /d0/f.26: attempting to move from drep-replicate-0 to d rep-replicate-4 [2011-08-19 08:42:27.650134] W [dict.c:356:dict_del] (-->/opt/glusterfs/3.3.0//lib/glusterfs/3.3.0qa3/xlator/cluster/distribute.so(dht_mi grate_file+0x54e) [0x2aaaada2e7be] (-->/opt/glusterfs/3.3.0//lib/libglusterfs.so.0(syncop_lookup+0x148) [0x2b540f63f508] (-->/opt/gluster fs/3.3.0//lib/glusterfs/3.3.0qa3/xlator/cluster/replicate.so(afr_lookup+0x403) [0x2aaaad8105f3]))) 0-dict: !this || key=gfid-req [2011-08-19 08:42:27.652039] W [dht-rebalance.c:244:__dht_check_free_space] 0-drep-dht: data movement attempted from node (drep-replicate -0) with higher disk space to a node (drep-replicate-4) with lesser disk space (/d0/f.26) [2011-08-19 08:42:27.652098] C [fuse-bridge.c:876:fuse_setxattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage [2011-08-19 08:42:27.652233] W [fuse-bridge.c:847:fuse_err_cbk] 0-glusterfs-fuse: 142: SETXATTR() /d0/f.26 => -1 (Operation not supported) pending frames: frame : type(1) op(SETXATTR) frame : type(1) op(SETXATTR) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2011-08-19 08:42:27 configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.3.0qa3 /lib64/libc.so.6[0x3470c302d0] /opt/glusterfs/3.3.0//lib/libglusterfs.so.0(synctask_wrap+0x4)[0x2b540f63c834] /lib64/libc.so.6[0x3470c419c0] --------- : : I tried to reproduce this in my setup.. i get the same log [2011-08-19 22:41:41.595461] W [dict.c:356:dict_del] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(+0xbab2) [0x7f06d70abab2] (-->/usr/local/lib/libglusterfs.so.0(syncop_lookup+0x350) [0x7f06dce6fbc9] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/replicate.so(afr_lookup+0x524) [0x7f06d73444a5]))) 0-dict: !this || key=gfid-req [2011-08-19 22:41:41.596551] W [dht-rebalance.c:244:__dht_check_free_space] 0-test-dht: data movement attempted from node (test-replicate-2) with higher disk space to a node (test-replicate-1) with lesser disk space (/testdir/16) [2011-08-19 22:41:41.596588] W [fuse-bridge.c:847:fuse_err_cbk] 0-glusterfs-fuse: 135: SETXATTR() /testdir/16 => -1 (Operation not supported) but, the crash doesn't happen. Saurabh, can you share details of the machine login on which the rebalance failed? Want to have a look at the core. With bug 765368 getting fixed this should be fixed. Please reopen if seen again. |