Bug 1224144

Summary: rebalance failed after attaching the tier to the volume.
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Triveni Rao <trao>
Component: tierAssignee: Bug Updates Notification Mailing List <rhs-bugs>
Status: CLOSED WORKSFORME QA Contact: Nag Pavan Chilakam <nchilaka>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: rhgs-3.1CC: bugs, ndevos, rhs-bugs, rkavunga, sashinde, storage-qa-internal, vagarwal
Target Milestone: ---Keywords: TestBlocker
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: TIERING
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1221534 Environment:
Last Closed: 2015-10-30 17:37:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1221534, 1222092    
Bug Blocks: 1221957, 1229269, 1273726, 1274411    

Description Triveni Rao 2015-05-22 09:11:45 UTC 
+++ This bug was initially created as a clone of Bug #1221534 +++

Description of problem:

After attaching the tier to the volume rebalance status shows failed and rebalance doesnot happen between cold and hot tier.


Version-Release number of selected component (if applicable):
root@rhsqa14-vm3 ~]# glusterfs --version
glusterfs 3.7.0beta2 built on May 11 2015 01:27:45
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.
You have new mail in /var/spool/mail/root
[root@rhsqa14-vm3 ~]# 
[root@rhsqa14-vm3 ~]# 
[root@rhsqa14-vm3 ~]# rpm -qa | grep gluster
glusterfs-libs-3.7.0beta2-0.0.el6.x86_64
glusterfs-fuse-3.7.0beta2-0.0.el6.x86_64
glusterfs-rdma-3.7.0beta2-0.0.el6.x86_64
glusterfs-3.7.0beta2-0.0.el6.x86_64
glusterfs-api-3.7.0beta2-0.0.el6.x86_64
glusterfs-cli-3.7.0beta2-0.0.el6.x86_64
glusterfs-geo-replication-3.7.0beta2-0.0.el6.x86_64
glusterfs-extra-xlators-3.7.0beta2-0.0.el6.x86_64
glusterfs-client-xlators-3.7.0beta2-0.0.el6.x86_64
glusterfs-server-3.7.0beta2-0.0.el6.x86_64
[root@rhsqa14-vm3 ~]#


How reproducible:
easily

Steps to Reproduce:
1.create dist-rep volume.
2.fuse mount it and add some dirs.
3. attach tier to the volume.
4. rebalance status shows failed.
5. on mount point dirs are missing.


Actual results:

rebalance fails.

Additional info:

[root@rhsqa14-vm1 linux-4.0]# gluster  v info test

Volume Name: test
Type: Distribute
Volume ID: d5b37bbc-af62-48cd-8942-43b5a395da65
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 10.70.46.233:/rhs/brick4/t1
Brick2: 10.70.46.236:/rhs/brick4/t1
Options Reconfigured:
performance.readdir-ahead: on 
[root@rhsqa14-vm1 linux-4.0]# 
[root@rhsqa14-vm1 linux-4.0]# gluster v attach-tier test replica 2 10.70.46.233:/rhs/brick3/t2 10.70.46.236:/rhs/brick3/t2 10.70.46.233:/rhs
/brick5/t2 10.70.46.236:/rhs/brick5/t2 force
Attach tier is recommended only for testing purposes in this release. Do you want to continue? (y/n) y
volume attach-tier: success   
volume rebalance: test: success: Rebalance on test has been started successfully. Use rebalance status command to check status of the rebala
nce process.
ID: a89419aa-c45b-42ef-8998-1a23cde16efe

[root@rhsqa14-vm1 linux-4.0]# 

[root@rhsqa14-vm1 linux-4.0]# gluster v rebalance test status                                                                               
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run
 time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     -
-------------
                               localhost                0        0Bytes             0             0             0               failed               0.00
                            10.70.46.236                0        0Bytes             0             0             0               failed               0.00
volume rebalance: test: success:
[root@rhsqa14-vm1 linux-4.0]# 
[root@rhsqa14-vm1 linux-4.0]# gluster v info test

Volume Name: test
Type: Tier
Volume ID: d5b37bbc-af62-48cd-8942-43b5a395da65
Status: Started
Number of Bricks: 6
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4   
Brick1: 10.70.46.236:/rhs/brick5/t2
Brick2: 10.70.46.233:/rhs/brick5/t2
Brick3: 10.70.46.236:/rhs/brick3/t2
Brick4: 10.70.46.233:/rhs/brick3/t2
Cold Bricks:
Cold Tier Type : Distribute   
Number of Bricks: 2
Brick5: 10.70.46.233:/rhs/brick4/t1
Brick6: 10.70.46.236:/rhs/brick4/t1
Options Reconfigured:
performance.readdir-ahead: on 


Before attaching the tier: on mount poit
[root@rhsqa14-vm5 disk1]# ls -la
total 4
drwxr-xr-x.  5 root root  133 May 14  2015 .
dr-xr-xr-x. 30 root root 4096 May 14 05:47 ..
-rw-r--r--.  1 root root    0 May 14 05:47 t1
-rw-r--r--.  1 root root    0 May 14 05:47 t2
-rw-r--r--.  1 root root    0 May 14 05:47 t4
drwxr-xr-x.  3 root root   48 May 14  2015 .trashcan
drwxr-xr-x.  2 root root   12 May 14  2015 triveni
[root@rhsqa14-vm5 disk1]#


After attaching the tier on mount point

[root@rhsqa14-vm5 disk1]# ls -la
total 4
drwxr-xr-x.  4 root root  211 May 14  2015 .
dr-xr-xr-x. 30 root root 4096 May 14 05:47 ..
-rw-r--r--.  1 root root    0 May 14 05:47 t1
-rw-r--r--.  1 root root    0 May 14 05:47 t2
-rw-r--r--.  1 root root    0 May 14 05:47 t4
drwxr-xr-x.  3 root root   96 May 14  2015 .trashcan
[root@rhsqa14-vm5 disk1]# 


rebalance Log messages:

[2015-05-14 09:52:37.116716] I [graph.c:269:gf_add_cmdline_options] 0-test-hot-replicate-1: adding option 'data-self-heal' for volume 'test-hot-replicate-1' with value 'off'
[2015-05-14 09:52:37.116736] I [graph.c:269:gf_add_cmdline_options] 0-test-hot-replicate-0: adding option 'readdir-failover' for volume 'test-hot-replicate-0' with value 'off'
[2015-05-14 09:52:37.116747] I [graph.c:269:gf_add_cmdline_options] 0-test-hot-replicate-0: adding option 'entry-self-heal' for volume 'test-hot-replicate-0' with value 'off'
[2015-05-14 09:52:37.116757] I [graph.c:269:gf_add_cmdline_options] 0-test-hot-replicate-0: adding option 'metadata-self-heal' for volume 'test-hot-replicate-0' with value 'off'
[2015-05-14 09:52:37.116767] I [graph.c:269:gf_add_cmdline_options] 0-test-hot-replicate-0: adding option 'data-self-heal' for volume 'test-hot-replicate-0' with value 'off'
[2015-05-14 09:52:37.116778] I [graph.c:269:gf_add_cmdline_options] 0-test-cold-dht: adding option 'commit-hash' for volume 'test-cold-dht' with value '2862842620'
[2015-05-14 09:52:37.116788] I [graph.c:269:gf_add_cmdline_options] 0-test-cold-dht: adding option 'node-uuid' for volume 'test-cold-dht' with value '87acbf29-e821-48bf-9aa8-bbda9321e609'
[2015-05-14 09:52:37.116798] I [graph.c:269:gf_add_cmdline_options] 0-test-cold-dht: adding option 'rebalance-cmd' for volume 'test-cold-dht' with value '6'
[2015-05-14 09:52:37.116809] I [graph.c:269:gf_add_cmdline_options] 0-test-cold-dht: adding option 'readdir-optimize' for volume 'test-cold-dht' with value 'on'
[2015-05-14 09:52:37.116820] I [graph.c:269:gf_add_cmdline_options] 0-test-cold-dht: adding option 'assert-no-child-down' for volume 'test-cold-dht' with value 'yes'
[2015-05-14 09:52:37.116835] I [graph.c:269:gf_add_cmdline_options] 0-test-cold-dht: adding option 'lookup-unhashed' for volume 'test-cold-dht' with value 'yes'
[2015-05-14 09:52:37.116846] I [graph.c:269:gf_add_cmdline_options] 0-test-cold-dht: adding option 'use-readdirp' for volume 'test-cold-dht' with value 'yes'
[2015-05-14 09:52:37.118410] I [dht-shared.c:598:dht_init] 0-tier-dht: dht_init using commit hash 2862842620
[2015-05-14 09:52:37.120852] E [MSGID: 109037] [tier.c:1007:tier_load_externals] 0-tier-dht: Error loading libgfdb.so /usr/lib64/libgfdb.so: cannot open shared object file: No such file or directory

[2015-05-14 09:52:37.120892] E [MSGID: 109037] [tier.c:1070:tier_init] 0-tier-dht: Could not load externals. Aborting
[2015-05-14 09:52:37.120903] E [xlator.c:426:xlator_init] 0-tier-dht: Initialization of volume 'tier-dht' failed, review your volfile again
[2015-05-14 09:52:37.120913] E [graph.c:322:glusterfs_graph_init] 0-tier-dht: initializing translator failed
[2015-05-14 09:52:37.120921] E [graph.c:661:glusterfs_graph_activate] 0-graph: init failed
[2015-05-14 09:52:37.124323] W [glusterfsd.c:1219:cleanup_and_exit] (--> 0-: received signum (0), shutting down
~
~

--- Additional comment from Triveni Rao on 2015-05-15 05:37:19 EDT ---



This was reproducible on new downstream build also:

[root@rhsqa14-vm1 ~]# rpm -qa | grep gluster
glusterfs-3.7.0-2.el6rhs.x86_64
glusterfs-cli-3.7.0-2.el6rhs.x86_64
glusterfs-libs-3.7.0-2.el6rhs.x86_64
glusterfs-client-xlators-3.7.0-2.el6rhs.x86_64
glusterfs-api-3.7.0-2.el6rhs.x86_64
glusterfs-server-3.7.0-2.el6rhs.x86_64
glusterfs-fuse-3.7.0-2.el6rhs.x86_64
[root@rhsqa14-vm1 ~]# 


[root@rhsqa14-vm1 ~]# glusterfs --version
glusterfs 3.7.0 built on May 15 2015 01:31:10
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.
[root@rhsqa14-vm1 ~]# 


[root@rhsqa14-vm1 ~]# gluster v attach-tier vol1 replica 2 10.70.46.233:/rhs/brick3/m1 10.70.46.236:/rhs/brick3/m1 10.70.46.233:/rhs/brick5/m1 10.70.4
6.236:/rhs/brick5/m1  
Attach tier is recommended only for testing purposes in this release. Do you want to continue? (y/n) y
volume attach-tier: success
volume rebalance: vol1: success: Rebalance on vol1 has been started successfully. Use rebalance status command to check status of the rebalance proces
s.
ID: b954b5e0-c4fa-4619-92ca-3c5e657269aa

You have new mail in /var/spool/mail/root
[root@rhsqa14-vm1 ~]# 
[root@rhsqa14-vm1 ~]# gluster v rebalance vol1  status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in s
ecs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     -----------
---
                               localhost                0        0Bytes             0             0             0               failed               0
.00
                            10.70.46.236                0        0Bytes             0             0             0               failed               0
.00
volume rebalance: vol1: success:
[root@rhsqa14-vm1 ~]# 
[root@rhsqa14-vm1 ~]# 
[root@rhsqa14-vm1 ~]# 



[root@rhsqa14-vm1 ~]# less /var/log/glusterfs/vol1-rebalance.log 
[2015-05-15 08:55:31.691592] I [MSGID: 100030] [glusterfsd.c:2294:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.0 (args: /usr/sbin/glusterfs -s localhost --volfile-id rebalance/vol1 --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *tier-dht.xattr-name=trusted.tier-gfid --xlator-option *dht.rebalance-cmd=6 --xlator-option *dht.node-uuid=87acbf29-e821-48bf-9aa8-bbda9321e609 --xlator-option *dht.commit-hash=2863506458 --socket-file /var/run/gluster/gluster-rebalance-37d0a9c0-21c1-46cf-ba95-419f9fbfbab0.sock --pid-file /var/lib/glusterd/vols/vol1/rebalance/87acbf29-e821-48bf-9aa8-bbda9321e609.pid -l /var/log/glusterfs/vol1-rebalance.log)
[2015-05-15 08:55:31.730515] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2015-05-15 08:55:36.690982] I [graph.c:269:gf_add_cmdline_options] 0-tier-dht: adding option 'commit-hash' for volume 'tier-dht' with value '2863506458'
[2015-05-15 08:55:36.691003] I [graph.c:269:gf_add_cmdline_options] 0-tier-dht: adding option 'node-uuid' for volume 'tier-dht' with value '87acbf29-e821-48bf-9aa8-bbda9321e609'
[2015-05-15 08:55:36.691014] I [graph.c:269:gf_add_cmdline_options] 0-tier-dht: adding option 'rebalance-cmd' for volume 'tier-dht' with value '6'
[2015-05-15 08:55:36.691025] I [graph.c:269:gf_add_cmdline_options] 0-tier-dht: adding option 'xattr-name' for volume 'tier-dht' with value 'trusted.tier-gfid'
[2015-05-15 08:55:36.691035] I [graph.c:269:gf_add_cmdline_options] 0-tier-dht: adding option 'readdir-optimize' for volume 'tier-dht' with value 'on'
[2015-05-15 08:55:36.691046] I [graph.c:269:gf_add_cmdline_options] 0-tier-dht: adding option 'assert-no-child-down' for volume 'tier-dht' with value 'yes'
[2015-05-15 08:55:36.691056] I [graph.c:269:gf_add_cmdline_options] 0-tier-dht: adding option 'lookup-unhashed' for volume 'tier-dht' with value 'yes'
[2015-05-15 08:55:36.691069] I [graph.c:269:gf_add_cmdline_options] 0-tier-dht: adding option 'use-readdirp' for volume 'tier-dht' with value 'yes'
[2015-05-15 08:55:36.691080] I [graph.c:269:gf_add_cmdline_options] 0-vol1-hot-dht: adding option 'commit-hash' for volume 'vol1-hot-dht' with value '2863506458'
[2015-05-15 08:55:36.692412] I [graph.c:269:gf_add_cmdline_options] 0-vol1-hot-dht: adding option 'node-uuid' for volume 'vol1-hot-dht' with value '87acbf29-e821-48bf-9aa8-bbda9321e609'
[2015-05-15 08:55:36.692425] I [graph.c:269:gf_add_cmdline_options] 0-vol1-hot-dht: adding option 'rebalance-cmd' for volume 'vol1-hot-dht' with value '6'
[2015-05-15 08:55:36.692437] I [graph.c:269:gf_add_cmdline_options] 0-vol1-hot-dht: adding option 'readdir-optimize' for volume 'vol1-hot-dht' with value 'on'
[2015-05-15 08:55:36.692449] I [graph.c:269:gf_add_cmdline_options] 0-vol1-hot-dht: adding option 'assert-no-child-down' for volume 'vol1-hot-dht' with value 'yes'
[2015-05-15 08:55:36.692459] I [graph.c:269:gf_add_cmdline_options] 0-vol1-hot-dht: adding option 'lookup-unhashed' for volume 'vol1-hot-dht' with val...skipping...
ith value 'yes'
[2015-05-15 08:57:58.911456] I [graph.c:269:gf_add_cmdline_options] 0-vol1-cold-dht: adding option 'lookup-unhashed' for volume 'vol1-cold-dht' with value 'yes'
[2015-05-15 08:57:58.911466] I [graph.c:269:gf_add_cmdline_options] 0-vol1-cold-dht: adding option 'use-readdirp' for volume 'vol1-cold-dht' with value 'yes'
[2015-05-15 08:57:58.911477] I [graph.c:269:gf_add_cmdline_options] 0-vol1-cold-replicate-1: adding option 'readdir-failover' for volume 'vol1-cold-replicate-1' with value 'off'
[2015-05-15 08:57:58.911486] I [graph.c:269:gf_add_cmdline_options] 0-vol1-cold-replicate-1: adding option 'entry-self-heal' for volume 'vol1-cold-replicate-1' with value 'off'
[2015-05-15 08:57:58.911496] I [graph.c:269:gf_add_cmdline_options] 0-vol1-cold-replicate-1: adding option 'metadata-self-heal' for volume 'vol1-cold-replicate-1' with value 'off'
[2015-05-15 08:57:58.911505] I [graph.c:269:gf_add_cmdline_options] 0-vol1-cold-replicate-1: adding option 'data-self-heal' for volume 'vol1-cold-replicate-1' with value 'off'
[2015-05-15 08:57:58.911517] I [graph.c:269:gf_add_cmdline_options] 0-vol1-cold-replicate-0: adding option 'readdir-failover' for volume 'vol1-cold-replicate-0' with value 'off'
[2015-05-15 08:57:58.911527] I [graph.c:269:gf_add_cmdline_options] 0-vol1-cold-replicate-0: adding option 'entry-self-heal' for volume 'vol1-cold-replicate-0' with value 'off'
[2015-05-15 08:57:58.911536] I [graph.c:269:gf_add_cmdline_options] 0-vol1-cold-replicate-0: adding option 'metadata-self-heal' for volume 'vol1-cold-replicate-0' with value 'off'
[2015-05-15 08:57:58.911545] I [graph.c:269:gf_add_cmdline_options] 0-vol1-cold-replicate-0: adding option 'data-self-heal' for volume 'vol1-cold-replicate-0' with value 'off'
[2015-05-15 08:57:58.912392] I [dht-shared.c:598:dht_init] 0-tier-dht: dht_init using commit hash 2863507596
[2015-05-15 08:57:58.912452] I [dht-shared.c:299:dht_parse_decommissioned_bricks] 0-tier-dht: decommissioning subvolume vol1-hot-dht
[2015-05-15 08:57:58.913862] E [MSGID: 109037] [tier.c:1007:tier_load_externals] 0-tier-dht: Error loading libgfdb.so /usr/lib64/libgfdb.so: cannot open shared object file: No such file or directory

[2015-05-15 08:57:58.913899] E [MSGID: 109037] [tier.c:1070:tier_init] 0-tier-dht: Could not load externals. Aborting
[2015-05-15 08:57:58.913946] E [xlator.c:426:xlator_init] 0-tier-dht: Initialization of volume 'tier-dht' failed, review your volfile again
[2015-05-15 08:57:58.913959] E [graph.c:322:glusterfs_graph_init] 0-tier-dht: initializing translator failed
[2015-05-15 08:57:58.913967] E [graph.c:661:glusterfs_graph_activate] 0-graph: init failed
[2015-05-15 08:57:58.914373] W [glusterfsd.c:1219:cleanup_and_exit] (--> 0-: received signum (0), shutting down
(END)

--- Additional comment from Anand Avati on 2015-05-16 04:33:24 EDT ---

REVIEW: http://review.gluster.org/10799 (cluster/tier: load libgfdb.so properly in all cases) posted (#1) for review on release-3.7 by Niels de Vos (ndevos)
Comment 3 Triveni Rao 2015-06-11 17:41:50 UTC 
after attaching the tier now it doesnot show as failed but in-progress state for longer period.

it doesnot complete rebalance.

[root@rhsqa14-vm1 ~]# gluster v rebalance venus status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                0        0Bytes             0             0             0          in progress             883.00
                            10.70.47.163                0        0Bytes             0             0             0          in progress             883.00
volume rebalance: venus: success: 
[root@rhsqa14-vm1 ~]# gluster volume rebalance venus tier status
Node                 Promoted files       Demoted files        Status              
---------            ---------            ---------            ---------           
localhost            0                    0                    in progress         
10.70.47.163         0                    0                    in progress         
volume rebalance: venus: success: 
[root@rhsqa14-vm1 ~]# 
[root@rhsqa14-vm1 ~]# 
[root@rhsqa14-vm1 ~]# 
[root@rhsqa14-vm1 ~]# gluster v rebalance venus status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                0        0Bytes             0             0             0          in progress            1401.00
                            10.70.47.163                0        0Bytes             0             0             0          in progress            1402.00
volume rebalance: venus: success: 
[root@rhsqa14-vm1 ~]#