Bug 1221534
| Summary: | rebalance failed after attaching the tier to the volume. | |||
|---|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Triveni Rao <trao> | |
| Component: | tiering | Assignee: | Niels de Vos <ndevos> | |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | bugs <bugs> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 3.7.0 | CC: | bugs, ndevos, sashinde | |
| Target Milestone: | --- | Keywords: | TestBlocker | |
| Target Release: | --- | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | TIERING | |||
| Fixed In Version: | glusterfs-3.7.1 | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1222092 1224144 (view as bug list) | Environment: | ||
| Last Closed: | 2015-06-02 08:02:52 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1222092 | |||
| Bug Blocks: | 1221957, 1224144, 1229269, 1260923, 1273726, 1274411 | |||
|
Description
Triveni Rao
2015-05-14 10:09:33 UTC
This was reproducible on new downstream build also: [root@rhsqa14-vm1 ~]# rpm -qa | grep gluster glusterfs-3.7.0-2.el6rhs.x86_64 glusterfs-cli-3.7.0-2.el6rhs.x86_64 glusterfs-libs-3.7.0-2.el6rhs.x86_64 glusterfs-client-xlators-3.7.0-2.el6rhs.x86_64 glusterfs-api-3.7.0-2.el6rhs.x86_64 glusterfs-server-3.7.0-2.el6rhs.x86_64 glusterfs-fuse-3.7.0-2.el6rhs.x86_64 [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# glusterfs --version glusterfs 3.7.0 built on May 15 2015 01:31:10 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/> GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation. [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# gluster v attach-tier vol1 replica 2 10.70.46.233:/rhs/brick3/m1 10.70.46.236:/rhs/brick3/m1 10.70.46.233:/rhs/brick5/m1 10.70.4 6.236:/rhs/brick5/m1 Attach tier is recommended only for testing purposes in this release. Do you want to continue? (y/n) y volume attach-tier: success volume rebalance: vol1: success: Rebalance on vol1 has been started successfully. Use rebalance status command to check status of the rebalance proces s. ID: b954b5e0-c4fa-4619-92ca-3c5e657269aa You have new mail in /var/spool/mail/root [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# gluster v rebalance vol1 status Node Rebalanced-files size scanned failures skipped status run time in s ecs --------- ----------- ----------- ----------- ----------- ----------- ------------ ----------- --- localhost 0 0Bytes 0 0 0 failed 0 .00 10.70.46.236 0 0Bytes 0 0 0 failed 0 .00 volume rebalance: vol1: success: [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# less /var/log/glusterfs/vol1-rebalance.log [2015-05-15 08:55:31.691592] I [MSGID: 100030] [glusterfsd.c:2294:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.0 (args: /usr/sbin/glusterfs -s localhost --volfile-id rebalance/vol1 --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *tier-dht.xattr-name=trusted.tier-gfid --xlator-option *dht.rebalance-cmd=6 --xlator-option *dht.node-uuid=87acbf29-e821-48bf-9aa8-bbda9321e609 --xlator-option *dht.commit-hash=2863506458 --socket-file /var/run/gluster/gluster-rebalance-37d0a9c0-21c1-46cf-ba95-419f9fbfbab0.sock --pid-file /var/lib/glusterd/vols/vol1/rebalance/87acbf29-e821-48bf-9aa8-bbda9321e609.pid -l /var/log/glusterfs/vol1-rebalance.log) [2015-05-15 08:55:31.730515] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2015-05-15 08:55:36.690982] I [graph.c:269:gf_add_cmdline_options] 0-tier-dht: adding option 'commit-hash' for volume 'tier-dht' with value '2863506458' [2015-05-15 08:55:36.691003] I [graph.c:269:gf_add_cmdline_options] 0-tier-dht: adding option 'node-uuid' for volume 'tier-dht' with value '87acbf29-e821-48bf-9aa8-bbda9321e609' [2015-05-15 08:55:36.691014] I [graph.c:269:gf_add_cmdline_options] 0-tier-dht: adding option 'rebalance-cmd' for volume 'tier-dht' with value '6' [2015-05-15 08:55:36.691025] I [graph.c:269:gf_add_cmdline_options] 0-tier-dht: adding option 'xattr-name' for volume 'tier-dht' with value 'trusted.tier-gfid' [2015-05-15 08:55:36.691035] I [graph.c:269:gf_add_cmdline_options] 0-tier-dht: adding option 'readdir-optimize' for volume 'tier-dht' with value 'on' [2015-05-15 08:55:36.691046] I [graph.c:269:gf_add_cmdline_options] 0-tier-dht: adding option 'assert-no-child-down' for volume 'tier-dht' with value 'yes' [2015-05-15 08:55:36.691056] I [graph.c:269:gf_add_cmdline_options] 0-tier-dht: adding option 'lookup-unhashed' for volume 'tier-dht' with value 'yes' [2015-05-15 08:55:36.691069] I [graph.c:269:gf_add_cmdline_options] 0-tier-dht: adding option 'use-readdirp' for volume 'tier-dht' with value 'yes' [2015-05-15 08:55:36.691080] I [graph.c:269:gf_add_cmdline_options] 0-vol1-hot-dht: adding option 'commit-hash' for volume 'vol1-hot-dht' with value '2863506458' [2015-05-15 08:55:36.692412] I [graph.c:269:gf_add_cmdline_options] 0-vol1-hot-dht: adding option 'node-uuid' for volume 'vol1-hot-dht' with value '87acbf29-e821-48bf-9aa8-bbda9321e609' [2015-05-15 08:55:36.692425] I [graph.c:269:gf_add_cmdline_options] 0-vol1-hot-dht: adding option 'rebalance-cmd' for volume 'vol1-hot-dht' with value '6' [2015-05-15 08:55:36.692437] I [graph.c:269:gf_add_cmdline_options] 0-vol1-hot-dht: adding option 'readdir-optimize' for volume 'vol1-hot-dht' with value 'on' [2015-05-15 08:55:36.692449] I [graph.c:269:gf_add_cmdline_options] 0-vol1-hot-dht: adding option 'assert-no-child-down' for volume 'vol1-hot-dht' with value 'yes' [2015-05-15 08:55:36.692459] I [graph.c:269:gf_add_cmdline_options] 0-vol1-hot-dht: adding option 'lookup-unhashed' for volume 'vol1-hot-dht' with val...skipping... ith value 'yes' [2015-05-15 08:57:58.911456] I [graph.c:269:gf_add_cmdline_options] 0-vol1-cold-dht: adding option 'lookup-unhashed' for volume 'vol1-cold-dht' with value 'yes' [2015-05-15 08:57:58.911466] I [graph.c:269:gf_add_cmdline_options] 0-vol1-cold-dht: adding option 'use-readdirp' for volume 'vol1-cold-dht' with value 'yes' [2015-05-15 08:57:58.911477] I [graph.c:269:gf_add_cmdline_options] 0-vol1-cold-replicate-1: adding option 'readdir-failover' for volume 'vol1-cold-replicate-1' with value 'off' [2015-05-15 08:57:58.911486] I [graph.c:269:gf_add_cmdline_options] 0-vol1-cold-replicate-1: adding option 'entry-self-heal' for volume 'vol1-cold-replicate-1' with value 'off' [2015-05-15 08:57:58.911496] I [graph.c:269:gf_add_cmdline_options] 0-vol1-cold-replicate-1: adding option 'metadata-self-heal' for volume 'vol1-cold-replicate-1' with value 'off' [2015-05-15 08:57:58.911505] I [graph.c:269:gf_add_cmdline_options] 0-vol1-cold-replicate-1: adding option 'data-self-heal' for volume 'vol1-cold-replicate-1' with value 'off' [2015-05-15 08:57:58.911517] I [graph.c:269:gf_add_cmdline_options] 0-vol1-cold-replicate-0: adding option 'readdir-failover' for volume 'vol1-cold-replicate-0' with value 'off' [2015-05-15 08:57:58.911527] I [graph.c:269:gf_add_cmdline_options] 0-vol1-cold-replicate-0: adding option 'entry-self-heal' for volume 'vol1-cold-replicate-0' with value 'off' [2015-05-15 08:57:58.911536] I [graph.c:269:gf_add_cmdline_options] 0-vol1-cold-replicate-0: adding option 'metadata-self-heal' for volume 'vol1-cold-replicate-0' with value 'off' [2015-05-15 08:57:58.911545] I [graph.c:269:gf_add_cmdline_options] 0-vol1-cold-replicate-0: adding option 'data-self-heal' for volume 'vol1-cold-replicate-0' with value 'off' [2015-05-15 08:57:58.912392] I [dht-shared.c:598:dht_init] 0-tier-dht: dht_init using commit hash 2863507596 [2015-05-15 08:57:58.912452] I [dht-shared.c:299:dht_parse_decommissioned_bricks] 0-tier-dht: decommissioning subvolume vol1-hot-dht [2015-05-15 08:57:58.913862] E [MSGID: 109037] [tier.c:1007:tier_load_externals] 0-tier-dht: Error loading libgfdb.so /usr/lib64/libgfdb.so: cannot open shared object file: No such file or directory [2015-05-15 08:57:58.913899] E [MSGID: 109037] [tier.c:1070:tier_init] 0-tier-dht: Could not load externals. Aborting [2015-05-15 08:57:58.913946] E [xlator.c:426:xlator_init] 0-tier-dht: Initialization of volume 'tier-dht' failed, review your volfile again [2015-05-15 08:57:58.913959] E [graph.c:322:glusterfs_graph_init] 0-tier-dht: initializing translator failed [2015-05-15 08:57:58.913967] E [graph.c:661:glusterfs_graph_activate] 0-graph: init failed [2015-05-15 08:57:58.914373] W [glusterfsd.c:1219:cleanup_and_exit] (--> 0-: received signum (0), shutting down (END) REVIEW: http://review.gluster.org/10799 (cluster/tier: load libgfdb.so properly in all cases) posted (#1) for review on release-3.7 by Niels de Vos (ndevos) COMMIT: http://review.gluster.org/10799 committed in release-3.7 by Niels de Vos (ndevos) ------ commit 2bbc891f05d7d68c70f998b8276f23f18f13115e Author: Niels de Vos <ndevos> Date: Sat May 16 10:32:23 2015 +0200 cluster/tier: load libgfdb.so properly in all cases We should load libgfdb.so.0, not libgfdb.so Cherry picked from commit 628406f28364f6019261a3bb37335a494ccf8dda: > Change-Id: I7a0d64018ccd9893b1685de391e99b5392bd1879 > BUG: 1222092 > Signed-off-by: Dan Lambright <dlambrig> > Reviewed-on: http://review.gluster.org/10796 > Reviewed-by: Kaleb KEITHLEY <kkeithle> > Reviewed-by: Joseph Fernandes > Reviewed-by: Niels de Vos <ndevos> > Tested-by: Gluster Build System <jenkins.com> Change-Id: I7a0d64018ccd9893b1685de391e99b5392bd1879 BUG: 1221534 Signed-off-by: Niels de Vos <ndevos> Reviewed-on: http://review.gluster.org/10799 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Dan Lambright <dlambrig> Reviewed-by: mohammed rafi kc <rkavunga> This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.1, please reopen this bug report. glusterfs-3.7.1 has been announced on the Gluster Packaging mailinglist [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.packaging/1 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user |