+++ This bug was initially created as a clone of Bug #1221061 +++ Description of problem: Detaching tier start failed on dist-rep volume Version-Release number of selected component (if applicable): root@rhsqa14-vm3 ~]# glusterfs --version glusterfs 3.7.0beta2 built on May 11 2015 01:27:45 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/> GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation. You have new mail in /var/spool/mail/root [root@rhsqa14-vm3 ~]# [root@rhsqa14-vm3 ~]# [root@rhsqa14-vm3 ~]# rpm -qa | grep gluster glusterfs-libs-3.7.0beta2-0.0.el6.x86_64 glusterfs-fuse-3.7.0beta2-0.0.el6.x86_64 glusterfs-rdma-3.7.0beta2-0.0.el6.x86_64 glusterfs-3.7.0beta2-0.0.el6.x86_64 glusterfs-api-3.7.0beta2-0.0.el6.x86_64 glusterfs-cli-3.7.0beta2-0.0.el6.x86_64 glusterfs-geo-replication-3.7.0beta2-0.0.el6.x86_64 glusterfs-extra-xlators-3.7.0beta2-0.0.el6.x86_64 glusterfs-client-xlators-3.7.0beta2-0.0.el6.x86_64 glusterfs-server-3.7.0beta2-0.0.el6.x86_64 [root@rhsqa14-vm3 ~]# How reproducible: easily Steps to Reproduce: 1. create dist-rep volume, attach tier. 2. detach tier from the volume 3. detaching tier start failed Actual results: detach tier start failed Expected results: It should not fail. Additional info: [root@rhsqa14-vm3 ~]# gluster v info V1 Volume Name: V1 Type: Tier Volume ID: 99f99d6d-b24d-4cc8-96e0-25444dbf10fd Status: Started Number of Bricks: 6 Transport-type: tcp Hot Tier : Hot Tier Type : Replicate Number of Bricks: 1 x 2 = 2 Brick1: 10.70.46.240:/rhs/brick3/t2 Brick2: 10.70.46.243:/rhs/brick3/t2 Cold Bricks: Cold Tier Type : Distributed-Replicate Number of Bricks: 2 x 2 = 4 Brick3: 10.70.46.243:/rhs/brick1/t2 Brick4: 10.70.46.240:/rhs/brick1/t2 Brick5: 10.70.46.243:/rhs/brick2/t2 Brick6: 10.70.46.240:/rhs/brick2/t2 Options Reconfigured: features.quota-deem-statfs: on features.inode-quota: on features.quota: on cluster.min-free-disk: 10 performance.readdir-ahead: on [root@rhsqa14-vm3 ~]# root@rhsqa14-vm3 ~]# gluster v detach-tier V1 start volume detach-tier start: failed: Commit failed on localhost. Please check the log file for more details. [root@rhsqa14-vm3 ~]# Log messages: [2015-05-13 08:46:13.905887] W [socket.c:642:__socket_rwv] 0-quotad: readv on /var/run/gluster/a450b664cf93d1dbf6c16c47de60860e.socket failed (Invalid argument) [2015-05-13 08:46:15.906141] W [socket.c:3059:socket_connect] 0-snapd: Ignore failed connection attempt on /var/run/gluster/91270fab7e9fc0459f2cc706e0e0ef5f.socket, (No such file or directory) [2015-05-13 08:46:15.906165] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/98d97d20378bb5f3b753533d86271c8d.socket failed (Invalid argument) [2015-05-13 08:46:15.906248] W [socket.c:642:__socket_rwv] 0-snapd: readv on /var/run/gluster/91270fab7e9fc0459f2cc706e0e0ef5f.socket failed (Invalid argument) [2015-05-13 08:46:16.910658] W [socket.c:642:__socket_rwv] 0-quotad: readv on /var/run/gluster/a450b664cf93d1dbf6c16c47de60860e.socket failed (Invalid argument) [2015-05-13 08:46:17.454556] I [glusterd-brick-ops.c:770:__glusterd_handle_remove_brick] 0-management: Received rem brick req [2015-05-13 08:46:17.489518] I [glusterd-utils.c:8599:glusterd_generate_and_set_task_id] 0-management: Generated task-id 955bc0fd-5253-4537-ae0c-8fe1ca9d02ee for key remove-brick-id [2015-05-13 08:46:18.300221] E [graph.y:153:new_volume] 0-parser: Line 1295: volume 'tier-dht' defined again [2015-05-13 08:46:18.315339] W [glusterd-brick-ops.c:2253:glusterd_op_remove_brick] 0-management: Unable to reconfigure NFS-Server [2015-05-13 08:46:18.315381] E [glusterd-syncop.c:1372:gd_commit_op_phase] 0-management: Commit of operation 'Volume Remove brick' failed on localhost [2015-05-13 08:46:18.911020] W [socket.c:3059:socket_connect] 0-snapd: Ignore failed connection attempt on /var/run/gluster/91270fab7e9fc0459f2cc706e0e0ef5f.socket, (No such file or directory) [2015-05-13 08:46:18.913711] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/98d97d20378bb5f3b753533d86271c8d.socket failed (Invalid argument) [2015-05-13 08:46:18.914225] W [socket.c:642:__socket_rwv] 0-snapd: readv on /var/run/gluster/91270fab7e9fc0459f2cc706e0e0ef5f.socket failed (Invalid argument) [2015-05-13 08:46:19.917878] W [socket.c:642:__socket_rwv] 0-quotad: readv on /var/run/gluster/a450b664cf93d1dbf6c16c47de60860e.socket failed (Invalid argument) [2015-05-13 08:46:21.918057] W [socket.c:3059:socket_connect] 0-snapd: Ignore failed connection attempt on /var/run/gluster/91270fab7e9fc0459f2cc706e0e0ef5f.socket, (No such file or directory) [2015-05-13 08:46:21.918160] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/98d97d20378bb5f3b753533d86271c8d.socket failed (Invalid argument) --- Additional comment from Mohammed Rafi KC on 2015-05-14 02:18:39 EDT --- I couldn't reproduce the issue when follow the steps given. Can you make sure this is reproducible ? --- Additional comment from Triveni Rao on 2015-05-15 06:50:31 EDT --- i could reproduce this issue on new downstream build: root@rhsqa14-vm1 ~]# rpm -qa | grep gluster glusterfs-3.7.0-2.el6rhs.x86_64 glusterfs-cli-3.7.0-2.el6rhs.x86_64 glusterfs-libs-3.7.0-2.el6rhs.x86_64 glusterfs-client-xlators-3.7.0-2.el6rhs.x86_64 glusterfs-api-3.7.0-2.el6rhs.x86_64 glusterfs-server-3.7.0-2.el6rhs.x86_64 glusterfs-fuse-3.7.0-2.el6rhs.x86_64 [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# glusterfs --version glusterfs 3.7.0 built on May 15 2015 01:31:10 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/> GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation. [root@rhsqa14-vm1 ~]# root@rhsqa14-vm1 ~]# gluster v info vol2 Volume Name: vol2 Type: Tier Volume ID: 46c79842-2d5d-4f0a-9776-10504fbc93e4 Status: Started Number of Bricks: 8 Transport-type: tcp Hot Tier : Hot Tier Type : Distributed-Replicate Number of Bricks: 2 x 2 = 4 Brick1: 10.70.46.236:/rhs/brick5/v2 Brick2: 10.70.46.233:/rhs/brick5/v2 Brick3: 10.70.46.236:/rhs/brick3/v2 Brick4: 10.70.46.233:/rhs/brick3/v2 Cold Bricks: Cold Tier Type : Distributed-Replicate Number of Bricks: 2 x 2 = 4 Brick5: 10.70.46.233:/rhs/brick1/v2 Brick6: 10.70.46.236:/rhs/brick1/v2 Brick7: 10.70.46.233:/rhs/brick2/v2 Brick8: 10.70.46.236:/rhs/brick2/v2 Options Reconfigured: features.uss: enable features.inode-quota: on features.quota: on cluster.min-free-disk: 10 performance.readdir-ahead: on [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# gluster v detach-tier vol2 start volume detach-tier start: failed: Commit failed on localhost. Please check the log file for more details. You have new mail in /var/spool/mail/root [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# less /var/log/glusterfs/etc-glusterfs-glusterd.vol.log [2015-05-14 06:34:47.596813] I [MSGID: 100030] [glusterfsd.c:2294:main] 0-glusterd: Started running glusterd version 3.7.0beta2 (args: glusterd --xlator-option *.upgrade=on -N) [2015-05-14 06:34:47.605211] I [graph.c:269:gf_add_cmdline_options] 0-management: adding option 'upgrade' for volume 'management' with value 'on' [2015-05-14 06:34:47.605328] I [glusterd.c:1282:init] 0-management: Maximum allowed open file descriptors set to 65536 [2015-05-14 06:34:47.605370] I [glusterd.c:1327:init] 0-management: Using /var/lib/glusterd as working directory [2015-05-14 06:34:47.630137] E [rpc-transport.c:291:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.7.0beta2/rpc-transport/rdma.so: cannot open shared object file: No such file or directory [2015-05-14 06:34:47.630198] W [rpc-transport.c:295:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine [2015-05-14 06:34:47.630218] W [rpcsvc.c:1595:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed [2015-05-14 06:34:47.630235] E [glusterd.c:1515:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2015-05-14 06:34:47.649973] I [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system [2015-05-14 06:34:47.650135] E [store.c:432:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory) [2015-05-14 06:34:47.650161] E [store.c:432:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory) [2015-05-14 06:34:47.650173] I [glusterd-store.c:2005:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 30700 [2015-05-14 06:34:47.650462] E [store.c:432:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory) [2015-05-14 06:34:47.650781] I [glusterd.c:184:glusterd_uuid_generate_save] 0-management: generated UUID: 87acbf29-e821-48bf-9aa8-bbda9321e609 [2015-05-14 06:34:47.817571] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600 [2015-05-14 06:34:47.818755] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600 [2015-05-14 06:34:47.819295] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600 [2015-05-14 06:34:47.819873] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600 [2015-05-14 06:34:47.820373] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600 [2015-05-14 06:34:47.820867] I [glusterd-store.c:3371:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. [2015-05-14 06:34:47.821075] E [store.c:432:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options, returned error: (No such file or directory) Final graph: +------------------------------------------------------------------------------+ ...skipping... gument) [2015-05-15 10:49:42.993889] W [socket.c:642:__socket_rwv] 0-quotad: readv on /var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid argument) [2015-05-15 10:49:45.994646] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/cd9131b65e498c98e62c155fa7f02179.socket failed (Invalid argument) [2015-05-15 10:49:45.994734] W [socket.c:642:__socket_rwv] 0-quotad: readv on /var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid argument) [2015-05-15 10:49:48.995525] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/cd9131b65e498c98e62c155fa7f02179.socket failed (Invalid argument) [2015-05-15 10:49:48.995645] W [socket.c:642:__socket_rwv] 0-quotad: readv on /var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid argument) [2015-05-15 10:49:51.996236] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/cd9131b65e498c98e62c155fa7f02179.socket failed (Invalid argument) [2015-05-15 10:49:51.996339] W [socket.c:642:__socket_rwv] 0-quotad: readv on /var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid argument) [2015-05-15 10:49:54.997046] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/cd9131b65e498c98e62c155fa7f02179.socket failed (Invalid argument) [2015-05-15 10:49:54.997175] W [socket.c:642:__socket_rwv] 0-quotad: readv on /var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid argument) [2015-05-15 10:49:57.997799] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/cd9131b65e498c98e62c155fa7f02179.socket failed (Invalid argument) [2015-05-15 10:49:57.997881] W [socket.c:642:__socket_rwv] 0-quotad: readv on /var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid argument) [2015-05-15 10:50:00.998300] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/cd9131b65e498c98e62c155fa7f02179.socket failed (Invalid argument) [2015-05-15 10:50:00.998384] W [socket.c:642:__socket_rwv] 0-quotad: readv on /var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid argument) [2015-05-15 10:50:03.999585] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/cd9131b65e498c98e62c155fa7f02179.socket failed (Invalid argument) [2015-05-15 10:50:03.999684] W [socket.c:642:__socket_rwv] 0-quotad: readv on /var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid argument) --- Additional comment from Mohammed Rafi KC on 2015-05-19 05:41:17 EDT --- upstream patch : http://review.gluster.org/10820 --- Additional comment from Mohammed Rafi KC on 2015-05-19 05:47:10 EDT --- http://review.gluster.org/10820
this bug is not re-producible with below build: [root@rhsqa14-vm1 ~]# glusterfs --version glusterfs 3.7.1 built on Jun 9 2015 02:31:54 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/> GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation. [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# rpm -qa | grep gluster glusterfs-3.7.1-1.el6rhs.x86_64 glusterfs-cli-3.7.1-1.el6rhs.x86_64 glusterfs-libs-3.7.1-1.el6rhs.x86_64 glusterfs-client-xlators-3.7.1-1.el6rhs.x86_64 glusterfs-fuse-3.7.1-1.el6rhs.x86_64 glusterfs-server-3.7.1-1.el6rhs.x86_64 glusterfs-api-3.7.1-1.el6rhs.x86_64 [root@rhsqa14-vm1 ~]# this bug is verified
[root@rhsqa14-vm1 ~]# gluster v detach-tier mix start volume detach-tier start: success ID: 5f08c911-0007-4fd5-b88f-f8ba6b3aefa2 [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# gluster v detach-tier mix status Node Rebalanced-files size scanned failures skipped status run time in s --------- ----------- ----------- ----------- ----------- ----------- ------------ ----------- localhost 0 0Bytes 0 0 0 completed 0 10.70.47.163 0 0Bytes 0 0 0 completed 0 [root@rhsqa14-vm1 ~]# gluster v detach-tier mix commit volume detach-tier commit: success Check the detached bricks to ensure all files are migrated. If files with data are found on the brick path, copy them via a gluster mount point before re-purposing the removed brick. [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# gluster v info mix Volume Name: mix Type: Distribute Volume ID: 0f1a01aa-9b1a-4cd5-bc9c-64b14630f264 Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: 10.70.47.165:/rhs/brick3/h0 Brick2: 10.70.47.163:/rhs/brick3/h0 Options Reconfigured: features.quota-deem-statfs: on features.uss: enable features.inode-quota: on features.quota: on cluster.min-free-disk: 10 performance.readdir-ahead: on [root@rhsqa14-vm1 ~]# Tested only on dist-rep volume
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-1495.html