Description of problem: Detaching tier start failed on dist-rep volume Version-Release number of selected component (if applicable): root@rhsqa14-vm3 ~]# glusterfs --version glusterfs 3.7.0beta2 built on May 11 2015 01:27:45 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/> GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation. You have new mail in /var/spool/mail/root [root@rhsqa14-vm3 ~]# [root@rhsqa14-vm3 ~]# [root@rhsqa14-vm3 ~]# rpm -qa | grep gluster glusterfs-libs-3.7.0beta2-0.0.el6.x86_64 glusterfs-fuse-3.7.0beta2-0.0.el6.x86_64 glusterfs-rdma-3.7.0beta2-0.0.el6.x86_64 glusterfs-3.7.0beta2-0.0.el6.x86_64 glusterfs-api-3.7.0beta2-0.0.el6.x86_64 glusterfs-cli-3.7.0beta2-0.0.el6.x86_64 glusterfs-geo-replication-3.7.0beta2-0.0.el6.x86_64 glusterfs-extra-xlators-3.7.0beta2-0.0.el6.x86_64 glusterfs-client-xlators-3.7.0beta2-0.0.el6.x86_64 glusterfs-server-3.7.0beta2-0.0.el6.x86_64 [root@rhsqa14-vm3 ~]# How reproducible: easily Steps to Reproduce: 1. create dist-rep volume, attach tier. 2. detach tier from the volume 3. detaching tier start failed Actual results: detach tier start failed Expected results: It should not fail. Additional info: [root@rhsqa14-vm3 ~]# gluster v info V1 Volume Name: V1 Type: Tier Volume ID: 99f99d6d-b24d-4cc8-96e0-25444dbf10fd Status: Started Number of Bricks: 6 Transport-type: tcp Hot Tier : Hot Tier Type : Replicate Number of Bricks: 1 x 2 = 2 Brick1: 10.70.46.240:/rhs/brick3/t2 Brick2: 10.70.46.243:/rhs/brick3/t2 Cold Bricks: Cold Tier Type : Distributed-Replicate Number of Bricks: 2 x 2 = 4 Brick3: 10.70.46.243:/rhs/brick1/t2 Brick4: 10.70.46.240:/rhs/brick1/t2 Brick5: 10.70.46.243:/rhs/brick2/t2 Brick6: 10.70.46.240:/rhs/brick2/t2 Options Reconfigured: features.quota-deem-statfs: on features.inode-quota: on features.quota: on cluster.min-free-disk: 10 performance.readdir-ahead: on [root@rhsqa14-vm3 ~]# root@rhsqa14-vm3 ~]# gluster v detach-tier V1 start volume detach-tier start: failed: Commit failed on localhost. Please check the log file for more details. [root@rhsqa14-vm3 ~]# Log messages: [2015-05-13 08:46:13.905887] W [socket.c:642:__socket_rwv] 0-quotad: readv on /var/run/gluster/a450b664cf93d1dbf6c16c47de60860e.socket failed (Invalid argument) [2015-05-13 08:46:15.906141] W [socket.c:3059:socket_connect] 0-snapd: Ignore failed connection attempt on /var/run/gluster/91270fab7e9fc0459f2cc706e0e0ef5f.socket, (No such file or directory) [2015-05-13 08:46:15.906165] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/98d97d20378bb5f3b753533d86271c8d.socket failed (Invalid argument) [2015-05-13 08:46:15.906248] W [socket.c:642:__socket_rwv] 0-snapd: readv on /var/run/gluster/91270fab7e9fc0459f2cc706e0e0ef5f.socket failed (Invalid argument) [2015-05-13 08:46:16.910658] W [socket.c:642:__socket_rwv] 0-quotad: readv on /var/run/gluster/a450b664cf93d1dbf6c16c47de60860e.socket failed (Invalid argument) [2015-05-13 08:46:17.454556] I [glusterd-brick-ops.c:770:__glusterd_handle_remove_brick] 0-management: Received rem brick req [2015-05-13 08:46:17.489518] I [glusterd-utils.c:8599:glusterd_generate_and_set_task_id] 0-management: Generated task-id 955bc0fd-5253-4537-ae0c-8fe1ca9d02ee for key remove-brick-id [2015-05-13 08:46:18.300221] E [graph.y:153:new_volume] 0-parser: Line 1295: volume 'tier-dht' defined again [2015-05-13 08:46:18.315339] W [glusterd-brick-ops.c:2253:glusterd_op_remove_brick] 0-management: Unable to reconfigure NFS-Server [2015-05-13 08:46:18.315381] E [glusterd-syncop.c:1372:gd_commit_op_phase] 0-management: Commit of operation 'Volume Remove brick' failed on localhost [2015-05-13 08:46:18.911020] W [socket.c:3059:socket_connect] 0-snapd: Ignore failed connection attempt on /var/run/gluster/91270fab7e9fc0459f2cc706e0e0ef5f.socket, (No such file or directory) [2015-05-13 08:46:18.913711] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/98d97d20378bb5f3b753533d86271c8d.socket failed (Invalid argument) [2015-05-13 08:46:18.914225] W [socket.c:642:__socket_rwv] 0-snapd: readv on /var/run/gluster/91270fab7e9fc0459f2cc706e0e0ef5f.socket failed (Invalid argument) [2015-05-13 08:46:19.917878] W [socket.c:642:__socket_rwv] 0-quotad: readv on /var/run/gluster/a450b664cf93d1dbf6c16c47de60860e.socket failed (Invalid argument) [2015-05-13 08:46:21.918057] W [socket.c:3059:socket_connect] 0-snapd: Ignore failed connection attempt on /var/run/gluster/91270fab7e9fc0459f2cc706e0e0ef5f.socket, (No such file or directory) [2015-05-13 08:46:21.918160] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/98d97d20378bb5f3b753533d86271c8d.socket failed (Invalid argument)
I couldn't reproduce the issue when follow the steps given. Can you make sure this is reproducible ?
i could reproduce this issue on new downstream build: root@rhsqa14-vm1 ~]# rpm -qa | grep gluster glusterfs-3.7.0-2.el6rhs.x86_64 glusterfs-cli-3.7.0-2.el6rhs.x86_64 glusterfs-libs-3.7.0-2.el6rhs.x86_64 glusterfs-client-xlators-3.7.0-2.el6rhs.x86_64 glusterfs-api-3.7.0-2.el6rhs.x86_64 glusterfs-server-3.7.0-2.el6rhs.x86_64 glusterfs-fuse-3.7.0-2.el6rhs.x86_64 [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# glusterfs --version glusterfs 3.7.0 built on May 15 2015 01:31:10 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/> GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation. [root@rhsqa14-vm1 ~]# root@rhsqa14-vm1 ~]# gluster v info vol2 Volume Name: vol2 Type: Tier Volume ID: 46c79842-2d5d-4f0a-9776-10504fbc93e4 Status: Started Number of Bricks: 8 Transport-type: tcp Hot Tier : Hot Tier Type : Distributed-Replicate Number of Bricks: 2 x 2 = 4 Brick1: 10.70.46.236:/rhs/brick5/v2 Brick2: 10.70.46.233:/rhs/brick5/v2 Brick3: 10.70.46.236:/rhs/brick3/v2 Brick4: 10.70.46.233:/rhs/brick3/v2 Cold Bricks: Cold Tier Type : Distributed-Replicate Number of Bricks: 2 x 2 = 4 Brick5: 10.70.46.233:/rhs/brick1/v2 Brick6: 10.70.46.236:/rhs/brick1/v2 Brick7: 10.70.46.233:/rhs/brick2/v2 Brick8: 10.70.46.236:/rhs/brick2/v2 Options Reconfigured: features.uss: enable features.inode-quota: on features.quota: on cluster.min-free-disk: 10 performance.readdir-ahead: on [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# gluster v detach-tier vol2 start volume detach-tier start: failed: Commit failed on localhost. Please check the log file for more details. You have new mail in /var/spool/mail/root [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# less /var/log/glusterfs/etc-glusterfs-glusterd.vol.log [2015-05-14 06:34:47.596813] I [MSGID: 100030] [glusterfsd.c:2294:main] 0-glusterd: Started running glusterd version 3.7.0beta2 (args: glusterd --xlator-option *.upgrade=on -N) [2015-05-14 06:34:47.605211] I [graph.c:269:gf_add_cmdline_options] 0-management: adding option 'upgrade' for volume 'management' with value 'on' [2015-05-14 06:34:47.605328] I [glusterd.c:1282:init] 0-management: Maximum allowed open file descriptors set to 65536 [2015-05-14 06:34:47.605370] I [glusterd.c:1327:init] 0-management: Using /var/lib/glusterd as working directory [2015-05-14 06:34:47.630137] E [rpc-transport.c:291:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.7.0beta2/rpc-transport/rdma.so: cannot open shared object file: No such file or directory [2015-05-14 06:34:47.630198] W [rpc-transport.c:295:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine [2015-05-14 06:34:47.630218] W [rpcsvc.c:1595:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed [2015-05-14 06:34:47.630235] E [glusterd.c:1515:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2015-05-14 06:34:47.649973] I [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system [2015-05-14 06:34:47.650135] E [store.c:432:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory) [2015-05-14 06:34:47.650161] E [store.c:432:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory) [2015-05-14 06:34:47.650173] I [glusterd-store.c:2005:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 30700 [2015-05-14 06:34:47.650462] E [store.c:432:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory) [2015-05-14 06:34:47.650781] I [glusterd.c:184:glusterd_uuid_generate_save] 0-management: generated UUID: 87acbf29-e821-48bf-9aa8-bbda9321e609 [2015-05-14 06:34:47.817571] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600 [2015-05-14 06:34:47.818755] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600 [2015-05-14 06:34:47.819295] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600 [2015-05-14 06:34:47.819873] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600 [2015-05-14 06:34:47.820373] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600 [2015-05-14 06:34:47.820867] I [glusterd-store.c:3371:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. [2015-05-14 06:34:47.821075] E [store.c:432:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options, returned error: (No such file or directory) Final graph: +------------------------------------------------------------------------------+ ...skipping... gument) [2015-05-15 10:49:42.993889] W [socket.c:642:__socket_rwv] 0-quotad: readv on /var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid argument) [2015-05-15 10:49:45.994646] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/cd9131b65e498c98e62c155fa7f02179.socket failed (Invalid argument) [2015-05-15 10:49:45.994734] W [socket.c:642:__socket_rwv] 0-quotad: readv on /var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid argument) [2015-05-15 10:49:48.995525] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/cd9131b65e498c98e62c155fa7f02179.socket failed (Invalid argument) [2015-05-15 10:49:48.995645] W [socket.c:642:__socket_rwv] 0-quotad: readv on /var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid argument) [2015-05-15 10:49:51.996236] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/cd9131b65e498c98e62c155fa7f02179.socket failed (Invalid argument) [2015-05-15 10:49:51.996339] W [socket.c:642:__socket_rwv] 0-quotad: readv on /var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid argument) [2015-05-15 10:49:54.997046] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/cd9131b65e498c98e62c155fa7f02179.socket failed (Invalid argument) [2015-05-15 10:49:54.997175] W [socket.c:642:__socket_rwv] 0-quotad: readv on /var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid argument) [2015-05-15 10:49:57.997799] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/cd9131b65e498c98e62c155fa7f02179.socket failed (Invalid argument) [2015-05-15 10:49:57.997881] W [socket.c:642:__socket_rwv] 0-quotad: readv on /var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid argument) [2015-05-15 10:50:00.998300] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/cd9131b65e498c98e62c155fa7f02179.socket failed (Invalid argument) [2015-05-15 10:50:00.998384] W [socket.c:642:__socket_rwv] 0-quotad: readv on /var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid argument) [2015-05-15 10:50:03.999585] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/cd9131b65e498c98e62c155fa7f02179.socket failed (Invalid argument) [2015-05-15 10:50:03.999684] W [socket.c:642:__socket_rwv] 0-quotad: readv on /var/run/gluster/87382ddc53b616370f1b86e694eba7fc.socket failed (Invalid argument)
upstream patch : http://review.gluster.org/10820
http://review.gluster.org/10820
This bug was ON_QA status, and on GlusterFS product in bugzilla, we don't have that as a valid status. We are closing it as 'CURRENT RELEASE ' to indicate the availability of the fix, please reopen if found again.