Description of problem: transport endpoint is not connected message when tried to access .snaps dir on an ec volume with distribute hot tier attached and IO is going on. Version-Release number of selected component (if applicable): [root@rhsqa14-vm1 ~]# rpm -qa | grep gluster glusterfs-3.7.0-2.el6rhs.x86_64 glusterfs-cli-3.7.0-2.el6rhs.x86_64 glusterfs-libs-3.7.0-2.el6rhs.x86_64 glusterfs-client-xlators-3.7.0-2.el6rhs.x86_64 glusterfs-api-3.7.0-2.el6rhs.x86_64 glusterfs-server-3.7.0-2.el6rhs.x86_64 glusterfs-fuse-3.7.0-2.el6rhs.x86_64 glusterfs-debuginfo-3.7.0-2.el6rhs.x86_64 [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# gluster --version glusterfs 3.7.0 built on May 15 2015 01:31:12 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. [root@rhsqa14-vm1 ~]# How reproducible: easily Steps to Reproduce: 1. create an EC volume as cold tier and attach distribute hot tier. uss enabled 2. fuse mount it and untar linux kernel, create snapshot 3. Activate snapshot while IO is going on. 4. on the mount point try accessing .snaps dir, Additional info: [root@rhsqa14-vm1 ~]# gluster v info jim Volume Name: jim Type: Tier Volume ID: 633cc77d-68b7-48e7-8081-509ac29fdff9 Status: Started Number of Bricks: 10 Transport-type: tcp Hot Tier : Hot Tier Type : Distribute Number of Bricks: 4 Brick1: 10.70.46.236:/rhs/brick5/p Brick2: 10.70.46.233:/rhs/brick5/p Brick3: 10.70.46.236:/rhs/brick4/p Brick4: 10.70.46.233:/rhs/brick4/p Cold Bricks: Cold Tier Type : Disperse Number of Bricks: 1 x (4 + 2) = 6 Brick5: 10.70.46.233:/rhs/brick1/p Brick6: 10.70.46.236:/rhs/brick1/p Brick7: 10.70.46.233:/rhs/brick2/p Brick8: 10.70.46.236:/rhs/brick2/p Brick9: 10.70.46.233:/rhs/brick3/p Brick10: 10.70.46.236:/rhs/brick3/p Options Reconfigured: features.barrier: disable features.uss: enable features.inode-quota: on features.quota: on cluster.min-free-disk: 10 performance.readdir-ahead: on [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm5 ~]# cd /mnt [root@rhsqa14-vm5 mnt]# ls -la total 80388 drwxr-xr-x. 5 root root 243 May 25 03:30 . dr-xr-xr-x. 31 root root 4096 May 25 02:24 .. drwx------. 4 root root 302 May 25 03:33 linux-4.0 -rw-r--r--. 1 root root 82313052 May 25 03:30 linux-4.0.tar.xz drwxr-xr-x. 3 root root 96 May 25 03:24 .trashcan [root@rhsqa14-vm5 mnt]# cd .snaps -bash: cd: .snaps: Transport endpoint is not connected [root@rhsqa14-vm5 mnt]#
Tried to recreate the bug on EC volumes with all the features enabled except tiering. It is working fine and don't see any kind of error. No "I/O Error". No "Transport endpoint is not connected". [root@aspandey gfsa]# ll total 16 drwx------. 7 root root 4096 Jun 4 14:45 linux-2.6.39 [root@aspandey gfsa]# pwd /mnt/gfsa [root@aspandey gfsa]# cd .snaps [root@aspandey .snaps]# ll total 2 drwxr-xr-x. 5 root root 58 Jun 4 14:24 test_1_GMT-2015.06.04-09.03.26 drwxr-xr-x. 5 root root 58 Jun 4 14:24 test_2_GMT-2015.06.04-09.13.20 drwxr-xr-x. 5 root root 58 Jun 4 14:24 test_GMT-2015.06.04-08.57.31 [root@aspandey .snaps]# pwd /mnt/gfsa/.snaps [root@aspandey .snaps]# cd test_GMT-2015.06.04-08.57.31/ [root@aspandey test_GMT-2015.06.04-08.57.31]# ll total 4 drwx------. 4 root root 4096 Jun 4 14:26 linux-2.6.39 [root@aspandey test_GMT-2015.06.04-08.57.31]# pwd /mnt/gfsa/.snaps/test_GMT-2015.06.04-08.57.31 [root@aspandey test_GMT-2015.06.04-08.57.31]# [root@rhs3 glusterfs]# gluster --version glusterfs 3.7.1 built on Jun 4 2015 13:17:30 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License [root@rhs3 glusterfs]# gluster v info gv1 Volume Name: gv1 Type: Disperse Volume ID: f139b6c8-fe00-44a6-a9f9-317b1e365afd Status: Started Number of Bricks: 1 x (4 + 2) = 6 Transport-type: tcp Bricks: Brick1: 10.70.42.64:/brick/a1 Brick2: 10.70.42.64:/brick/a2 Brick3: 10.70.42.64:/brick/a3 Brick4: 10.70.43.118:/brick/a4 Brick5: 10.70.43.118:/brick/a5 Brick6: 10.70.43.118:/brick/a6 Options Reconfigured: features.barrier: disable features.uss: on features.inode-quota: on features.quota: on performance.readdir-ahead: on cluster.disperse-self-heal-daemon: enable
I tried to reproduce the issue as mentioned, but I could able to access .snaps with out any error and no I/O errors, but tier daemon crashed during the course. But I think it is nothing to do with snapshot, the problem is being investigated.
Karthik, can you try to reproduce this? If you cannot, I believe it should be closed.