Description of problem: 1. Transport endpoint is not connected error when tried to create file/dir after attaching tier to distribute volume. 2. I created pure distribute volume, attached pure distribute tier. 3. fuse mounted it, tried to create files/dirs and got Transport endpoint is not connected error message even though mount point is accessible. Version-Release number of selected component (if applicable): root@rhsqa14-vm1 ~]# rpm -qa | grep gluster glusterfs-3.7.0-2.el6rhs.x86_64 glusterfs-cli-3.7.0-2.el6rhs.x86_64 glusterfs-libs-3.7.0-2.el6rhs.x86_64 glusterfs-client-xlators-3.7.0-2.el6rhs.x86_64 glusterfs-api-3.7.0-2.el6rhs.x86_64 glusterfs-server-3.7.0-2.el6rhs.x86_64 glusterfs-fuse-3.7.0-2.el6rhs.x86_64 [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# glusterfs --version glusterfs 3.7.0 built on May 15 2015 01:31:10 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/> GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation. [root@rhsqa14-vm1 ~]# How reproducible: easily Additional info: [root@rhsqa14-vm1 ~]# gluster v attach-tier test 10.70.46.233:/rhs/brick3/pk 10.70.46.236:/rhs/brick3/pk Attach tier is recommended only for testing purposes in this release. Do you want to continue? (y/n) y volume attach-tier: success volume rebalance: test: success: Rebalance on test has been started successfully. Use rebalance status command to check status of the rebalance proces s. ID: 180739c3-e5a9-48e6-99aa-3469d69d18a7 You have new mail in /var/spool/mail/root [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# gluster v info test Volume Name: test Type: Tier Volume ID: 102100c7-6a81-4ffc-9736-b9aa773b5044 Status: Started Number of Bricks: 4 Transport-type: tcp Hot Tier : Hot Tier Type : Distribute Number of Bricks: 2 Brick1: 10.70.46.236:/rhs/brick3/pk Brick2: 10.70.46.233:/rhs/brick3/pk Cold Bricks: Cold Tier Type : Distribute Number of Bricks: 2 Brick3: 10.70.46.233:/rhs/brick1/t1 Brick4: 10.70.46.236:/rhs/brick1/t1 Options Reconfigured: features.uss: enable features.quota-deem-statfs: on features.inode-quota: on features.quota: on cluster.min-free-disk: 10 performance.readdir-ahead: on [root@rhsqa14-vm1 ~]# On mount point: [root@rhsqa14-vm5 disk1]# cd /root [root@rhsqa14-vm5 ~]# mount -t glusterfs 10.70.46.233:test /mnt1 [root@rhsqa14-vm5 ~]# cd /mnt1 [root@rhsqa14-vm5 mnt1]# ls -la total 4 drwxr-xr-x. 4 root root 156 May 15 2015 . dr-xr-xr-x. 30 root root 4096 May 15 04:16 .. drwxr-xr-x. 3 root root 96 May 15 2015 .trashcan [root@rhsqa14-vm5 mnt1]# [root@rhsqa14-vm5 mnt1]# cd /root [root@rhsqa14-vm5 ~]# ls anaconda-ks.cfg install.log install.log.syslog linux-4.0 linux-4.0.tar.xz rpms [root@rhsqa14-vm5 ~]# cp linux-4.0 /mnt1 cp: omitting directory `linux-4.0' [root@rhsqa14-vm5 ~]# cp -r linux-4.0 /mnt1 cp: cannot create directory `/mnt1/linux-4.0': Transport endpoint is not connected [root@rhsqa14-vm5 ~]# cd /mnt1 [root@rhsqa14-vm5 mnt1]# ls -la total 4 drwxr-xr-x. 4 root root 156 May 15 2015 . dr-xr-xr-x. 30 root root 4096 May 15 04:16 .. drwxr-xr-x. 3 root root 96 May 15 2015 .trashcan [root@rhsqa14-vm5 mnt1]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_rhsqa14vm5-lv_root 18G 3.2G 14G 20% / tmpfs 3.8G 0 3.8G 0% /dev/shm /dev/vda1 477M 33M 419M 8% /boot 10.70.46.233:vol1 20G 84M 20G 1% /disk1 10.70.46.233:test 20G 0 20G 0% /mnt1 [root@rhsqa14-vm5 mnt1]# [root@rhsqa14-vm5 mnt1]# touch f1 touch: cannot touch `f1': Transport endpoint is not connected [root@rhsqa14-vm5 mnt1]# mkdir p mkdir: cannot create directory `p': Transport endpoint is not connected [root@rhsqa14-vm5 mnt1]# NOTE: this issue was not seen on dist-rep volume
I was able to reproduce with dist-rep and distribute tier combination of volume. [root@rhsqa14-vm1 ~]# gluster v info Tim Volume Name: Tim Type: Tier Volume ID: 68286fbf-d7ac-4f05-9578-2919cbbfba68 Status: Started Number of Bricks: 8 Transport-type: tcp Hot Tier : Hot Tier Type : Distribute Number of Bricks: 2 Brick1: 10.70.46.236:/rhs/brick3/j0 Brick2: 10.70.46.233:/rhs/brick3/j0 Cold Bricks: Cold Tier Type : Distributed-Replicate Number of Bricks: 3 x 2 = 6 Brick3: 10.70.46.233:/rhs/brick3/b1 Brick4: 10.70.46.236:/rhs/brick3/b1 Brick5: 10.70.46.233:/rhs/brick4/b1 Brick6: 10.70.46.236:/rhs/brick4/b1 Brick7: 10.70.46.233:/rhs/brick5/b1 Brick8: 10.70.46.236:/rhs/brick5/b1 Options Reconfigured: nfs.disable: on performance.readdir-ahead: on cluster.min-free-disk: 10 features.quota: on features.inode-quota: on features.quota-deem-statfs: on features.uss: enable [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# gluster v status Tim Status of volume: Tim Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Hot Bricks: Brick 10.70.46.236:/rhs/brick3/j0 49189 0 Y 5074 Brick 10.70.46.233:/rhs/brick3/j0 49196 0 Y 19060 Brick 10.70.46.233:/rhs/brick3/b1 49192 0 Y 18619 Brick 10.70.46.236:/rhs/brick3/b1 49185 0 Y 4715 Brick 10.70.46.233:/rhs/brick4/b1 49193 0 Y 18636 Brick 10.70.46.236:/rhs/brick4/b1 49186 0 Y 4735 Brick 10.70.46.233:/rhs/brick5/b1 49194 0 Y 18653 Brick 10.70.46.236:/rhs/brick5/b1 49187 0 Y 4756 Snapshot Daemon on localhost 49195 0 Y 18813 Quota Daemon on localhost N/A N/A Y 19078 Snapshot Daemon on 10.70.46.240 49154 0 Y 9098 Quota Daemon on 10.70.46.240 N/A N/A N N/A Snapshot Daemon on 10.70.46.243 49157 0 Y 8171 Quota Daemon on 10.70.46.243 N/A N/A Y 8399 Snapshot Daemon on 10.70.46.236 49188 0 Y 4953 Quota Daemon on 10.70.46.236 N/A N/A Y 5097 Task Status of Volume Tim ------------------------------------------------------------------------------ Task : Rebalance ID : c4daee48-1103-4a9c-b13a-d68c8acf0b9c Status : in progress [root@rhsqa14-vm1 ~]# On mount point: [root@rhsqa14-vm5 ~]# mount -t glusterfs 10.70.46.233:/Tim /mnt [root@rhsqa14-vm5 ~]# ls -la /mnt total 6 drwxr-xr-x. 4 root root 195 May 19 05:55 . dr-xr-xr-x. 30 root root 4096 May 15 04:16 .. drwxr-xr-x. 3 root root 120 May 19 05:55 .trashcan [root@rhsqa14-vm5 ~]# cd /mnt [root@rhsqa14-vm5 mnt]# touch trivnei touch: setting times of `trivnei': No such file or directory [root@rhsqa14-vm5 mnt]# mkdir tri mkdir: cannot create directory `tri': Invalid argument [root@rhsqa14-vm5 mnt]# [root@rhsqa14-vm5 mnt]#
[root@rhsqa14-vm1 ~]# gluster v create jack 10.70.46.233:/rhs/brick3/x0 10.70.46.236:/rhs/brick3/x0 10.70.46.233:/rhs/brick4/x0 10.70.46.236:/rhs/brick4/x0 volume create: jack: success: please start the volume to access data [root@rhsqa14-vm1 ~]# gluster v start jack volume start: jack: success [root@rhsqa14-vm1 ~]# You have new mail in /var/spool/mail/root [root@rhsqa14-vm1 ~]# gluster v info jack Volume Name: jack Type: Distribute Volume ID: babe202e-a03c-4ed3-abb0-e5edd528bb84 Status: Started Number of Bricks: 4 Transport-type: tcp Bricks: Brick1: 10.70.46.233:/rhs/brick3/x0 Brick2: 10.70.46.236:/rhs/brick3/x0 Brick3: 10.70.46.233:/rhs/brick4/x0 Brick4: 10.70.46.236:/rhs/brick4/x0 Options Reconfigured: performance.readdir-ahead: on [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# ./options.sh jack volume set: success volume quota : success volume set: success volume quota : success volume set: success [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# gluster v attach-tier jack replica 2 10.70.46.233:/rhs/brick5/l0 10.70.46.236:/rhs/brick5/l0 10.70.46.233:/rhs/brick2/l0 10.70.46.236:/rhs/brick2/l0 Attach tier is recommended only for testing purposes in this release. Do you want to continue? (y/n) y volume attach-tier: success volume rebalance: jack: success: Rebalance on jack has been started successfully. Use rebalance status command to check status of the rebalance process. ID: 5b4396fc-8d7b-4a91-a495-99f8b7afc5c1 You have new mail in /var/spool/mail/root [root@rhsqa14-vm1 ~]# gluster v info jack Volume Name: jack Type: Tier Volume ID: babe202e-a03c-4ed3-abb0-e5edd528bb84 Status: Started Number of Bricks: 8 Transport-type: tcp Hot Tier : Hot Tier Type : Distributed-Replicate Number of Bricks: 2 x 2 = 4 Brick1: 10.70.46.236:/rhs/brick2/l0 Brick2: 10.70.46.233:/rhs/brick2/l0 Brick3: 10.70.46.236:/rhs/brick5/l0 Brick4: 10.70.46.233:/rhs/brick5/l0 Cold Bricks: Cold Tier Type : Distribute Number of Bricks: 4 Brick5: 10.70.46.233:/rhs/brick3/x0 Brick6: 10.70.46.236:/rhs/brick3/x0 Brick7: 10.70.46.233:/rhs/brick4/x0 Brick8: 10.70.46.236:/rhs/brick4/x0 Options Reconfigured: features.uss: enable features.quota-deem-statfs: on features.inode-quota: on features.quota: on cluster.min-free-disk: 10 performance.readdir-ahead: on [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~] [root@rhsqa14-vm5 mnt]# mount -t glusterfs 10.70.46.233:/jack /mnt1 [root@rhsqa14-vm5 mnt]# cd /mnt1 [root@rhsqa14-vm5 mnt1]# [root@rhsqa14-vm5 mnt1]# [root@rhsqa14-vm5 mnt1]# [root@rhsqa14-vm5 mnt1]# touch triveni touch: cannot touch `triveni': Transport endpoint is not connected [root@rhsqa14-vm5 mnt1]# ls -la total 5 drwxr-xr-x. 4 root root 234 May 19 2015 . dr-xr-xr-x. 30 root root 4096 May 15 04:16 .. drwxr-xr-x. 3 root root 144 May 19 2015 .trashcan [root@rhsqa14-vm5 mnt1]# [root@rhsqa14-vm5 mnt1]# cp /root/linux-4.0.tar.xz . cp: cannot create regular file `./linux-4.0.tar.xz': Transport endpoint is not connected [root@rhsqa14-vm5 mnt1]#
I have recreated the problem on mainline with the configuration below. I will update the bug once its fixed. Hot Tier : Hot Tier Type : Distributed-Replicate Number of Bricks: 2 x 2 = 4 Brick1: rhs-cli-02:/home/t8 Brick2: rhs-cli-01:/home/t7 Brick3: rhs-cli-02:/home/t6 Brick4: rhs-cli-01:/home/t5 Cold Bricks: Cold Tier Type : Distribute Number of Bricks: 4 Brick5: rhs-cli-01:/home/t1 Brick6: rhs-cli-02:/home/t2 Brick7: rhs-cli-01:/home/t3 Brick8: rhs-cli-02:/home/t4 Options Reconfigured:
I spoke too soon. I cannot recreate this. Lets discuss tomorrow in scrum.
*** This bug has been marked as a duplicate of bug 1224141 ***