Bug 1221942

Summary: Transport endpoint is not connected error when tried to create file/dir after attaching tier to distribute volume
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Triveni Rao <trao>
Component: coreAssignee: Dan Lambright <dlambrig>
Status: CLOSED DUPLICATE QA Contact: Sudhir D <sdharane>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: rhgs-3.1CC: dlambrig, rhs-bugs, sashinde, storage-qa-internal, vagarwal
Target Milestone: ---Keywords: TestBlocker
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: TIERING
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1224141 (view as bug list) Environment:
Last Closed: 2015-10-30 18:02:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1224141    

Description Triveni Rao 2015-05-15 10:06:10 UTC 
Description of problem:

1. Transport endpoint is not connected error when tried to create file/dir after attaching tier to distribute volume.
2. I created pure distribute volume, attached pure distribute tier. 
3. fuse mounted it, tried to create files/dirs and got  Transport endpoint is not connected error message even though mount point is accessible.


Version-Release number of selected component (if applicable):
root@rhsqa14-vm1 ~]# rpm -qa | grep gluster
glusterfs-3.7.0-2.el6rhs.x86_64
glusterfs-cli-3.7.0-2.el6rhs.x86_64
glusterfs-libs-3.7.0-2.el6rhs.x86_64
glusterfs-client-xlators-3.7.0-2.el6rhs.x86_64
glusterfs-api-3.7.0-2.el6rhs.x86_64
glusterfs-server-3.7.0-2.el6rhs.x86_64
glusterfs-fuse-3.7.0-2.el6rhs.x86_64
[root@rhsqa14-vm1 ~]# 
[root@rhsqa14-vm1 ~]# glusterfs --version
glusterfs 3.7.0 built on May 15 2015 01:31:10
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.
[root@rhsqa14-vm1 ~]# 


How reproducible:

easily


Additional info:


[root@rhsqa14-vm1 ~]# gluster v attach-tier test 10.70.46.233:/rhs/brick3/pk 10.70.46.236:/rhs/brick3/pk                                              
Attach tier is recommended only for testing purposes in this release. Do you want to continue? (y/n) y
volume attach-tier: success
volume rebalance: test: success: Rebalance on test has been started successfully. Use rebalance status command to check status of the rebalance proces
s.
ID: 180739c3-e5a9-48e6-99aa-3469d69d18a7

You have new mail in /var/spool/mail/root
[root@rhsqa14-vm1 ~]# 
[root@rhsqa14-vm1 ~]# 
[root@rhsqa14-vm1 ~]# gluster v info test

Volume Name: test
Type: Tier
Volume ID: 102100c7-6a81-4ffc-9736-b9aa773b5044
Status: Started
Number of Bricks: 4   
Transport-type: tcp   
Hot Tier :
Hot Tier Type : Distribute
Number of Bricks: 2   
Brick1: 10.70.46.236:/rhs/brick3/pk
Brick2: 10.70.46.233:/rhs/brick3/pk
Cold Bricks:
Cold Tier Type : Distribute
Number of Bricks: 2   
Brick3: 10.70.46.233:/rhs/brick1/t1
Brick4: 10.70.46.236:/rhs/brick1/t1
Options Reconfigured: 
features.uss: enable  
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
cluster.min-free-disk: 10
performance.readdir-ahead: on
[root@rhsqa14-vm1 ~]# 


On mount point:

[root@rhsqa14-vm5 disk1]# cd /root
[root@rhsqa14-vm5 ~]# mount -t glusterfs 10.70.46.233:test /mnt1
[root@rhsqa14-vm5 ~]# cd /mnt1
[root@rhsqa14-vm5 mnt1]# ls -la
total 4
drwxr-xr-x.  4 root root  156 May 15  2015 .
dr-xr-xr-x. 30 root root 4096 May 15 04:16 ..
drwxr-xr-x.  3 root root   96 May 15  2015 .trashcan
[root@rhsqa14-vm5 mnt1]#
[root@rhsqa14-vm5 mnt1]# cd /root
[root@rhsqa14-vm5 ~]# ls
anaconda-ks.cfg  install.log  install.log.syslog  linux-4.0  linux-4.0.tar.xz  rpms
[root@rhsqa14-vm5 ~]# cp linux-4.0 /mnt1
cp: omitting directory `linux-4.0'
[root@rhsqa14-vm5 ~]# cp -r linux-4.0 /mnt1
cp: cannot create directory `/mnt1/linux-4.0': Transport endpoint is not connected
[root@rhsqa14-vm5 ~]# cd /mnt1
[root@rhsqa14-vm5 mnt1]# ls -la
total 4
drwxr-xr-x.  4 root root  156 May 15  2015 .
dr-xr-xr-x. 30 root root 4096 May 15 04:16 ..
drwxr-xr-x.  3 root root   96 May 15  2015 .trashcan
[root@rhsqa14-vm5 mnt1]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_rhsqa14vm5-lv_root
                       18G  3.2G   14G  20% /
tmpfs                 3.8G     0  3.8G   0% /dev/shm
/dev/vda1             477M   33M  419M   8% /boot
10.70.46.233:vol1      20G   84M   20G   1% /disk1
10.70.46.233:test      20G     0   20G   0% /mnt1
[root@rhsqa14-vm5 mnt1]# 
[root@rhsqa14-vm5 mnt1]# touch f1
touch: cannot touch `f1': Transport endpoint is not connected
[root@rhsqa14-vm5 mnt1]# mkdir p
mkdir: cannot create directory `p': Transport endpoint is not connected
[root@rhsqa14-vm5 mnt1]# 

NOTE: this issue was not seen on dist-rep volume
Comment 1 Triveni Rao 2015-05-19 10:14:03 UTC 
I was able to reproduce with dist-rep and distribute tier combination of volume.


[root@rhsqa14-vm1 ~]# gluster v info Tim

Volume Name: Tim
Type: Tier
Volume ID: 68286fbf-d7ac-4f05-9578-2919cbbfba68
Status: Started
Number of Bricks: 8
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distribute
Number of Bricks: 2
Brick1: 10.70.46.236:/rhs/brick3/j0
Brick2: 10.70.46.233:/rhs/brick3/j0
Cold Bricks:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 3 x 2 = 6
Brick3: 10.70.46.233:/rhs/brick3/b1
Brick4: 10.70.46.236:/rhs/brick3/b1
Brick5: 10.70.46.233:/rhs/brick4/b1
Brick6: 10.70.46.236:/rhs/brick4/b1
Brick7: 10.70.46.233:/rhs/brick5/b1
Brick8: 10.70.46.236:/rhs/brick5/b1
Options Reconfigured:
nfs.disable: on
performance.readdir-ahead: on
cluster.min-free-disk: 10
features.quota: on
features.inode-quota: on
features.quota-deem-statfs: on
features.uss: enable
[root@rhsqa14-vm1 ~]# 


[root@rhsqa14-vm1 ~]# gluster v status Tim
Status of volume: Tim 
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick 10.70.46.236:/rhs/brick3/j0           49189     0          Y       5074
Brick 10.70.46.233:/rhs/brick3/j0           49196     0          Y       19060
Brick 10.70.46.233:/rhs/brick3/b1           49192     0          Y       18619
Brick 10.70.46.236:/rhs/brick3/b1           49185     0          Y       4715
Brick 10.70.46.233:/rhs/brick4/b1           49193     0          Y       18636
Brick 10.70.46.236:/rhs/brick4/b1           49186     0          Y       4735
Brick 10.70.46.233:/rhs/brick5/b1           49194     0          Y       18653
Brick 10.70.46.236:/rhs/brick5/b1           49187     0          Y       4756
Snapshot Daemon on localhost                49195     0          Y       18813
Quota Daemon on localhost                   N/A       N/A        Y       19078
Snapshot Daemon on 10.70.46.240             49154     0          Y       9098
Quota Daemon on 10.70.46.240                N/A       N/A        N       N/A
Snapshot Daemon on 10.70.46.243             49157     0          Y       8171
Quota Daemon on 10.70.46.243                N/A       N/A        Y       8399
Snapshot Daemon on 10.70.46.236             49188     0          Y       4953
Quota Daemon on 10.70.46.236                N/A       N/A        Y       5097

Task Status of Volume Tim
------------------------------------------------------------------------------
Task                 : Rebalance
ID                   : c4daee48-1103-4a9c-b13a-d68c8acf0b9c
Status               : in progress

[root@rhsqa14-vm1 ~]# 


On mount point:

[root@rhsqa14-vm5 ~]# mount -t glusterfs 10.70.46.233:/Tim /mnt
[root@rhsqa14-vm5 ~]# ls -la /mnt
total 6
drwxr-xr-x.  4 root root  195 May 19 05:55 .
dr-xr-xr-x. 30 root root 4096 May 15 04:16 ..
drwxr-xr-x.  3 root root  120 May 19 05:55 .trashcan
[root@rhsqa14-vm5 ~]# cd /mnt
[root@rhsqa14-vm5 mnt]# touch trivnei
touch: setting times of `trivnei': No such file or directory
[root@rhsqa14-vm5 mnt]# mkdir tri
mkdir: cannot create directory `tri': Invalid argument
[root@rhsqa14-vm5 mnt]#
[root@rhsqa14-vm5 mnt]#
Comment 2 Triveni Rao 2015-05-19 11:04:36 UTC 
[root@rhsqa14-vm1 ~]# gluster v create jack 10.70.46.233:/rhs/brick3/x0 10.70.46.236:/rhs/brick3/x0 10.70.46.233:/rhs/brick4/x0 10.70.46.236:/rhs/brick4/x0
volume create: jack: success: please start the volume to access data
[root@rhsqa14-vm1 ~]# gluster v start jack
volume start: jack: success
[root@rhsqa14-vm1 ~]# 
You have new mail in /var/spool/mail/root
[root@rhsqa14-vm1 ~]# gluster v info jack

Volume Name: jack
Type: Distribute
Volume ID: babe202e-a03c-4ed3-abb0-e5edd528bb84
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: 10.70.46.233:/rhs/brick3/x0
Brick2: 10.70.46.236:/rhs/brick3/x0
Brick3: 10.70.46.233:/rhs/brick4/x0
Brick4: 10.70.46.236:/rhs/brick4/x0
Options Reconfigured:
performance.readdir-ahead: on
[root@rhsqa14-vm1 ~]#
[root@rhsqa14-vm1 ~]# ./options.sh jack
volume set: success
volume quota : success
volume set: success
volume quota : success
volume set: success
[root@rhsqa14-vm1 ~]#
[root@rhsqa14-vm1 ~]# gluster v attach-tier jack replica 2 10.70.46.233:/rhs/brick5/l0 10.70.46.236:/rhs/brick5/l0 10.70.46.233:/rhs/brick2/l0 10.70.46.236:/rhs/brick2/l0
Attach tier is recommended only for testing purposes in this release. Do you want to continue? (y/n) y
volume attach-tier: success
volume rebalance: jack: success: Rebalance on jack has been started successfully. Use rebalance status command to check status of the rebalance process.
ID: 5b4396fc-8d7b-4a91-a495-99f8b7afc5c1

You have new mail in /var/spool/mail/root
[root@rhsqa14-vm1 ~]# gluster v info jack

Volume Name: jack
Type: Tier
Volume ID: babe202e-a03c-4ed3-abb0-e5edd528bb84
Status: Started
Number of Bricks: 8
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick1: 10.70.46.236:/rhs/brick2/l0
Brick2: 10.70.46.233:/rhs/brick2/l0
Brick3: 10.70.46.236:/rhs/brick5/l0
Brick4: 10.70.46.233:/rhs/brick5/l0
Cold Bricks:
Cold Tier Type : Distribute
Number of Bricks: 4
Brick5: 10.70.46.233:/rhs/brick3/x0
Brick6: 10.70.46.236:/rhs/brick3/x0
Brick7: 10.70.46.233:/rhs/brick4/x0
Brick8: 10.70.46.236:/rhs/brick4/x0
Options Reconfigured: 
features.uss: enable  
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
cluster.min-free-disk: 10
performance.readdir-ahead: on
[root@rhsqa14-vm1 ~]# 
[root@rhsqa14-vm1 ~]



[root@rhsqa14-vm5 mnt]# mount -t glusterfs 10.70.46.233:/jack /mnt1
[root@rhsqa14-vm5 mnt]# cd /mnt1
[root@rhsqa14-vm5 mnt1]#
[root@rhsqa14-vm5 mnt1]#
[root@rhsqa14-vm5 mnt1]#
[root@rhsqa14-vm5 mnt1]# touch triveni
touch: cannot touch `triveni': Transport endpoint is not connected
[root@rhsqa14-vm5 mnt1]# ls -la
total 5
drwxr-xr-x.  4 root root  234 May 19  2015 .
dr-xr-xr-x. 30 root root 4096 May 15 04:16 ..
drwxr-xr-x.  3 root root  144 May 19  2015 .trashcan
[root@rhsqa14-vm5 mnt1]#
[root@rhsqa14-vm5 mnt1]# cp /root/linux-4.0.tar.xz .
cp: cannot create regular file `./linux-4.0.tar.xz': Transport endpoint is not connected
[root@rhsqa14-vm5 mnt1]#
Comment 5 Dan Lambright 2015-05-19 19:29:29 UTC 
I have recreated the problem on mainline with the configuration below. I will update the bug once its fixed.

Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick1: rhs-cli-02:/home/t8
Brick2: rhs-cli-01:/home/t7
Brick3: rhs-cli-02:/home/t6
Brick4: rhs-cli-01:/home/t5
Cold Bricks:
Cold Tier Type : Distribute
Number of Bricks: 4
Brick5: rhs-cli-01:/home/t1
Brick6: rhs-cli-02:/home/t2
Brick7: rhs-cli-01:/home/t3
Brick8: rhs-cli-02:/home/t4
Options Reconfigured:
Comment 6 Dan Lambright 2015-05-19 19:48:17 UTC 
I spoke too soon. I cannot recreate this. Lets discuss tomorrow in scrum.
Comment 7 Vivek Agarwal 2015-10-30 18:02:26 UTC 

*** This bug has been marked as a duplicate of bug 1224141 ***