Bug 762073 (GLUSTER-341)

Summary: mountpoint first access failure in single address space mode client/server
Product: [Community] GlusterFS Reporter: Anand Avati <aavati>
Component: coreAssignee: Anand Avati <aavati>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: low Docs Contact:
Priority: low    
Version: mainlineCC: chrisw, gluster-bugs
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: RTP Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Anand Avati 2009-10-28 06:44:56 UTC 
volfile -

---
volume posix0
  type storage/posix
  option directory /home/avati/work/install/export/posix0
end-volume

volume locks0
  type features/locks
  subvolumes posix0
end-volume

volume iot0
  type performance/io-threads
  subvolumes locks0
end-volume

volume posix1
  type storage/posix
  option directory /home/avati/work/install/export/posix1
end-volume

volume locks1
  type features/locks
  subvolumes posix1
end-volume

volume iot1
  type performance/io-threads
  subvolumes locks1
end-volume

volume posix2
  type storage/posix
  option directory /home/avati/work/install/export/posix2
end-volume

volume locks2
  type features/locks
  subvolumes posix2
end-volume

volume iot2
  type performance/io-threads
  subvolumes locks2
end-volume

volume posix3
  type storage/posix
  option directory /home/avati/work/install/export/posix3
end-volume

volume locks3
  type features/locks
  subvolumes posix3
end-volume

volume iot3
  type performance/io-threads
  subvolumes locks3
end-volume

volume server0
  type protocol/server
  option transport-type tcp
  option listen-port 11234
  option auth.ip.iot0.allow *
  option auth.ip.iot1.allow *
  option auth.ip.iot2.allow *
  option auth.ip.iot3.allow *
  subvolumes iot0 iot1 iot2 iot3
end-volume

volume client0
  type protocol/client
  option transport-type tcp
  option remote-host localhost
  option remote-port 11234
  option remote-subvolume iot0
end-volume

volume client1
  type protocol/client
  option transport-type tcp
  option remote-host localhost
  option remote-port 11234
  option remote-subvolume iot1
end-volume

volume client2
  type protocol/client
  option transport-type tcp
  option remote-host localhost
  option remote-port 11234
  option remote-subvolume iot2
end-volume

volume client3
  type protocol/client
  option transport-type tcp
  option remote-host localhost
  option remote-port 11234
  option remote-subvolume iot3
end-volume

volume afr0
  type cluster/replicate
  subvolumes client0 client1
end-volume

volume afr1
  type cluster/replicate
  subvolumes client2 client3
end-volume

volume dht0
  type cluster/distribute
  subvolumes afr0 afr1
end-volume

volume wb0
  type performance/write-behind
  option flush-behind on
  subvolumes dht0
end-volume
Comment 1 Anand Avati 2009-10-28 06:46:23 UTC 
================================================================================
Version      : glusterfs 2.1.0git built on Oct 28 2009 01:11:43
git: git://git.sv.gnu.org/gluster.git
Starting Time: 2009-10-28 02:40:27
Command line : ./master/sbin/glusterfs -f ./glusterfs.vol -l ./glusterfs.log /home/avati/work/install/mount/ 
PID          : 18581
System name  : Linux
Nodename     : dev
Kernel Release : 2.6.26-1-amd64
Hardware Identifier: x86_64

Given volfile:
+------------------------------------------------------------------------------+
  1: volume posix0
  2:   type storage/posix
  3:   option directory /home/avati/work/install/export/posix0
  4: end-volume
  5: 
  6: volume locks0
  7:   type features/locks
  8:   subvolumes posix0
  9: end-volume
 10: 
 11: volume iot0
 12:   type performance/io-threads
 13:   subvolumes locks0
 14: end-volume
 15: 
 16: volume posix1
 17:   type storage/posix
 18:   option directory /home/avati/work/install/export/posix1
 19: end-volume
 20: 
 21: volume locks1
 22:   type features/locks
 23:   subvolumes posix1
 24: end-volume
 25: 
 26: volume iot1
 27:   type performance/io-threads
 28:   subvolumes locks1
 29: end-volume
 30: 
 31: volume posix2
 32:   type storage/posix
 33:   option directory /home/avati/work/install/export/posix2
 34: end-volume
 35: 
 36: volume locks2
 37:   type features/locks
 38:   subvolumes posix2
 39: end-volume
 40: 
 41: volume iot2
 42:   type performance/io-threads
 43:   subvolumes locks2
 44: end-volume
 45: 
 46: volume posix3
 47:   type storage/posix
 48:   option directory /home/avati/work/install/export/posix3
 49: end-volume
 50: 
 51: volume locks3
 52:   type features/locks
 53:   subvolumes posix3
 54: end-volume
 55: 
 56: volume iot3
 57:   type performance/io-threads
 58:   subvolumes locks3
 59: end-volume
 60: 
 61: volume server0
 62:   type protocol/server
 63:   option transport-type tcp
 64:   option listen-port 11234
 65:   option auth.ip.iot0.allow *
 66:   option auth.ip.iot1.allow *
 67:   option auth.ip.iot2.allow *
 68:   option auth.ip.iot3.allow *
 69:   subvolumes iot0 iot1 iot2 iot3
 70: end-volume
 71: 
 72: volume client0
 73:   type protocol/client
 74:   option transport-type tcp
 75:   option remote-host localhost
 76:   option remote-port 11234
 77:   option remote-subvolume iot0
 78: end-volume
 79: 
 80: volume client1
 81:   type protocol/client
 82:   option transport-type tcp
 83:   option remote-host localhost
 84:   option remote-port 11234
 85:   option remote-subvolume iot1
 86: end-volume
 87: 
 88: volume client2
 89:   type protocol/client
 90:   option transport-type tcp
 91:   option remote-host localhost
 92:   option remote-port 11234
 93:   option remote-subvolume iot2
 94: end-volume
 95: 
 96: volume client3
 97:   type protocol/client
 98:   option transport-type tcp
 99:   option remote-host localhost
100:   option remote-port 11234
101:   option remote-subvolume iot3
102: end-volume
103: 
104: volume afr0
105:   type cluster/replicate
106:   subvolumes client0 client1
107: end-volume
108: 
109: volume afr1
110:   type cluster/replicate
111:   subvolumes client2 client3
112: end-volume
113: 
114: volume dht0
115:   type cluster/distribute
116:   subvolumes afr0 afr1
117: end-volume
118: 
119: volume wb0
120:   type performance/write-behind
121:   option flush-behind on
122:   subvolumes dht0
123: end-volume

+------------------------------------------------------------------------------+
[2009-10-28 02:40:27] W [xlator.c:655:validate_xlator_volume_options] server0: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction
[2009-10-28 02:40:27] W [server-protocol.c:6145:get_auth_types] server: assuming 'auth.ip' to be 'auth.addr'
[2009-10-28 02:40:27] W [server-protocol.c:6145:get_auth_types] server: assuming 'auth.ip' to be 'auth.addr'
[2009-10-28 02:40:27] W [server-protocol.c:6145:get_auth_types] server: assuming 'auth.ip' to be 'auth.addr'
[2009-10-28 02:40:27] W [server-protocol.c:6145:get_auth_types] server: assuming 'auth.ip' to be 'auth.addr'
[2009-10-28 02:40:27] N [glusterfsd.c:1360:main] glusterfs: Successfully started
[2009-10-28 02:40:27] E [socket.c:744:socket_connect_finish] client0: connection to  failed (Connection refused)
[2009-10-28 02:40:27] N [fuse-bridge.c:2912:fuse_init] glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.12 kernel 7.9
[2009-10-28 02:40:27] E [socket.c:744:socket_connect_finish] client0: connection to  failed (Connection refused)
[2009-10-28 02:40:27] E [socket.c:744:socket_connect_finish] client1: connection to  failed (Connection refused)
[2009-10-28 02:40:27] E [socket.c:744:socket_connect_finish] client1: connection to  failed (Connection refused)
[2009-10-28 02:40:27] E [socket.c:744:socket_connect_finish] client2: connection to  failed (Connection refused)
[2009-10-28 02:40:27] E [socket.c:744:socket_connect_finish] client2: connection to  failed (Connection refused)
[2009-10-28 02:40:27] E [socket.c:744:socket_connect_finish] client3: connection to  failed (Connection refused)
[2009-10-28 02:40:27] E [socket.c:744:socket_connect_finish] client3: connection to  failed (Connection refused)
[2009-10-28 02:40:27] W [fuse-bridge.c:766:fuse_attr_cbk] glusterfs-fuse: 2: LOOKUP() / => -1 (No such file or directory)
[2009-10-28 02:40:27] N [server-protocol.c:5501:mop_setvolume] server0: accepted client from 127.0.0.1:1007
[2009-10-28 02:40:27] N [server-protocol.c:5501:mop_setvolume] server0: accepted client from 127.0.0.1:1006
[2009-10-28 02:40:27] W [client-protocol.c:5669:client_setvolume_cbk] client0: attaching to the local volume 'iot0'
[2009-10-28 02:40:27] N [client-protocol.c:5678:client_setvolume_cbk] client0: Connected to 127.0.0.1:11234, attached to remote volume 'iot0'.
[2009-10-28 02:40:27] N [afr.c:2307:notify] afr0: Subvolume 'client0' came back up; going online.
[2009-10-28 02:40:27] N [server-protocol.c:5501:mop_setvolume] server0: accepted client from 127.0.0.1:1005
[2009-10-28 02:40:27] W [client-protocol.c:5669:client_setvolume_cbk] client1: attaching to the local volume 'iot1'
[2009-10-28 02:40:27] N [client-protocol.c:5678:client_setvolume_cbk] client1: Connected to 127.0.0.1:11234, attached to remote volume 'iot1'.
[2009-10-28 02:40:27] N [server-protocol.c:5501:mop_setvolume] server0: accepted client from 127.0.0.1:1004
[2009-10-28 02:40:27] W [client-protocol.c:5669:client_setvolume_cbk] client2: attaching to the local volume 'iot2'
[2009-10-28 02:40:27] N [client-protocol.c:5678:client_setvolume_cbk] client2: Connected to 127.0.0.1:11234, attached to remote volume 'iot2'.
[2009-10-28 02:40:27] N [afr.c:2307:notify] afr1: Subvolume 'client2' came back up; going online.
[2009-10-28 02:40:27] W [client-protocol.c:5669:client_setvolume_cbk] client3: attaching to the local volume 'iot3'
[2009-10-28 02:40:27] N [client-protocol.c:5678:client_setvolume_cbk] client3: Connected to 127.0.0.1:11234, attached to remote volume 'iot3'.
[2009-10-28 02:40:38] N [server-protocol.c:5501:mop_setvolume] server0: accepted client from 127.0.0.1:1003
[2009-10-28 02:40:38] N [server-protocol.c:5501:mop_setvolume] server0: accepted client from 127.0.0.1:1002
[2009-10-28 02:40:38] W [client-protocol.c:5669:client_setvolume_cbk] client0: attaching to the local volume 'iot0'
[2009-10-28 02:40:38] N [client-protocol.c:5678:client_setvolume_cbk] client0: Connected to 127.0.0.1:11234, attached to remote volume 'iot0'.
[2009-10-28 02:40:38] N [server-protocol.c:5501:mop_setvolume] server0: accepted client from 127.0.0.1:1001
[2009-10-28 02:40:38] W [client-protocol.c:5669:client_setvolume_cbk] client1: attaching to the local volume 'iot1'
[2009-10-28 02:40:38] N [client-protocol.c:5678:client_setvolume_cbk] client1: Connected to 127.0.0.1:11234, attached to remote volume 'iot1'.
[2009-10-28 02:40:38] N [server-protocol.c:5501:mop_setvolume] server0: accepted client from 127.0.0.1:1000
[2009-10-28 02:40:38] W [client-protocol.c:5669:client_setvolume_cbk] client2: attaching to the local volume 'iot2'
[2009-10-28 02:40:38] N [client-protocol.c:5678:client_setvolume_cbk] client2: Connected to 127.0.0.1:11234, attached to remote volume 'iot2'.
[2009-10-28 02:40:38] W [client-protocol.c:5669:client_setvolume_cbk] client3: attaching to the local volume 'iot3'
[2009-10-28 02:40:38] N [client-protocol.c:5678:client_setvolume_cbk] client3: Connected to 127.0.0.1:11234, attached to remote volume 'iot3'.
Comment 2 Anand Avati 2009-10-28 06:47:39 UTC 
The problem is because of the way initialization of disconnected graphs happen. init() and notify() of one subgraph is completed before init() of the next subgraph is called.
Comment 3 Anand Avati 2009-10-28 06:56:11 UTC 
PATCH: http://patches.gluster.com/patch/2033 in master (xlator: initialize all xlators in a loop instead of top down initialization)
Comment 4 Anand Avati 2009-10-28 09:42:11 UTC 
accessing the mountpoint right after starting the single address space client/server process results in ENOENT.  Accessing the mountpoint the second time or after a small delay works fine. Session log -

root@dev:~/work/install# ./master/sbin/glusterfs -f ./glusterfs.vol -l ./glusterfs.log /home/avati/work/install/mount/; stat mount
stat: cannot stat `mount': No such file or directory
root@dev:~/work/install# stat mount
  File: `mount'
  Size: 8192      	Blocks: 32         IO Block: 131072 directory
Device: 17h/23d	Inode: 2           Links: 2
Access: (0775/drwxrwxr-x)  Uid: ( 1005/   avati)   Gid: ( 1005/   avati)
Access: 2009-10-28 01:13:22.000000000 -0700
Modify: 2009-10-28 01:46:23.000000000 -0700
Change: 2009-10-28 02:40:27.000000000 -0700