Bug 762308 (GLUSTER-576)

Summary: crossing device (2056) + fuse LOOKUP error
Product: [Community] GlusterFS Reporter: Eatdirt <dirteat>
Component: unclassifiedAssignee: Vikas Gorur <vikas>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: low    
Version: 3.0.0CC: gluster-bugs, pavan
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Eatdirt 2010-01-27 10:35:28 UTC 
I have a nufa client/server cluster system on one side and a distribute server/client on the other.

On the nufa side, I get these errors in the logs, although it does not seen to affect the the filesystem, I can access all files from every nodes:

[2010-01-27 11:11:06] W [posix.c:246:posix_lstat_with_gen] posix: Access to /data//.. (on dev 2051) is crossing device (2056)
[2010-01-27 11:12:23] W [posix.c:246:posix_lstat_with_gen] posix: Access to /data//.. (on dev 2051) is crossing device (2056)
[2010-01-27 11:19:54] W [posix.c:246:posix_lstat_with_gen] posix: Access to /data//.. (on dev 2051) is crossing device (2056)


The configuration file for the nufa part is this one (for client-02, similar for the others)

Given volfile:
+------------------------------------------------------------------------------+
  1: volume posix
  2:  type storage/posix
  3:  option directory /data
  4: end-volume
  5: 
  6: #volume locks
  7: #  type features/locks
  8: #  subvolumes posix
  9: #end-volume
 10: 
 11: volume threads
 12:  type performance/io-threads
 13:  option thread-count 8
 14:  subvolumes posix
 15: end-volume
 16: 
 17: volume brick
 18:   type performance/read-ahead
 19:   option page-count 16
 20:   subvolumes threads
 21: end-volume
 22: 
 23: volume server
 24:  type protocol/server
 25:  option transport-type tcp/server
 26:  option auth.ip.brick.allow *
 27:  subvolumes brick
 28: end-volume
 29: 
 30: #volume client-02
 31: # type protocol/client
 32: # option transport-type tcp/client
 33: # option remote-host mercury
 34: # option remote-subvolume brick
 35: #end-volume
 36: 
 37: volume client-03
 38:  type protocol/client
 39:  option transport-type tcp/client
 40:  option remote-host venus
 41:  option remote-subvolume brick
 42: end-volume
 43: 
 44: volume client-04
 45:  type protocol/client
 46:  option transport-type tcp/client
 47:  option remote-host earth
 48:  option remote-subvolume brick
 49: end-volume
 50: 
 51: volume client-05
 52:  type protocol/client
 53:  option transport-type tcp/client
 54:  option remote-host mars
 55:  option remote-subvolume brick
 56: end-volume
 57: 
 58: volume client-06
 59:  type protocol/client
 60:  option transport-type tcp/client
 61:  option remote-host jupiter
 62:  option remote-subvolume brick
 63: end-volume
 64: 
 65: volume client-07
 66:  type protocol/client
 67:  option transport-type tcp/client
 68:  option remote-host saturn
 69:  option remote-subvolume brick
 70: end-volume
 71: 
 72: volume unify
 73:  type cluster/nufa
 74:  option lookup-unhashed yes
 75:  option local-volume-name brick
 76:  subvolumes brick client-03 client-04 client-05 client-06 client-07
 77: end-volume
 78: 
 79: volume writebehind
 80:  type performance/write-behind
 81:  option cache-size 1MB
 82:  subvolumes unify
 83: end-volume
 84: 
 85: volume cache
 86:  type performance/io-cache
 87:  option cache-size 128MB
 88:  subvolumes writebehind
 89: end-volume
 90: 
 91: 

+------------------------------------------------------------------------------+
[2010-01-27 11:10:48] W [server-protocol.c:6475:get_auth_types] server: assuming 'auth.ip' to be 'auth.addr'
[2010-01-27 11:10:48] N [fuse-bridge.c:2931:fuse_init] glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.10
[2010-01-27 11:10:48] N [glusterfsd.c:1361:main] glusterfs: Successfully started
[2010-01-27 11:10:48] E [socket.c:760:socket_connect_finish] client-07: connection to  failed (Connection refused)
[2010-01-27 11:10:48] E [socket.c:760:socket_connect_finish] client-07: connection to  failed (Connection refused)
[2010-01-27 11:10:48] E [socket.c:760:socket_connect_finish] client-06: connection to  failed (Connection refused)
[2010-01-27 11:10:48] E [socket.c:760:socket_connect_finish] client-06: connection to  failed (Connection refused)
[2010-01-27 11:10:48] E [socket.c:760:socket_connect_finish] client-05: connection to  failed (Connection refused)
[2010-01-27 11:10:48] E [socket.c:760:socket_connect_finish] client-05: connection to  failed (Connection refused)
[2010-01-27 11:10:48] E [socket.c:760:socket_connect_finish] client-04: connection to  failed (Connection refused)
[2010-01-27 11:10:48] E [socket.c:760:socket_connect_finish] client-04: connection to  failed (Connection refused)
[2010-01-27 11:10:48] E [socket.c:760:socket_connect_finish] client-03: connection to  failed (Connection refused)
[2010-01-27 11:10:48] E [socket.c:760:socket_connect_finish] client-03: connection to  failed (Connection refused)
[2010-01-27 11:10:49] N [server-protocol.c:5809:mop_setvolume] server: accepted client from 192.168.0.3:1023
[2010-01-27 11:10:49] N [server-protocol.c:5809:mop_setvolume] server: accepted client from 192.168.0.3:1022
[2010-01-27 11:10:50] N [server-protocol.c:5809:mop_setvolume] server: accepted client from 192.168.0.4:1023
[2010-01-27 11:10:50] N [server-protocol.c:5809:mop_setvolume] server: accepted client from 192.168.0.4:1022
[2010-01-27 11:10:50] N [server-protocol.c:5809:mop_setvolume] server: accepted client from 192.168.0.5:1023
[2010-01-27 11:10:50] N [server-protocol.c:5809:mop_setvolume] server: accepted client from 192.168.0.5:1022
[2010-01-27 11:10:51] N [server-protocol.c:5809:mop_setvolume] server: accepted client from 192.168.0.6:1023
[2010-01-27 11:10:51] N [server-protocol.c:5809:mop_setvolume] server: accepted client from 192.168.0.6:1022
[2010-01-27 11:10:52] N [server-protocol.c:5809:mop_setvolume] server: accepted client from 192.168.0.7:1023
[2010-01-27 11:10:52] N [server-protocol.c:5809:mop_setvolume] server: accepted client from 192.168.0.7:1022
[2010-01-27 11:10:56] N [server-protocol.c:5809:mop_setvolume] server: accepted client from 192.168.0.1:1023
[2010-01-27 11:10:56] N [server-protocol.c:5809:mop_setvolume] server: accepted client from 192.168.0.1:1022
[2010-01-27 11:10:59] N [client-protocol.c:6224:client_setvolume_cbk] client-03: Connected to 192.168.0.3:6996, attached to remote volume 'brick'.
[2010-01-27 11:10:59] N [client-protocol.c:6224:client_setvolume_cbk] client-03: Connected to 192.168.0.3:6996, attached to remote volume 'brick'.
[2010-01-27 11:10:59] N [client-protocol.c:6224:client_setvolume_cbk] client-04: Connected to 192.168.0.4:6996, attached to remote volume 'brick'.
[2010-01-27 11:10:59] N [client-protocol.c:6224:client_setvolume_cbk] client-05: Connected to 192.168.0.5:6996, attached to remote volume 'brick'.
[2010-01-27 11:10:59] N [client-protocol.c:6224:client_setvolume_cbk] client-04: Connected to 192.168.0.4:6996, attached to remote volume 'brick'.
[2010-01-27 11:10:59] N [client-protocol.c:6224:client_setvolume_cbk] client-06: Connected to 192.168.0.6:6996, attached to remote volume 'brick'.
[2010-01-27 11:10:59] N [client-protocol.c:6224:client_setvolume_cbk] client-05: Connected to 192.168.0.5:6996, attached to remote volume 'brick'.
[2010-01-27 11:10:59] N [client-protocol.c:6224:client_setvolume_cbk] client-06: Connected to 192.168.0.6:6996, attached to remote volume 'brick'.
[2010-01-27 11:10:59] N [client-protocol.c:6224:client_setvolume_cbk] client-07: Connected to 192.168.0.7:6996, attached to remote volume 'brick'.
[2010-01-27 11:10:59] N [client-protocol.c:6224:client_setvolume_cbk] client-07: Connected to 192.168.0.7:6996, attached to remote volume 'brick'.
[2010-01-27 11:11:06] W [posix.c:246:posix_lstat_with_gen] posix: Access to /data//.. (on dev 2051) is crossing device (2056)
[2010-01-27 11:12:23] W [posix.c:246:posix_lstat_with_gen] posix: Access to /data//.. (on dev 2051) is crossing device (2056)
[2010-01-27 11:19:54] W [posix.c:246:posix_lstat_with_gen] posix: Access to /data//.. (on dev 2051) is crossing device (2056)


--------------------------------


Now on the distribute side, sometimes, files are not found, especially when a bash script tries to rename the files. I am getting this error in the logs:

[2010-01-27 10:42:32] W [fuse-bridge.c:491:fuse_entry_cbk] glusterfs-fuse: LOOKUP(/chris/simusp
ace/runspace/mbx50m221_4) inode (ptr=0x2aaaac015a80, ino=14516760005, gen=5431049006933344394) 
found conflict (ptr=0x11becdf0, ino=14516760005, gen=5431049006933344394)



On the distribute side, the config file is:

Given volfile:
+------------------------------------------------------------------------------+
  1: volume client-02
  2:  type protocol/client
  3:  option transport-type tcp/client
  4:  option remote-host mercury
  5:  option remote-subvolume brick
  6: end-volume
  7: 
  8: volume client-03
  9:  type protocol/client
 10:  option transport-type tcp/client
 11:  option remote-host venus
 12:  option remote-subvolume brick
 13: end-volume
 14: 
 15: volume client-04
 16:  type protocol/client
 17:  option transport-type tcp/client
 18:  option remote-host earth
 19:  option remote-subvolume brick
 20: end-volume
 21: 
 22: volume client-05
 23:  type protocol/client
 24:  option transport-type tcp/client
 25:  option remote-host mars
 26:  option remote-subvolume brick
 27: end-volume
 28: 
 29: volume client-06
 30:  type protocol/client
 31:  option transport-type tcp/client
 32:  option remote-host jupiter
 33:  option remote-subvolume brick
 34: end-volume
 35: 
 36: volume client-07
 37:  type protocol/client
 38:  option transport-type tcp/client
 39:  option remote-host saturn
 40:  option remote-subvolume brick
 41: end-volume
 42: 
 43: 
 44: volume unify
 45:  type cluster/distribute
 46:  option lookup-unhashed yes
 47:  option min-free-disk 5%
 48:  subvolumes client-02 client-03 client-04 client-05 client-06 client-07 
 49: end-volume
 50: 
 51: volume unify-iot
 52:   type performance/io-threads
 53:   option thread-count 16
 54:   subvolumes unify
 55: end-volume
 56: 
 57: volume unify-wbd
 58:   type performance/write-behind
 59:   subvolumes unify-iot
 60: end-volume
 61: 
 62: volume unify-ioc
 63:   type performance/io-cache
 64:   option cache-size 1024MB
 65:   subvolumes unify-wbd
 66: end-volume

+------------------------------------------------------------------------------+
[2010-01-27 11:10:56] N [glusterfsd.c:1361:main] glusterfs: Successfully started
[2010-01-27 11:10:56] N [client-protocol.c:6224:client_setvolume_cbk] client-06: Connected to 192.168.0.6:6996, attached to remote volume 'brick'.
[2010-01-27 11:10:56] N [fuse-bridge.c:2931:fuse_init] glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.10
[2010-01-27 11:10:56] N [client-protocol.c:6224:client_setvolume_cbk] client-02: Connected to 192.168.0.2:6996, attached to remote volume 'brick'.
[2010-01-27 11:10:56] N [client-protocol.c:6224:client_setvolume_cbk] client-03: Connected to 192.168.0.3:6996, attached to remote volume 'brick'.
[2010-01-27 11:10:56] N [client-protocol.c:6224:client_setvolume_cbk] client-05: Connected to 192.168.0.5:6996, attached to remote volume 'brick'.
[2010-01-27 11:10:56] N [client-protocol.c:6224:client_setvolume_cbk] client-04: Connected to 192.168.0.4:6996, attached to remote volume 'brick'.
[2010-01-27 11:10:56] N [client-protocol.c:6224:client_setvolume_cbk] client-02: Connected to 192.168.0.2:6996, attached to remote volume 'brick'.
[2010-01-27 11:10:56] N [client-protocol.c:6224:client_setvolume_cbk] client-03: Connected to 192.168.0.3:6996, attached to remote volume 'brick'.
[2010-01-27 11:10:56] N [client-protocol.c:6224:client_setvolume_cbk] client-06: Connected to 192.168.0.6:6996, attached to remote volume 'brick'.
[2010-01-27 11:10:56] N [client-protocol.c:6224:client_setvolume_cbk] client-04: Connected to 192.168.0.4:6996, attached to remote volume 'brick'.
[2010-01-27 11:10:56] N [client-protocol.c:6224:client_setvolume_cbk] client-05: Connected to 192.168.0.5:6996, attached to remote volume 'brick'.
[2010-01-27 11:10:56] N [client-protocol.c:6224:client_setvolume_cbk] client-07: Connected to 192.168.0.7:6996, attached to remote volume 'brick'.
[2010-01-27 11:10:56] N [client-protocol.c:6224:client_setvolume_cbk] client-07: Connected to 192.168.0.7:6996, attached to remote volume 'brick'.
Comment 1 Anand Avati 2010-01-28 11:11:32 UTC 
PATCH: http://patches.gluster.com/patch/2716 in master (storage/posix: Fix device number handling.)
Comment 2 Eatdirt 2010-02-04 09:55:37 UTC 
Cool, thanks. Does this patch also fixes the second mentioned issue?

[2010-02-04 13:41:02] W [fuse-bridge.c:491:fuse_entry_cbk] glusterfs-fuse: LOOKUP(/chris/simusace/runspace/bin50m239_6) inode (ptr=0x2aaaac004d90, ino=14547067457, gen=5431714842828341670) found conflict (ptr=0x2aaaac056cb0, ino=14547067457, gen=5431714842828341670))


Cheers,
Chris.
Comment 3 Anand Avati 2010-02-08 04:27:54 UTC 
PATCH: http://patches.gluster.com/patch/2716 in release-3.0 (storage/posix: Fix device number handling.)