Description of problem: I create a distributed volume on 2 FreeBSD nodes and mount it to a Linux client. After I move or rename files, client sees duplicated files like this. -rw------- 1 root root 762667 Nov 4 11:10 example ----------- 1 root root 0 Nov 7 11:10 example (node1 have real file but node2 have create index file) Version-Release number of selected component (if applicable): Glusterfs 3.6.1 How reproducible: 1.Add peer with 2 FreeBSD nodes 2.Create distributed volume 3.Mount volume on the Linux client 4.Create files and move files or rename files Steps to Reproduce: 192.168.231.1 (host 1 FreeBSD 9.2) 192.168.231.2 (host 2 FreeBSD 9.2) 192.168.231.4 (Client Linux 6.5) (host 1) # gluster peer probe 192.168.231.2 # gluster volume create dist-vol 192.168.231.1:/mnt/test12/brick01 192.168.231.2:/mnt/test12/brick01 # gluster volume start dist-vol (Client) # mount.glusterfs 192.168.231.1:/dist-vol /mnt/distributed # touch test # mv test example Actual results: -rw------- 1 root root 762667 Nov 4 11:10 example (Real file host1) ----------- 1 root root 0 Nov 7 11:10 example (Index file host2) Expected results: -rw------- 1 root root 762667 Nov 4 11:10 example Additional info:
This might be for the 'distribute' component, or 'fuse'... Can you confirm that this problem does not happen when you mount the volume on FreeBSD? Also, please mention what architectures you are using on the client and servers. Thanks!
We haven't tested it with FreeBSD clients because it requires fuse module. We got the error [2014-10-22 07:05:33.704998] E [mount.c:318:fuse_mount_sys] 0-glusterfs-fuse: ret = -1 [2014-10-22 07:05:33.705184] I [mount.c:365:gf_fuse_mount] 0-glusterfs-fuse: direct mount failed (Device not configured) errno 6, retry to mount via fusermount [2014-10-22 07:05:33.705301] D [logging.c:1782:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5 [2014-10-22 07:05:33.705834] E [mount.c:167:fuse_mount_fusermount] 0-glusterfs-fuse: failed to exec fusermount: No such file or directory when trying to install Fuse on FreeBSD. And we use X86_64 architectures on client and servers. Thank you.
(In reply to Niels de Vos from comment #1) > This might be for the 'distribute' component, or 'fuse'... > > Can you confirm that this problem does not happen when you mount the volume > on FreeBSD? Also, please mention what architectures you are using on the > client and servers. Thanks! We haven't tested it with FreeBSD clients because it requires fuse module. We got the error [2014-10-22 07:05:33.704998] E [mount.c:318:fuse_mount_sys] 0-glusterfs-fuse: ret = -1 [2014-10-22 07:05:33.705184] I [mount.c:365:gf_fuse_mount] 0-glusterfs-fuse: direct mount failed (Device not configured) errno 6, retry to mount via fusermount [2014-10-22 07:05:33.705301] D [logging.c:1782:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5 [2014-10-22 07:05:33.705834] E [mount.c:167:fuse_mount_fusermount] 0-glusterfs-fuse: failed to exec fusermount: No such file or directory when trying to install Fuse on FreeBSD. And we use X86_64 architectures on client and servers. Thank you.
Description of problem: I create a distributed replicate volume on 4 FreeBSD nodes and mount it to a Linux client. After I move or rename files, client sees duplicated files like this. ---------- 1 root wheel 0 Aug 23 21:18 dahyun2.jpg ---------- 1 root wheel 0 Aug 23 21:18 dahyun2.jpg Version-Release number of selected component (if applicable): Glusterfs 3.7.6 FreeBSD 10.2 How reproducible: 1.Add peer with 4 FreeBSD nodes 2.Create distributed replicate volume 3.Mount volume on the Linux client 4.Create files and move files or rename files Additional Information: root@gluster01:/glu1/images # lsextattr user dahyun2.jpg dahyun2.jpg trusted.glusterfs.dht.linkto trusted.gfid root@gluster02:/glu2/images # lsextattr user dahyun2.jpg dahyun2.jpg trusted.glusterfs.dht.linkto trusted.gfid root@gluster04:/glu4/images # lsextattr user dahyun2.jpg dahyun2.jpg trusted.glusterfs.shard.file-size root@gluster05:/glu5/images # lsextattr user dahyun2.jpg dahyun2.jpg trusted.glusterfs.shard.file-size The file in gluster01 and gluster02 have ---------- file permission. ---------- 2 root wheel 0 Aug 23 21:38 dahyun2.jpg The problem can be fix by chmod 1000 the file which have trusted.glusterfs.dht.linkto extattr.
Finally got hold of a FreeBSD system so I will try this out. What filesystem are you using for the gluster bricks?
(In reply to Nithya Balachandran from comment #6) > Finally got hold of a FreeBSD system so I will try this out. > > What filesystem are you using for the gluster bricks? Filesystem Type 1K-blocks Used Avail Capacity Mounted on /dev/ada0p2 ufs 66003644 3791312 56932044 6% / devfs devfs 1 1 0 100% /dev /dev/fuse fusefs 132007168 11226496 110220160 9% /mnt/fuse All servers are identical. /dev/fuse is my mount.
Hi, So does that mean that the bricks( gluster01:/glu1, gluster02:/glu2 etc) are using the UFS filesystem? Thanks, Nithya
(In reply to Nithya Balachandran from comment #8) > Hi, > So does that mean that the bricks( gluster01:/glu1, gluster02:/glu2 etc) are > using the UFS filesystem? > > > Thanks, > Nithya Hi, Yes it does. UFS is the filesystem of my bricks. I feel like that I should also mention this. To mount the gluster I used this command: glusterfs --log-file=/var/log/glusterfs/mnt-dr --log-level=INFO --volfile-id=dr --volfile-server=<volume ip> /mnt/fuse/ You need to load the fuse by using "kldload fuse" to mount the system. Thanks, Jan
(In reply to jmmartirez from comment #9) > (In reply to Nithya Balachandran from comment #8) > > Hi, > > So does that mean that the bricks( gluster01:/glu1, gluster02:/glu2 etc) are > > using the UFS filesystem? > > > > > > Thanks, > > Nithya > > Hi, > > Yes it does. UFS is the filesystem of my bricks. > > I feel like that I should also mention this. > > To mount the gluster I used this command: > > glusterfs --log-file=/var/log/glusterfs/mnt-dr --log-level=INFO > --volfile-id=dr --volfile-server=<volume ip> /mnt/fuse/ > > You need to load the fuse by using "kldload fuse" to mount the system. > > Thanks, > Jan Sorry for asking so many questions - I have never worked on FreeBSD before. How do I install gluster on FreeBSD - I am unable to build gluster directly on the system.
(In reply to Nithya Balachandran from comment #10) > (In reply to jmmartirez from comment #9) > > (In reply to Nithya Balachandran from comment #8) > > > Hi, > > > So does that mean that the bricks( gluster01:/glu1, gluster02:/glu2 etc) are > > > using the UFS filesystem? > > > > > > > > > Thanks, > > > Nithya > > > > Hi, > > > > Yes it does. UFS is the filesystem of my bricks. > > > > I feel like that I should also mention this. > > > > To mount the gluster I used this command: > > > > glusterfs --log-file=/var/log/glusterfs/mnt-dr --log-level=INFO > > --volfile-id=dr --volfile-server=<volume ip> /mnt/fuse/ > > > > You need to load the fuse by using "kldload fuse" to mount the system. > > > > Thanks, > > Jan > > Sorry for asking so many questions - I have never worked on FreeBSD before. > How do I install gluster on FreeBSD - I am unable to build gluster directly > on the system. Hi, You should update your port by using the following command (https://www.freebsd.org/doc/handbook/ports-using.html) 1. # portsnap fetch 2. # portsnap extract 3. # portsnap fetch 4. # portsnap update these should update the "port/installer" inside your system. Then you can install it by running this command: # cd /usr/ports/net/glusterfs && make install It should compile and install the port. Thanks, Jan
> You should update your port by using the following command > (https://www.freebsd.org/doc/handbook/ports-using.html) > 1. # portsnap fetch > 2. # portsnap extract > 3. # portsnap fetch > 4. # portsnap update > > these should update the "port/installer" inside your system. > > Then you can install it by running this command: > > # cd /usr/ports/net/glusterfs && make install > > It should compile and install the port. > > Thanks, > Jan Thank you. I have built and installed gluster on the FreeBSD system and am able to reproduce the issue. I will update the BZ once I have something.
From the mount logs: [2016-08-30 09:05:41.912159] W [MSGID: 114031] [client-rpc-fops.c:2399:client3_3_create_cbk] 0-testvol-client-0: remote operation failed. Path: /file-1 [Permission denied] [2016-08-30 09:05:41.912303] W [fuse-bridge.c:1989:fuse_create_cbk] 0-glusterfs-fuse: 21: /file-1 => -1 (Permission denied) [2016-08-30 09:05:54.756836] I [MSGID: 109066] [dht-rename.c:1411:dht_rename] 0-testvol-dht: renaming /file-1 (hash=testvol-client-0/cache=testvol-client-0) => /file-2 (hash=testvol-client-1/cache=<nul>) [2016-08-30 09:05:54.763159] W [MSGID: 114031] [client-rpc-fops.c:2809:client3_3_link_cbk] 0-testvol-client-0: remote operation failed: (/file-1 -> /file-2) [Permission denied] [2016-08-30 09:05:54.763856] W [MSGID: 114031] [client-rpc-fops.c:664:client3_3_unlink_cbk] 0-testvol-client-1: remote operation failed [Permission denied] [2016-08-30 09:05:54.763891] W [MSGID: 109034] [dht-rename.c:510:dht_rename_unlink_cbk] 0-testvol-dht: /file-1: Rename: unlink on testvol-client-1 failed [Permission denied] [2016-08-30 09:05:54.764158] W [fuse-bridge.c:1777:fuse_rename_cbk] 0-glusterfs-fuse: 50: /file-1 -> /file-2 => -1 (Permission denied) I do not understand why the EPERM error is returned as all these are running as root.
GlusterFS-3.6 is nearing its End-Of-Life, only important security bugs still make a chance on getting fixed. Moving this to the mainline 'version'. If this needs to get fixed in 3.7 or 3.8 this bug should get cloned.
Created attachment 1196243 [details] test code to create a glusterfs linkto file
From https://www.freebsd.org/cgi/man.cgi?query=chmod&sektion=2 The FreeBSD VM system totally ignores the sticky bit (S_ISVTX) for executables. On UFS-based file systems (FFS, LFS) the sticky bit may only be set upon directories. This does seem like we cannot support UFS for Glusterfs bricks. However, from my tests: I was able to set the T bit if I built and ran the attached test file as root. I wanted to try the same thing in the gluster code but I couldn't get it to build after I made the changes in /usr/ports/net/glusterfs/work/glusterfs-3.7.6/xlators/storage/posix/src/posix.c If you can let me know how to do that, I can try it out anyway. However, we do rely heavily on the sticky bit for some of our internal ops so I would not recommend that UFS be used for the bricks if this is the expected behaviour. Can you please try the same with XFS bricks and see if the problem exists?
Based on the above comments, I am moving this to WontFix.