I'm running a couple of OpenVZ virtual machines on a proxmox cluster (mostly Debian lenny). The virtual machines have their /opt directory on a replicated glusterfs volume. After booting and rebooting the virtual machines a couple of times, I get the above error message when running with loglevel debug. glusterfs seems to terminate at that point, the directory is no longer accessible. See attachment for full log.
The logs indicate someone calling a umount on the mountpoint. This is clearly initiated from the mountpoint and not from within the filesystem. It might be the case that some script is calling it automatically, or trying a mount -o remount which is not supported at the moment. Avati
I just tried this on an otherwise idle system. What I see in the logs upon terminating the virtual machine is this: [2011-01-21 08:25:08.965152] I [client-handshake.c:1005:select_server_supported_programs] common-opt-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310) [2011-01-21 08:25:08.965472] I [client-handshake.c:841:client_setvolume_cbk] common-opt-client-1: Connected to 10.100.100.165:24011, attached to remote volume '/media/gluster/brick0/common-opt'. [2011-01-21 08:25:08.965520] D [client-handshake.c:723:client_post_handshake] common-opt-client-1: No open fds - notifying all parents child up [2011-02-01 12:31:52.240616] D [stat-prefetch.c:3587:sp_release] common-opt-stat-prefetch: cache hits: 0, cache miss: 0 [2011-02-02 07:00:21.758839] D [stat-prefetch.c:3587:sp_release] common-opt-stat-prefetch: cache hits: 0, cache miss: 0 [2011-02-02 07:00:33.435716] D [stat-prefetch.c:3587:sp_release] common-opt-stat-prefetch: cache hits: 0, cache miss: 0 [2011-02-02 07:00:53.213114] D [stat-prefetch.c:3587:sp_release] common-opt-stat-prefetch: cache hits: 0, cache miss: 0 [2011-02-02 07:01:06.221764] D [fuse-bridge.c:3055:fuse_thread_proc] glusterfs-fuse: terminating upon getting ENODEV when reading /dev/fuse [2011-02-02 07:01:06.221827] I [fuse-bridge.c:3142:fuse_thread_proc] fuse: unmounting /server/gluster/common-opt [2011-02-02 07:01:06.221880] D [glusterfsd-mgmt.c:652:glusterfs_mgmt_pmap_signout] fsd-mgmt: portmapper signout arguments not given [2011-02-02 07:01:06.221928] I [glusterfsd.c:672:cleanup_and_exit] glusterfsd: shutting down Yes, there seems to be a umount, but this happens after the error message about ENODEV. Could the umount be a consequence of the error rather than the cause? Also, the mount is still listed in /proc/mounts, so it doesn't look like the umount is caused by some user space program.
Created attachment 429 Attached a trace log for the following test case: - mount filesystem - start openvz vm - stop openvz vm - try to access filesystem: ls: cannot access /server/gluster/common-opt/: Transport endpoint is not connected There are a couple of bind mounts within the virtual machine referring to subdirectories of the gluster filesystem. Those are unmounted when shutting down the virtual machine. This does not lead to problems when I use ext4 or ocfs2 as the underlying filesystem though.
(In reply to comment #3) > Created an attachment (id=429) [details] > trace log from client > > Attached a trace log for the following test case: > - mount filesystem > - start openvz vm > - stop openvz vm > - try to access filesystem: > ls: cannot access /server/gluster/common-opt/: Transport endpoint is not > connected > > There are a couple of bind mounts within the virtual machine referring to > subdirectories of the gluster filesystem. Those are unmounted when shutting > down the virtual machine. This does not lead to problems when I use ext4 or > ocfs2 as the underlying filesystem though. From what I see, this is the way fuse filesystems work. Can you try this with sshfs and see if that behaves any differently?. By your logs, it is clear that fuse dev is returning ENODEV first. Gluster is not initiating this. The umounting log which follows is just the clean-up process log. Avati
I have been able to reproduce the problem with sshfs, so the source of the bug seems to be within fuse. I've also been able to find a much easier test case: * mount -t glusterfs server:/share /mountpoint * mount -o bind /mountpoint /othermountpoint * umount -f /othermountpoint * ls /mountpoint --> ls: cannot access /mountpoint: Transport endpoint is not connected The -f in the umount is essential for the problem to occur. OpenVZ use regular shutdown/init scripts (in my case debian), and they use umount -f to unmount the filesystems on shutdown. So, the quick fix is to remove the -f option in the init script.
At this point I dont see how any further work is needed in Gluster towards the issue. Hence resolving.
How about moving this to the fuse component? This is a serious deficiency of glusterfs (even though it seems to be fuse's fault) and it should be resolved. In particular since glusterfs claims to be a perfect file system for virtualization.
The 'fuse' component here is the fuse-bridge code (mount/fuse translator) of GlusterFS and not the fuse kernel module itself. We would like to see this bug fixed too, but to get the fix you need to go here - http://fuse.sf.net and ask around on the mailing list so that the next upstream linux kernel gets your fix in (chances are better if you submit a patch yourself) Avati
For those who are interested. I posted the problem on the fuse-devel mailing list: http://permalink.gmane.org/gmane.comp.file-systems.fuse.devel/9961 No answer so far. I also reported it in the debian bug tracking system: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=613904 The answer there is quite surprising: this is how things are supposed to work. I still don't understand why things work differently with native file systems then (ext3 etc.).
(In reply to comment #9) > For those who are interested. I posted the problem on the fuse-devel mailing > list: > http://permalink.gmane.org/gmane.comp.file-systems.fuse.devel/9961 > > No answer so far. > > I also reported it in the debian bug tracking system: > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=613904 > > The answer there is quite surprising: this is how things are supposed to work. > I still don't understand why things work differently with native file systems > then (ext3 etc.). Just checked the link in debian bug tracking. The person who replied to you there is the fuse maintainer for kernel. Hence answers would be same as what you would get in fuse-devel mailing list. We too will keep an watch on this issue, and if any fixes get in fuse for this, you can use GlusterFS as per your need.