Created attachment 922114 [details] Gluster logs Description of problem: I followed howto to test volume performance stats. I created single volume and wanted to mount it on different host (engine installed on it). I tried: mount -t nfs 192.168.1.9:/vol1 /media/volume but got: mount.nfs: requested NFS version or transport protocol is not supported after trying several times all the connections were lost on the host that I wanted to mount a volume. I after the communication recovered I tried: mount -o mountproto=tcp -t nfs 192.168.1.9:/vol1 /media/volume but still got the same: mount.nfs: requested NFS version or transport protocol is not supported How reproducible: Steps to Reproduce: 1. Follow: http://www.ovirt.org/Features/Gluster_Volume_Performance_Statistics#HOW_TO 2. 3. Actual results: Unable to mount volume. All the connections (ssh, http) were dropped. Expected results: Volume mounted Additional info:
Was glusterfs-nfs server running on the node 192.168.1.9? Could you provide details of the versions of rpms used?
I have installed: Name : glusterfs Arch : x86_64 Version : 3.5.1 Release : 1.el6 I checked for service with the name "glusterfs-nfs" and it is not there. Not having this service should not break communication to the host that you are mounting a volume on.
Please try using following to mount - mount -o mountproto=tcp,vers=3 -t nfs 192.168.1.9:/vol1 /media/volume
I get timeout when running above command.
Could you please provide the following data: NB: Run the commands as root A. m/c where Gluster NFS server is running: 1. ps aux |grep gluster 2. iptables -L If firewall is running make sure to turn OFF (iptables -F) 3. showmount -e 4. rpcinfo -p 5. Gluster NFS server log e.g. /var/log/glusterfs/nfs.log B. From NFS client machine (from where mount is being tried): 1. showmount -e <NFS server machine name/IP addr> 2. mount -vvv -t nfs -o vers=3,tcp 192.168.1.9:/vol1 /media/volume
Ideally NFS mount (or mount command) should not be able to play with other SSH connections. If it's, then there might be other N/W issues. BTW, what is the OS (with version) running at client and server machines?
(In reply to santosh pradhan from comment #6) > Ideally NFS mount (or mount command) should not be able to play with other > SSH connections. If it's, then there might be other N/W issues. > > BTW, what is the OS (with version) running at client and server machines? Client is Fedora 20, and server is RHEL 6.5
Created attachment 923123 [details] gluster nfs log
(In reply to santosh pradhan from comment #5) > Could you please provide the following data: > > NB: Run the commands as root > > A. m/c where Gluster NFS server is running: > > 1. ps aux |grep gluster > root 2547 0.0 0.0 205884 9028 ? S 08:41 0:00 python /usr/lib64/nagios/plugins/gluster/check_proc_status.py start root 5286 0.0 0.0 413832 16164 ? Ssl 08:56 0:01 /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid root 5303 0.0 0.1 516820 18884 ? Ssl 08:56 0:00 /usr/sbin/glusterfsd -s 192.168.1.9 --volfile-id vol1.192.168.1.9.bricks -p /var/lib/glusterd/vols/vol1/run/192.168.1.9-bricks.pid -S /var/run/d516a329f1910e6130e8826f403d58c2.socket --brick-name /bricks -l /var/log/glusterfs/bricks/bricks.log --xlator-option *-posix.glusterd-uuid=928b5be9-6654-477a-aeaf-5520b2cfb9a1 --brick-port 49152 --xlator-option vol1-server.listen-port=49152 root 17461 0.0 0.0 103256 852 pts/0 S+ 09:40 0:00 grep gluster > > 2. iptables -L > If firewall is running make sure to turn OFF (iptables -F) > Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT all -- anywhere anywhere ACCEPT tcp -- anywhere anywhere tcp dpt:54321 ACCEPT tcp -- anywhere anywhere tcp dpt:ssh ACCEPT udp -- anywhere anywhere udp dpt:snmp ACCEPT tcp -- anywhere anywhere tcp dpt:16514 ACCEPT tcp -- anywhere anywhere multiport dports vnc-server:6923 ACCEPT tcp -- anywhere anywhere multiport dports 49152:49216 ACCEPT tcp -- anywhere anywhere tcp dpt:24007 ACCEPT tcp -- anywhere anywhere tcp dpt:webcache ACCEPT udp -- anywhere anywhere udp dpt:sunrpc ACCEPT tcp -- anywhere anywhere tcp dpt:38465 ACCEPT tcp -- anywhere anywhere tcp dpt:38466 ACCEPT tcp -- anywhere anywhere tcp dpt:sunrpc ACCEPT tcp -- anywhere anywhere tcp dpt:38467 ACCEPT tcp -- anywhere anywhere tcp dpt:nfs ACCEPT tcp -- anywhere anywhere tcp dpt:38469 ACCEPT tcp -- anywhere anywhere tcp dpt:5666 ACCEPT tcp -- anywhere anywhere tcp dpt:39543 ACCEPT tcp -- anywhere anywhere tcp dpt:55863 ACCEPT tcp -- anywhere anywhere tcp dpt:38468 ACCEPT udp -- anywhere anywhere udp dpt:963 ACCEPT tcp -- anywhere anywhere tcp dpt:965 ACCEPT tcp -- anywhere anywhere tcp dpt:ctdb ACCEPT tcp -- anywhere anywhere tcp dpt:netbios-ssn ACCEPT tcp -- anywhere anywhere tcp dpt:microsoft-ds ACCEPT tcp -- anywhere anywhere tcp dpts:24009:24108 ACCEPT tcp -- anywhere anywhere tcp dpts:49152:49251 REJECT all -- anywhere anywhere reject-with icmp-host-prohibited Chain FORWARD (policy ACCEPT) target prot opt source destination REJECT all -- anywhere anywhere PHYSDEV match ! --physdev-is-bridged reject-with icmp-host-prohibited Chain OUTPUT (policy ACCEPT) target prot opt source destination > > 3. showmount -e > Export list for rhel.localdomain: /var/lib/exports/iso 0.0.0.0/0.0.0.0 > > 4. rpcinfo -p > program vers proto port service 100000 4 tcp 111 portmapper 100000 3 tcp 111 portmapper 100000 2 tcp 111 portmapper 100000 4 udp 111 portmapper 100000 3 udp 111 portmapper 100000 2 udp 111 portmapper 100024 1 udp 662 status 100024 1 tcp 662 status 100011 1 udp 875 rquotad 100011 2 udp 875 rquotad 100011 1 tcp 875 rquotad 100011 2 tcp 875 rquotad 100005 1 udp 892 mountd 100005 1 tcp 892 mountd 100005 2 udp 892 mountd 100005 2 tcp 892 mountd 100005 3 udp 892 mountd 100005 3 tcp 892 mountd 100003 2 tcp 2049 nfs 100003 3 tcp 2049 nfs 100003 4 tcp 2049 nfs 100227 2 tcp 2049 nfs_acl 100227 3 tcp 2049 nfs_acl 100003 2 udp 2049 nfs 100003 3 udp 2049 nfs 100003 4 udp 2049 nfs 100227 2 udp 2049 nfs_acl 100227 3 udp 2049 nfs_acl 100021 1 udp 32769 nlockmgr 100021 3 udp 32769 nlockmgr 100021 4 udp 32769 nlockmgr 100021 1 tcp 32803 nlockmgr 100021 3 tcp 32803 nlockmgr 100021 4 tcp 32803 nlockmgr > > 5. Gluster NFS server log e.g. /var/log/glusterfs/nfs.log > Attached > > B. From NFS client machine (from where mount is being tried): > > 1. showmount -e <NFS server machine name/IP addr> > I had to stop iptables on Gluster machine. Export list for 192.168.1.9: /var/lib/exports/iso 0.0.0.0/0.0.0.0 > > 2. mount -vvv -t nfs -o vers=3,tcp 192.168.1.9:/vol1 /media/volume Running wiht iptables stopped on Gluster machine. mount.nfs: timeout set for Fri Aug 1 09:49:15 2014 mount.nfs: trying text-based options 'vers=3,tcp,addr=192.168.1.9' mount.nfs: prog 100003, trying vers=3, prot=6 mount.nfs: trying 192.168.1.9 prog 100003 vers 3 prot TCP port 2049 mount.nfs: prog 100005, trying vers=3, prot=6 mount.nfs: trying 192.168.1.9 prog 100005 vers 3 prot TCP port 892 mount.nfs: mount(2): Permission denied mount.nfs: access denied by server while mounting 192.168.1.9:/vol1
(In reply to Piotr Kliczewski from comment #9) > (In reply to santosh pradhan from comment #5) > > Could you please provide the following data: > > > > NB: Run the commands as root > > > > A. m/c where Gluster NFS server is running: > > > > 1. ps aux |grep gluster > > > root 2547 0.0 0.0 205884 9028 ? S 08:41 0:00 python > /usr/lib64/nagios/plugins/gluster/check_proc_status.py start > root 5286 0.0 0.0 413832 16164 ? Ssl 08:56 0:01 > /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid > root 5303 0.0 0.1 516820 18884 ? Ssl 08:56 0:00 > /usr/sbin/glusterfsd -s 192.168.1.9 --volfile-id vol1.192.168.1.9.bricks -p > /var/lib/glusterd/vols/vol1/run/192.168.1.9-bricks.pid -S > /var/run/d516a329f1910e6130e8826f403d58c2.socket --brick-name /bricks -l > /var/log/glusterfs/bricks/bricks.log --xlator-option > *-posix.glusterd-uuid=928b5be9-6654-477a-aeaf-5520b2cfb9a1 --brick-port > 49152 --xlator-option vol1-server.listen-port=49152 > root 17461 0.0 0.0 103256 852 pts/0 S+ 09:40 0:00 grep gluster > > [Santosh] Gluster NFS process is not running. So, Gluster NFS cant be mountable :) > > 2. iptables -L > > If firewall is running make sure to turn OFF (iptables -F) > > > Chain INPUT (policy ACCEPT) > target prot opt source destination > ACCEPT all -- anywhere anywhere state > RELATED,ESTABLISHED > ACCEPT all -- anywhere anywhere > ACCEPT tcp -- anywhere anywhere tcp dpt:54321 > ACCEPT tcp -- anywhere anywhere tcp dpt:ssh > ACCEPT udp -- anywhere anywhere udp dpt:snmp > ACCEPT tcp -- anywhere anywhere tcp dpt:16514 > ACCEPT tcp -- anywhere anywhere multiport > dports vnc-server:6923 > ACCEPT tcp -- anywhere anywhere multiport > dports 49152:49216 > ACCEPT tcp -- anywhere anywhere tcp dpt:24007 > ACCEPT tcp -- anywhere anywhere tcp > dpt:webcache > ACCEPT udp -- anywhere anywhere udp dpt:sunrpc > ACCEPT tcp -- anywhere anywhere tcp dpt:38465 > ACCEPT tcp -- anywhere anywhere tcp dpt:38466 > ACCEPT tcp -- anywhere anywhere tcp dpt:sunrpc > ACCEPT tcp -- anywhere anywhere tcp dpt:38467 > ACCEPT tcp -- anywhere anywhere tcp dpt:nfs > ACCEPT tcp -- anywhere anywhere tcp dpt:38469 > ACCEPT tcp -- anywhere anywhere tcp dpt:5666 > ACCEPT tcp -- anywhere anywhere tcp dpt:39543 > ACCEPT tcp -- anywhere anywhere tcp dpt:55863 > ACCEPT tcp -- anywhere anywhere tcp dpt:38468 > ACCEPT udp -- anywhere anywhere udp dpt:963 > ACCEPT tcp -- anywhere anywhere tcp dpt:965 > ACCEPT tcp -- anywhere anywhere tcp dpt:ctdb > ACCEPT tcp -- anywhere anywhere tcp > dpt:netbios-ssn > ACCEPT tcp -- anywhere anywhere tcp > dpt:microsoft-ds > ACCEPT tcp -- anywhere anywhere tcp > dpts:24009:24108 > ACCEPT tcp -- anywhere anywhere tcp > dpts:49152:49251 > REJECT all -- anywhere anywhere reject-with > icmp-host-prohibited > > Chain FORWARD (policy ACCEPT) > target prot opt source destination > REJECT all -- anywhere anywhere PHYSDEV match ! > --physdev-is-bridged reject-with icmp-host-prohibited > > Chain OUTPUT (policy ACCEPT) > target prot opt source destination > > [Santosh] Here, Firewall is running. I had suggested to you to turn it OFF (iptables -F). > > 3. showmount -e > > > Export list for rhel.localdomain: > /var/lib/exports/iso 0.0.0.0/0.0.0.0 [Santosh] Its coming from kernel NFS not from Gluster NFS. While running Gluster NFS, kernel NFS must be OFF (/etc/init.d/nfs stop or so). Looks like kernel NFS is running. > > > > 4. rpcinfo -p > > > program vers proto port service > 100000 4 tcp 111 portmapper > 100000 3 tcp 111 portmapper > 100000 2 tcp 111 portmapper > 100000 4 udp 111 portmapper > 100000 3 udp 111 portmapper > 100000 2 udp 111 portmapper > 100024 1 udp 662 status > 100024 1 tcp 662 status > 100011 1 udp 875 rquotad > 100011 2 udp 875 rquotad > 100011 1 tcp 875 rquotad > 100011 2 tcp 875 rquotad > 100005 1 udp 892 mountd > 100005 1 tcp 892 mountd > 100005 2 udp 892 mountd > 100005 2 tcp 892 mountd > 100005 3 udp 892 mountd > 100005 3 tcp 892 mountd > 100003 2 tcp 2049 nfs > 100003 3 tcp 2049 nfs > 100003 4 tcp 2049 nfs > 100227 2 tcp 2049 nfs_acl > 100227 3 tcp 2049 nfs_acl > 100003 2 udp 2049 nfs > 100003 3 udp 2049 nfs > 100003 4 udp 2049 nfs > 100227 2 udp 2049 nfs_acl > 100227 3 udp 2049 nfs_acl > 100021 1 udp 32769 nlockmgr > 100021 3 udp 32769 nlockmgr > 100021 4 udp 32769 nlockmgr > 100021 1 tcp 32803 nlockmgr > 100021 3 tcp 32803 nlockmgr > 100021 4 tcp 32803 nlockmgr > > [Santosh] Its confirmed that kernel NFS is running in the machine. > > 5. Gluster NFS server log e.g. /var/log/glusterfs/nfs.log > > [Santosh] [2014-08-01 06:56:51.870503] E [nfs.c:332:nfs_init_versions] 0-nfs: Program MOUNT3 registration failed [2014-08-01 06:56:51.870512] E [nfs.c:1312:init] 0-nfs: Failed to initialize protocols [2014-08-01 06:56:51.870520] E [xlator.c:403:xlator_init] 0-nfs-server: Initialization of volume 'nfs-server' failed, review your volfile again [2014-08-01 06:56:51.870526] E [graph.c:307:glusterfs_graph_init] 0-nfs-server: initializing translator failed [2014-08-01 06:56:51.870532] E [graph.c:502:glusterfs_graph_activate] 0-graph: init failed [2014-08-01 06:56:51.870731] W [glusterfsd.c:1095:cleanup_and_exit] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x335c60e6d5] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x320) [0x40bd50] (-->/usr/sbin/glusterfs(glusterfs_process_volfp+0x106) [0x405146]))) 0-: received signum (0), shutting down Gluster NFS is not able to register with rpcbind because ports are already used by kernel NFS. So, its not a bug. I ll be closing it. Suggestions: 1. Turn kernel NFS service off. 2. Turn iptables OFF (iptables -F). Let me know if you still face any issue. Thanks, Santosh