Bug 1124376
Summary: | [gluster] Lost connection during attempt to mount a volume | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | [Retired] oVirt | Reporter: | Piotr Kliczewski <pkliczew> | ||||||
Component: | vdsm | Assignee: | santosh pradhan <spradhan> | ||||||
Status: | CLOSED NOTABUG | QA Contact: | Gil Klein <gklein> | ||||||
Severity: | unspecified | Docs Contact: | |||||||
Priority: | unspecified | ||||||||
Version: | 3.5 | CC: | acathrow, bazulay, bugs, ecohen, gklein, iheim, mgoldboi, pkliczew, sabose, spradhan, vagarwal, yeylon | ||||||
Target Milestone: | --- | ||||||||
Target Release: | --- | ||||||||
Hardware: | Unspecified | ||||||||
OS: | Unspecified | ||||||||
Whiteboard: | gluster | ||||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||||
Doc Text: | Story Points: | --- | |||||||
Clone Of: | Environment: | ||||||||
Last Closed: | 2014-08-01 12:40:25 UTC | Type: | Bug | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Attachments: |
|
Description
Piotr Kliczewski
2014-07-29 11:28:59 UTC
Was glusterfs-nfs server running on the node 192.168.1.9? Could you provide details of the versions of rpms used? I have installed: Name : glusterfs Arch : x86_64 Version : 3.5.1 Release : 1.el6 I checked for service with the name "glusterfs-nfs" and it is not there. Not having this service should not break communication to the host that you are mounting a volume on. Please try using following to mount - mount -o mountproto=tcp,vers=3 -t nfs 192.168.1.9:/vol1 /media/volume I get timeout when running above command. Could you please provide the following data: NB: Run the commands as root A. m/c where Gluster NFS server is running: 1. ps aux |grep gluster 2. iptables -L If firewall is running make sure to turn OFF (iptables -F) 3. showmount -e 4. rpcinfo -p 5. Gluster NFS server log e.g. /var/log/glusterfs/nfs.log B. From NFS client machine (from where mount is being tried): 1. showmount -e <NFS server machine name/IP addr> 2. mount -vvv -t nfs -o vers=3,tcp 192.168.1.9:/vol1 /media/volume Ideally NFS mount (or mount command) should not be able to play with other SSH connections. If it's, then there might be other N/W issues. BTW, what is the OS (with version) running at client and server machines? (In reply to santosh pradhan from comment #6) > Ideally NFS mount (or mount command) should not be able to play with other > SSH connections. If it's, then there might be other N/W issues. > > BTW, what is the OS (with version) running at client and server machines? Client is Fedora 20, and server is RHEL 6.5 Created attachment 923123 [details]
gluster nfs log
(In reply to santosh pradhan from comment #5) > Could you please provide the following data: > > NB: Run the commands as root > > A. m/c where Gluster NFS server is running: > > 1. ps aux |grep gluster > root 2547 0.0 0.0 205884 9028 ? S 08:41 0:00 python /usr/lib64/nagios/plugins/gluster/check_proc_status.py start root 5286 0.0 0.0 413832 16164 ? Ssl 08:56 0:01 /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid root 5303 0.0 0.1 516820 18884 ? Ssl 08:56 0:00 /usr/sbin/glusterfsd -s 192.168.1.9 --volfile-id vol1.192.168.1.9.bricks -p /var/lib/glusterd/vols/vol1/run/192.168.1.9-bricks.pid -S /var/run/d516a329f1910e6130e8826f403d58c2.socket --brick-name /bricks -l /var/log/glusterfs/bricks/bricks.log --xlator-option *-posix.glusterd-uuid=928b5be9-6654-477a-aeaf-5520b2cfb9a1 --brick-port 49152 --xlator-option vol1-server.listen-port=49152 root 17461 0.0 0.0 103256 852 pts/0 S+ 09:40 0:00 grep gluster > > 2. iptables -L > If firewall is running make sure to turn OFF (iptables -F) > Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT all -- anywhere anywhere ACCEPT tcp -- anywhere anywhere tcp dpt:54321 ACCEPT tcp -- anywhere anywhere tcp dpt:ssh ACCEPT udp -- anywhere anywhere udp dpt:snmp ACCEPT tcp -- anywhere anywhere tcp dpt:16514 ACCEPT tcp -- anywhere anywhere multiport dports vnc-server:6923 ACCEPT tcp -- anywhere anywhere multiport dports 49152:49216 ACCEPT tcp -- anywhere anywhere tcp dpt:24007 ACCEPT tcp -- anywhere anywhere tcp dpt:webcache ACCEPT udp -- anywhere anywhere udp dpt:sunrpc ACCEPT tcp -- anywhere anywhere tcp dpt:38465 ACCEPT tcp -- anywhere anywhere tcp dpt:38466 ACCEPT tcp -- anywhere anywhere tcp dpt:sunrpc ACCEPT tcp -- anywhere anywhere tcp dpt:38467 ACCEPT tcp -- anywhere anywhere tcp dpt:nfs ACCEPT tcp -- anywhere anywhere tcp dpt:38469 ACCEPT tcp -- anywhere anywhere tcp dpt:5666 ACCEPT tcp -- anywhere anywhere tcp dpt:39543 ACCEPT tcp -- anywhere anywhere tcp dpt:55863 ACCEPT tcp -- anywhere anywhere tcp dpt:38468 ACCEPT udp -- anywhere anywhere udp dpt:963 ACCEPT tcp -- anywhere anywhere tcp dpt:965 ACCEPT tcp -- anywhere anywhere tcp dpt:ctdb ACCEPT tcp -- anywhere anywhere tcp dpt:netbios-ssn ACCEPT tcp -- anywhere anywhere tcp dpt:microsoft-ds ACCEPT tcp -- anywhere anywhere tcp dpts:24009:24108 ACCEPT tcp -- anywhere anywhere tcp dpts:49152:49251 REJECT all -- anywhere anywhere reject-with icmp-host-prohibited Chain FORWARD (policy ACCEPT) target prot opt source destination REJECT all -- anywhere anywhere PHYSDEV match ! --physdev-is-bridged reject-with icmp-host-prohibited Chain OUTPUT (policy ACCEPT) target prot opt source destination > > 3. showmount -e > Export list for rhel.localdomain: /var/lib/exports/iso 0.0.0.0/0.0.0.0 > > 4. rpcinfo -p > program vers proto port service 100000 4 tcp 111 portmapper 100000 3 tcp 111 portmapper 100000 2 tcp 111 portmapper 100000 4 udp 111 portmapper 100000 3 udp 111 portmapper 100000 2 udp 111 portmapper 100024 1 udp 662 status 100024 1 tcp 662 status 100011 1 udp 875 rquotad 100011 2 udp 875 rquotad 100011 1 tcp 875 rquotad 100011 2 tcp 875 rquotad 100005 1 udp 892 mountd 100005 1 tcp 892 mountd 100005 2 udp 892 mountd 100005 2 tcp 892 mountd 100005 3 udp 892 mountd 100005 3 tcp 892 mountd 100003 2 tcp 2049 nfs 100003 3 tcp 2049 nfs 100003 4 tcp 2049 nfs 100227 2 tcp 2049 nfs_acl 100227 3 tcp 2049 nfs_acl 100003 2 udp 2049 nfs 100003 3 udp 2049 nfs 100003 4 udp 2049 nfs 100227 2 udp 2049 nfs_acl 100227 3 udp 2049 nfs_acl 100021 1 udp 32769 nlockmgr 100021 3 udp 32769 nlockmgr 100021 4 udp 32769 nlockmgr 100021 1 tcp 32803 nlockmgr 100021 3 tcp 32803 nlockmgr 100021 4 tcp 32803 nlockmgr > > 5. Gluster NFS server log e.g. /var/log/glusterfs/nfs.log > Attached > > B. From NFS client machine (from where mount is being tried): > > 1. showmount -e <NFS server machine name/IP addr> > I had to stop iptables on Gluster machine. Export list for 192.168.1.9: /var/lib/exports/iso 0.0.0.0/0.0.0.0 > > 2. mount -vvv -t nfs -o vers=3,tcp 192.168.1.9:/vol1 /media/volume Running wiht iptables stopped on Gluster machine. mount.nfs: timeout set for Fri Aug 1 09:49:15 2014 mount.nfs: trying text-based options 'vers=3,tcp,addr=192.168.1.9' mount.nfs: prog 100003, trying vers=3, prot=6 mount.nfs: trying 192.168.1.9 prog 100003 vers 3 prot TCP port 2049 mount.nfs: prog 100005, trying vers=3, prot=6 mount.nfs: trying 192.168.1.9 prog 100005 vers 3 prot TCP port 892 mount.nfs: mount(2): Permission denied mount.nfs: access denied by server while mounting 192.168.1.9:/vol1 (In reply to Piotr Kliczewski from comment #9) > (In reply to santosh pradhan from comment #5) > > Could you please provide the following data: > > > > NB: Run the commands as root > > > > A. m/c where Gluster NFS server is running: > > > > 1. ps aux |grep gluster > > > root 2547 0.0 0.0 205884 9028 ? S 08:41 0:00 python > /usr/lib64/nagios/plugins/gluster/check_proc_status.py start > root 5286 0.0 0.0 413832 16164 ? Ssl 08:56 0:01 > /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid > root 5303 0.0 0.1 516820 18884 ? Ssl 08:56 0:00 > /usr/sbin/glusterfsd -s 192.168.1.9 --volfile-id vol1.192.168.1.9.bricks -p > /var/lib/glusterd/vols/vol1/run/192.168.1.9-bricks.pid -S > /var/run/d516a329f1910e6130e8826f403d58c2.socket --brick-name /bricks -l > /var/log/glusterfs/bricks/bricks.log --xlator-option > *-posix.glusterd-uuid=928b5be9-6654-477a-aeaf-5520b2cfb9a1 --brick-port > 49152 --xlator-option vol1-server.listen-port=49152 > root 17461 0.0 0.0 103256 852 pts/0 S+ 09:40 0:00 grep gluster > > [Santosh] Gluster NFS process is not running. So, Gluster NFS cant be mountable :) > > 2. iptables -L > > If firewall is running make sure to turn OFF (iptables -F) > > > Chain INPUT (policy ACCEPT) > target prot opt source destination > ACCEPT all -- anywhere anywhere state > RELATED,ESTABLISHED > ACCEPT all -- anywhere anywhere > ACCEPT tcp -- anywhere anywhere tcp dpt:54321 > ACCEPT tcp -- anywhere anywhere tcp dpt:ssh > ACCEPT udp -- anywhere anywhere udp dpt:snmp > ACCEPT tcp -- anywhere anywhere tcp dpt:16514 > ACCEPT tcp -- anywhere anywhere multiport > dports vnc-server:6923 > ACCEPT tcp -- anywhere anywhere multiport > dports 49152:49216 > ACCEPT tcp -- anywhere anywhere tcp dpt:24007 > ACCEPT tcp -- anywhere anywhere tcp > dpt:webcache > ACCEPT udp -- anywhere anywhere udp dpt:sunrpc > ACCEPT tcp -- anywhere anywhere tcp dpt:38465 > ACCEPT tcp -- anywhere anywhere tcp dpt:38466 > ACCEPT tcp -- anywhere anywhere tcp dpt:sunrpc > ACCEPT tcp -- anywhere anywhere tcp dpt:38467 > ACCEPT tcp -- anywhere anywhere tcp dpt:nfs > ACCEPT tcp -- anywhere anywhere tcp dpt:38469 > ACCEPT tcp -- anywhere anywhere tcp dpt:5666 > ACCEPT tcp -- anywhere anywhere tcp dpt:39543 > ACCEPT tcp -- anywhere anywhere tcp dpt:55863 > ACCEPT tcp -- anywhere anywhere tcp dpt:38468 > ACCEPT udp -- anywhere anywhere udp dpt:963 > ACCEPT tcp -- anywhere anywhere tcp dpt:965 > ACCEPT tcp -- anywhere anywhere tcp dpt:ctdb > ACCEPT tcp -- anywhere anywhere tcp > dpt:netbios-ssn > ACCEPT tcp -- anywhere anywhere tcp > dpt:microsoft-ds > ACCEPT tcp -- anywhere anywhere tcp > dpts:24009:24108 > ACCEPT tcp -- anywhere anywhere tcp > dpts:49152:49251 > REJECT all -- anywhere anywhere reject-with > icmp-host-prohibited > > Chain FORWARD (policy ACCEPT) > target prot opt source destination > REJECT all -- anywhere anywhere PHYSDEV match ! > --physdev-is-bridged reject-with icmp-host-prohibited > > Chain OUTPUT (policy ACCEPT) > target prot opt source destination > > [Santosh] Here, Firewall is running. I had suggested to you to turn it OFF (iptables -F). > > 3. showmount -e > > > Export list for rhel.localdomain: > /var/lib/exports/iso 0.0.0.0/0.0.0.0 [Santosh] Its coming from kernel NFS not from Gluster NFS. While running Gluster NFS, kernel NFS must be OFF (/etc/init.d/nfs stop or so). Looks like kernel NFS is running. > > > > 4. rpcinfo -p > > > program vers proto port service > 100000 4 tcp 111 portmapper > 100000 3 tcp 111 portmapper > 100000 2 tcp 111 portmapper > 100000 4 udp 111 portmapper > 100000 3 udp 111 portmapper > 100000 2 udp 111 portmapper > 100024 1 udp 662 status > 100024 1 tcp 662 status > 100011 1 udp 875 rquotad > 100011 2 udp 875 rquotad > 100011 1 tcp 875 rquotad > 100011 2 tcp 875 rquotad > 100005 1 udp 892 mountd > 100005 1 tcp 892 mountd > 100005 2 udp 892 mountd > 100005 2 tcp 892 mountd > 100005 3 udp 892 mountd > 100005 3 tcp 892 mountd > 100003 2 tcp 2049 nfs > 100003 3 tcp 2049 nfs > 100003 4 tcp 2049 nfs > 100227 2 tcp 2049 nfs_acl > 100227 3 tcp 2049 nfs_acl > 100003 2 udp 2049 nfs > 100003 3 udp 2049 nfs > 100003 4 udp 2049 nfs > 100227 2 udp 2049 nfs_acl > 100227 3 udp 2049 nfs_acl > 100021 1 udp 32769 nlockmgr > 100021 3 udp 32769 nlockmgr > 100021 4 udp 32769 nlockmgr > 100021 1 tcp 32803 nlockmgr > 100021 3 tcp 32803 nlockmgr > 100021 4 tcp 32803 nlockmgr > > [Santosh] Its confirmed that kernel NFS is running in the machine. > > 5. Gluster NFS server log e.g. /var/log/glusterfs/nfs.log > > [Santosh] [2014-08-01 06:56:51.870503] E [nfs.c:332:nfs_init_versions] 0-nfs: Program MOUNT3 registration failed [2014-08-01 06:56:51.870512] E [nfs.c:1312:init] 0-nfs: Failed to initialize protocols [2014-08-01 06:56:51.870520] E [xlator.c:403:xlator_init] 0-nfs-server: Initialization of volume 'nfs-server' failed, review your volfile again [2014-08-01 06:56:51.870526] E [graph.c:307:glusterfs_graph_init] 0-nfs-server: initializing translator failed [2014-08-01 06:56:51.870532] E [graph.c:502:glusterfs_graph_activate] 0-graph: init failed [2014-08-01 06:56:51.870731] W [glusterfsd.c:1095:cleanup_and_exit] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x335c60e6d5] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x320) [0x40bd50] (-->/usr/sbin/glusterfs(glusterfs_process_volfp+0x106) [0x405146]))) 0-: received signum (0), shutting down Gluster NFS is not able to register with rpcbind because ports are already used by kernel NFS. So, its not a bug. I ll be closing it. Suggestions: 1. Turn kernel NFS service off. 2. Turn iptables OFF (iptables -F). Let me know if you still face any issue. Thanks, Santosh |