Bug 1124376 - [gluster] Lost connection during attempt to mount a volume
Summary: [gluster] Lost connection during attempt to mount a volume
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: oVirt
Classification: Retired
Component: vdsm
Version: 3.5
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: santosh pradhan
QA Contact: Gil Klein
URL:
Whiteboard: gluster
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-07-29 11:28 UTC by Piotr Kliczewski
Modified: 2014-08-04 17:20 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-08-01 12:40:25 UTC
oVirt Team: ---


Attachments (Terms of Use)
Gluster logs (176.77 KB, application/gzip)
2014-07-29 11:28 UTC, Piotr Kliczewski
no flags Details
gluster nfs log (11.47 KB, text/x-log)
2014-08-01 07:44 UTC, Piotr Kliczewski
no flags Details

Description Piotr Kliczewski 2014-07-29 11:28:59 UTC
Created attachment 922114 [details]
Gluster logs

Description of problem:

I followed howto to test volume performance stats. I created single volume and wanted to mount it on different host (engine installed on it).

I tried:

mount -t nfs 192.168.1.9:/vol1 /media/volume

but got:

mount.nfs: requested NFS version or transport protocol is not supported

after trying several times all the connections were lost on the host that I wanted to mount a volume.

I after the communication recovered I tried:

mount -o mountproto=tcp -t nfs 192.168.1.9:/vol1 /media/volume

but still got the same:

mount.nfs: requested NFS version or transport protocol is not supported



How reproducible:


Steps to Reproduce:
1. Follow: http://www.ovirt.org/Features/Gluster_Volume_Performance_Statistics#HOW_TO
2.
3.

Actual results:
Unable to mount volume. All the connections (ssh, http) were dropped.


Expected results:
Volume mounted

Additional info:

Comment 1 Sahina Bose 2014-07-30 06:09:20 UTC
Was glusterfs-nfs server running on the node 192.168.1.9?

Could you provide details of the versions of rpms used?

Comment 2 Piotr Kliczewski 2014-07-30 07:42:34 UTC
I have installed:

Name        : glusterfs
Arch        : x86_64
Version     : 3.5.1
Release     : 1.el6

I checked for service with the name "glusterfs-nfs" and it is not there.
Not having this service should not break communication to the host that you
are mounting a volume on.

Comment 3 Sahina Bose 2014-07-30 11:42:10 UTC
Please try using following to mount -

mount -o mountproto=tcp,vers=3 -t nfs 192.168.1.9:/vol1 /media/volume

Comment 4 Piotr Kliczewski 2014-07-31 21:38:22 UTC
I get timeout when running above command.

Comment 5 santosh pradhan 2014-08-01 07:00:50 UTC
Could you please provide the following data:

NB: Run the commands as root

A. m/c where Gluster NFS server is running:

1. ps aux |grep gluster

2. iptables -L
If firewall is running make sure to turn OFF (iptables -F) 

3. showmount -e

4. rpcinfo -p

5. Gluster NFS server log e.g. /var/log/glusterfs/nfs.log

B. From NFS client machine (from where mount is being tried):

1. showmount -e <NFS server machine name/IP addr>

2. mount -vvv -t nfs -o vers=3,tcp 192.168.1.9:/vol1 /media/volume

Comment 6 santosh pradhan 2014-08-01 07:03:07 UTC
Ideally NFS mount (or mount command) should not be able to play with other SSH connections. If it's, then there might be other N/W issues.

BTW, what is the OS (with version) running at client and server machines?

Comment 7 Sahina Bose 2014-08-01 07:06:32 UTC
(In reply to santosh pradhan from comment #6)
> Ideally NFS mount (or mount command) should not be able to play with other
> SSH connections. If it's, then there might be other N/W issues.
> 
> BTW, what is the OS (with version) running at client and server machines?

Client is Fedora 20, and server is RHEL 6.5

Comment 8 Piotr Kliczewski 2014-08-01 07:44:00 UTC
Created attachment 923123 [details]
gluster nfs log

Comment 9 Piotr Kliczewski 2014-08-01 07:47:53 UTC
(In reply to santosh pradhan from comment #5)
> Could you please provide the following data:
> 
> NB: Run the commands as root
> 
> A. m/c where Gluster NFS server is running:
> 
> 1. ps aux |grep gluster
> 
root      2547  0.0  0.0 205884  9028 ?        S    08:41   0:00 python /usr/lib64/nagios/plugins/gluster/check_proc_status.py start
root      5286  0.0  0.0 413832 16164 ?        Ssl  08:56   0:01 /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid
root      5303  0.0  0.1 516820 18884 ?        Ssl  08:56   0:00 /usr/sbin/glusterfsd -s 192.168.1.9 --volfile-id vol1.192.168.1.9.bricks -p /var/lib/glusterd/vols/vol1/run/192.168.1.9-bricks.pid -S /var/run/d516a329f1910e6130e8826f403d58c2.socket --brick-name /bricks -l /var/log/glusterfs/bricks/bricks.log --xlator-option *-posix.glusterd-uuid=928b5be9-6654-477a-aeaf-5520b2cfb9a1 --brick-port 49152 --xlator-option vol1-server.listen-port=49152
root     17461  0.0  0.0 103256   852 pts/0    S+   09:40   0:00 grep gluster
>
> 2. iptables -L
> If firewall is running make sure to turn OFF (iptables -F) 
>
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED 
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:54321 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ssh 
ACCEPT     udp  --  anywhere             anywhere            udp dpt:snmp 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:16514 
ACCEPT     tcp  --  anywhere             anywhere            multiport dports vnc-server:6923 
ACCEPT     tcp  --  anywhere             anywhere            multiport dports 49152:49216 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:24007 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:webcache 
ACCEPT     udp  --  anywhere             anywhere            udp dpt:sunrpc 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:38465 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:38466 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:sunrpc 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:38467 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:nfs 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:38469 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:5666 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:39543 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:55863 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:38468 
ACCEPT     udp  --  anywhere             anywhere            udp dpt:963 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:965 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ctdb 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:netbios-ssn 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:microsoft-ds 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpts:24009:24108 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpts:49152:49251 
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited 

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
REJECT     all  --  anywhere             anywhere            PHYSDEV match ! --physdev-is-bridged reject-with icmp-host-prohibited 

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination          
>
> 3. showmount -e
>
Export list for rhel.localdomain:
/var/lib/exports/iso 0.0.0.0/0.0.0.0
> 
> 4. rpcinfo -p
> 
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp    662  status
    100024    1   tcp    662  status
    100011    1   udp    875  rquotad
    100011    2   udp    875  rquotad
    100011    1   tcp    875  rquotad
    100011    2   tcp    875  rquotad
    100005    1   udp    892  mountd
    100005    1   tcp    892  mountd
    100005    2   udp    892  mountd
    100005    2   tcp    892  mountd
    100005    3   udp    892  mountd
    100005    3   tcp    892  mountd
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    2   tcp   2049  nfs_acl
    100227    3   tcp   2049  nfs_acl
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100227    2   udp   2049  nfs_acl
    100227    3   udp   2049  nfs_acl
    100021    1   udp  32769  nlockmgr
    100021    3   udp  32769  nlockmgr
    100021    4   udp  32769  nlockmgr
    100021    1   tcp  32803  nlockmgr
    100021    3   tcp  32803  nlockmgr
    100021    4   tcp  32803  nlockmgr
>
> 5. Gluster NFS server log e.g. /var/log/glusterfs/nfs.log
> 

Attached

>
> B. From NFS client machine (from where mount is being tried):
> 
> 1. showmount -e <NFS server machine name/IP addr>
>
I had to stop iptables on Gluster machine.

Export list for 192.168.1.9:
/var/lib/exports/iso 0.0.0.0/0.0.0.0

> 
> 2. mount -vvv -t nfs -o vers=3,tcp 192.168.1.9:/vol1 /media/volume

Running wiht iptables stopped on Gluster machine.

mount.nfs: timeout set for Fri Aug  1 09:49:15 2014
mount.nfs: trying text-based options 'vers=3,tcp,addr=192.168.1.9'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.1.9 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 192.168.1.9 prog 100005 vers 3 prot TCP port 892
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting 192.168.1.9:/vol1

Comment 10 santosh pradhan 2014-08-01 12:40:25 UTC
(In reply to Piotr Kliczewski from comment #9)
> (In reply to santosh pradhan from comment #5)
> > Could you please provide the following data:
> > 
> > NB: Run the commands as root
> > 
> > A. m/c where Gluster NFS server is running:
> > 
> > 1. ps aux |grep gluster
> > 
> root      2547  0.0  0.0 205884  9028 ?        S    08:41   0:00 python
> /usr/lib64/nagios/plugins/gluster/check_proc_status.py start
> root      5286  0.0  0.0 413832 16164 ?        Ssl  08:56   0:01
> /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid
> root      5303  0.0  0.1 516820 18884 ?        Ssl  08:56   0:00
> /usr/sbin/glusterfsd -s 192.168.1.9 --volfile-id vol1.192.168.1.9.bricks -p
> /var/lib/glusterd/vols/vol1/run/192.168.1.9-bricks.pid -S
> /var/run/d516a329f1910e6130e8826f403d58c2.socket --brick-name /bricks -l
> /var/log/glusterfs/bricks/bricks.log --xlator-option
> *-posix.glusterd-uuid=928b5be9-6654-477a-aeaf-5520b2cfb9a1 --brick-port
> 49152 --xlator-option vol1-server.listen-port=49152
> root     17461  0.0  0.0 103256   852 pts/0    S+   09:40   0:00 grep gluster
> >


[Santosh] Gluster NFS process is not running. So, Gluster NFS cant be mountable :)

> > 2. iptables -L
> > If firewall is running make sure to turn OFF (iptables -F) 
> >
> Chain INPUT (policy ACCEPT)
> target     prot opt source               destination         
> ACCEPT     all  --  anywhere             anywhere            state
> RELATED,ESTABLISHED 
> ACCEPT     all  --  anywhere             anywhere            
> ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:54321 
> ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ssh 
> ACCEPT     udp  --  anywhere             anywhere            udp dpt:snmp 
> ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:16514 
> ACCEPT     tcp  --  anywhere             anywhere            multiport
> dports vnc-server:6923 
> ACCEPT     tcp  --  anywhere             anywhere            multiport
> dports 49152:49216 
> ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:24007 
> ACCEPT     tcp  --  anywhere             anywhere            tcp
> dpt:webcache 
> ACCEPT     udp  --  anywhere             anywhere            udp dpt:sunrpc 
> ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:38465 
> ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:38466 
> ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:sunrpc 
> ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:38467 
> ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:nfs 
> ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:38469 
> ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:5666 
> ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:39543 
> ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:55863 
> ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:38468 
> ACCEPT     udp  --  anywhere             anywhere            udp dpt:963 
> ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:965 
> ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ctdb 
> ACCEPT     tcp  --  anywhere             anywhere            tcp
> dpt:netbios-ssn 
> ACCEPT     tcp  --  anywhere             anywhere            tcp
> dpt:microsoft-ds 
> ACCEPT     tcp  --  anywhere             anywhere            tcp
> dpts:24009:24108 
> ACCEPT     tcp  --  anywhere             anywhere            tcp
> dpts:49152:49251 
> REJECT     all  --  anywhere             anywhere            reject-with
> icmp-host-prohibited 
> 
> Chain FORWARD (policy ACCEPT)
> target     prot opt source               destination         
> REJECT     all  --  anywhere             anywhere            PHYSDEV match !
> --physdev-is-bridged reject-with icmp-host-prohibited 
> 
> Chain OUTPUT (policy ACCEPT)
> target     prot opt source               destination          
> >


[Santosh] Here, Firewall is running. I had suggested to you to turn it OFF (iptables -F).

> > 3. showmount -e
> >
> Export list for rhel.localdomain:
> /var/lib/exports/iso 0.0.0.0/0.0.0.0

[Santosh] Its coming from kernel NFS not from Gluster NFS. While running Gluster NFS, kernel NFS must be OFF (/etc/init.d/nfs stop or so). Looks like kernel NFS is running.

> > 
> > 4. rpcinfo -p
> > 
>    program vers proto   port  service
>     100000    4   tcp    111  portmapper
>     100000    3   tcp    111  portmapper
>     100000    2   tcp    111  portmapper
>     100000    4   udp    111  portmapper
>     100000    3   udp    111  portmapper
>     100000    2   udp    111  portmapper
>     100024    1   udp    662  status
>     100024    1   tcp    662  status
>     100011    1   udp    875  rquotad
>     100011    2   udp    875  rquotad
>     100011    1   tcp    875  rquotad
>     100011    2   tcp    875  rquotad
>     100005    1   udp    892  mountd
>     100005    1   tcp    892  mountd
>     100005    2   udp    892  mountd
>     100005    2   tcp    892  mountd
>     100005    3   udp    892  mountd
>     100005    3   tcp    892  mountd
>     100003    2   tcp   2049  nfs
>     100003    3   tcp   2049  nfs
>     100003    4   tcp   2049  nfs
>     100227    2   tcp   2049  nfs_acl
>     100227    3   tcp   2049  nfs_acl
>     100003    2   udp   2049  nfs
>     100003    3   udp   2049  nfs
>     100003    4   udp   2049  nfs
>     100227    2   udp   2049  nfs_acl
>     100227    3   udp   2049  nfs_acl
>     100021    1   udp  32769  nlockmgr
>     100021    3   udp  32769  nlockmgr
>     100021    4   udp  32769  nlockmgr
>     100021    1   tcp  32803  nlockmgr
>     100021    3   tcp  32803  nlockmgr
>     100021    4   tcp  32803  nlockmgr
> >


[Santosh] Its confirmed that kernel NFS is running in the machine.


> > 5. Gluster NFS server log e.g. /var/log/glusterfs/nfs.log
> > 

[Santosh]

[2014-08-01 06:56:51.870503] E [nfs.c:332:nfs_init_versions] 0-nfs: Program  MOUNT3 registration failed
[2014-08-01 06:56:51.870512] E [nfs.c:1312:init] 0-nfs: Failed to initialize protocols
[2014-08-01 06:56:51.870520] E [xlator.c:403:xlator_init] 0-nfs-server: Initialization of volume 'nfs-server' failed, review your volfile again
[2014-08-01 06:56:51.870526] E [graph.c:307:glusterfs_graph_init] 0-nfs-server: initializing translator failed
[2014-08-01 06:56:51.870532] E [graph.c:502:glusterfs_graph_activate] 0-graph: init failed
[2014-08-01 06:56:51.870731] W [glusterfsd.c:1095:cleanup_and_exit] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x335c60e6d5] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x320) [0x40bd50] (-->/usr/sbin/glusterfs(glusterfs_process_volfp+0x106) [0x405146]))) 0-: received signum (0), shutting down


Gluster NFS is not able to register with rpcbind because ports are already used by kernel NFS.

So, its not a bug. I ll be closing it.

Suggestions:
1. Turn kernel NFS service off.
2. Turn iptables OFF (iptables -F).


Let me know if you still face any issue.

Thanks,
Santosh


Note You need to log in before you can comment on or make changes to this bug.