Bug 849122

Summary: Client complains on non existent server running on port 24008
Product: Red Hat Gluster Storage Reporter: Vidya Sakar <vinaraya>
Component: glusterfs-rdmaAssignee: Raghavendra G <rgowdapp>
Status: CLOSED DUPLICATE QA Contact: Sudhir D <sdharane>
Severity: medium Docs Contact:
Priority: low    
Version: 2.0CC: aavati, amarts, andrei, fharshav, gluster-bugs, mozes, rfortier, vijay
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: GLUSTER-3605 Environment:
Last Closed: 2013-01-24 05:19:44 EST Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Bug Depends On: 765337    
Bug Blocks:    

Description Vidya Sakar 2012-08-17 07:41:51 EDT
+++ This bug was initially created as a clone of Bug #765337 +++

Server is running on port 24010. A client mount strangely defaults to 24008 all the time until for a while. 

While during this time i forced remote-port to 24010 by writing a new volfile. 

After a umount and a remount fetching the file from server made client start connecting on 24010. 

Seems like RDMA internal RPC exchange causes a dummy port to be listened. So after the first RPC exchange mount starts working.

--- Additional comment from raghavendra@gluster.com on 2011-09-21 21:50:50 EDT ---

glusterd rdma transport listens on 24008. Clients first connect to glusterd (through port 24008), fetch the volfile and then connect to the server exporting appropriate brick.

--- Additional comment from fharshav@redhat.com on 2011-09-21 22:00:06 EDT ---

(In reply to comment #1)
> glusterd rdma transport listens on 24008. Clients first connect to glusterd
> (through port 24008), fetch the volfile and then connect to the server
> exporting appropriate brick.

But i was receiving 'Connection Refused' does this mean that the 'glusterd' started over RDMA failed in some ways? 

I have seen that many times Now.

--- Additional comment from vijay@dev.gluster.com on 2011-09-21 22:16:46 EDT ---


> But i was receiving 'Connection Refused' does this mean that the 'glusterd'
> started over RDMA failed in some ways? 
> 

Was glusterd started before the ib modules were loaded?

--- Additional comment from fharshav@redhat.com on 2011-09-21 22:24:41 EDT ---

(In reply to comment #3)
> > But i was receiving 'Connection Refused' does this mean that the 'glusterd'
> > started over RDMA failed in some ways? 
> > 
> 
> Was glusterd started before the ib modules were loaded?

glusterd was installed like a day later after infiniband was configured, this is CentOS 6.0.

--- Additional comment from amarts@redhat.com on 2011-09-21 22:29:59 EDT ---

If rdma is present in the machine while starting glusterd, then it should be listening on 24008. Can you confirm the port is open for listening by 'netstat -ntlp' ?

--- Additional comment from fharshav@redhat.com on 2011-09-21 23:03:39 EDT ---

(In reply to comment #5)
> If rdma is present in the machine while starting glusterd, then it should be
> listening on 24008. Can you confirm the port is open for listening by 'netstat
> -ntlp' ?

From what i remember starting glusterd never showed on netstat 24008, so i had to restart it 2-3 times. 

Then i wrote a new vol file just to connect to the server process from client by specifying remote-port. 

After that i umounted and fetched again from server. This time the client connected to the volume. 

We have seen this at repetitive occurrences on couple of customer sites. 

Hopefully it is reproducible in our labs.

--- Additional comment from amarts@redhat.com on 2011-09-28 00:29:33 EDT ---

Will try to reproduce in our labs and update you. But would take some time as we have demand for machines with IB.
Comment 2 Raghavendra G 2013-01-24 05:19:44 EST

*** This bug has been marked as a duplicate of bug 878883 ***
Comment 3 Vijay Bellur 2013-02-05 20:58:48 EST
CHANGE: http://review.gluster.org/4384 (rpc-transport/rdma: use 24008 as default listen port.) merged in release-3.3 by Anand Avati (avati@redhat.com)
Comment 4 Raghavendra G 2013-02-05 21:54:21 EST
*** Bug 907695 has been marked as a duplicate of this bug. ***