Bug 763815 (GLUSTER-2083)

[2010-11-10 16:52:40.622842] E [client-handshake.c:1067:client_query_portmap_cbk
] dist-client-1: failed to get the port number for remote subvolume
[2010-11-10 16:52:43.628288] E [client-handshake.c:1067:client_query_portmap_cbk] dist-client-1: failed to get the port number for remote subvolume
[2010-11-10 16:52:46.633742] E [client-handshake.c:1067:client_query_portmap_cbk] dist-client-1: failed to get the port number for remote subvolume
[2010-11-10 16:52:49.638282] E [client-handshake.c:1067:client_query_portmap_cbk] dist-client-1: failed to get the port number for remote subvolume
[2010-11-10 16:52:52.643817] E [client-handshake.c:1067:client_query_portmap_cbk] dist-client-1: failed to get the port number for remote subvolume
[2010-11-10 16:52:55.648590] E [client-handshake.c:1067:client_query_portmap_cbk] dist-client-1: failed to get the port number for remote subvolume
[2010-11-10 16:52:58.653951] E [client-handshake.c:1067:client_query_portmap_cbk] dist-client-1: failed to get the port number for remote subvolume
[2010-11-10 16:53:01.661029] E [client-handshake.c:1067:client_query_portmap_cbk] dist-client-1: failed to get the port number for remote subvolume
[2010-11-10 16:53:04.667193] E [client-handshake.c:1067:client_query_portmap_cbk] dist-client-1: failed to get the port number for remote subvolume
[2010-11-10 16:53:07.672639] E [client-handshake.c:1067:client_query_portmap_cbk] dist-client-1: failed to get the port number for remote subvolume
[2010-11-10 16:53:10.677231] E [client-handshake.c:1067:client_query_portmap_cbk] dist-client-1: failed to get the port number for remote subvolume
[2010-11-10 16:53:13.682807] E [client-handshake.c:1067:client_query_portmap_cbk] dist-client-1: failed to get the port number for remote subvolume
[2010-11-10 16:53:16.692675] E [client-handshake.c:1067:client_query_portmap_cbk] dist-client-1: failed


Process to reproduce 


[root@compel1 ~]# gluster volume info 

Volume Name: dist
Type: Distribute
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: compel1:/export1
Brick2: compel4:/export1


copy some 4000 files in the volume

gluster volume replace-brick dist compel4:/export1 compel1:/export2 start

gluster volume replace-brick dist compel4:/export1 compel1:/export2 status 

Migration is progress...  message with number of files once it is finished. 

gluster volume replace-brick dist compel4;/export1 compel1:/export2 commit

All files are properly migrated checked with the backend. 

[root@compel4 ~]# gluster volume info dist

Volume Name: dist
Type: Distribute
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: compel1:/export1
Brick2: compel1:/export2

Now "showmount -e compel1" results in 

[root@compel4 ~]# showmount -e compel1
mount clntudp_create: RPC: Program not registered
[root@compel4 ~]# mount compel1:/dist /mnt
mount: mount to NFS server 'compel1' failed: System Error: Connection refused.
[root@compel4 ~]# 


Looks like NFS is never restarted and fails since the running NFS server has still old config as shown below

"
+------------------------------------------------------------------------------+
  1: volume dist-client-0
  2:     type protocol/client
  3:     option remote-host compel1
  4:     option remote-subvolume /export1
  5:     option transport-type tcp
  6: end-volume
  7: 
  8: volume dist-client-1
  9:     type protocol/client
 10:     option remote-host compel4
 11:     option remote-subvolume /export1
 12:     option transport-type tcp
 13: end-volume
 14: 
 15: volume dist-dht
 16:     type cluster/distribute
 17:     subvolumes dist-client-0 dist-client-1
 18: end-volume
 19: 
 20: volume dist-write-behind
 21:     type performance/write-behind
 22:     subvolumes dist-dht
 23: end-volume
 24: 
 25: volume dist-read-ahead
 26:     type performance/read-ahead
 27:     subvolumes dist-write-behind
 28: end-volume
 29: 
 30: volume dist-io-cache
 31:     type performance/io-cache
 32:     subvolumes dist-read-ahead
 33: end-volume

"

(In reply to comment #1)
> A volume stop and start makes it working again.

hi Harsha,
      I tried the steps provided in the bug description. It is not happening. Could you please let me know if I missed anything?.

pranith @ ~/workspace/1repo
15:28:03 :) $ sudo gluster volume info

Volume Name: pranith
Type: Distribute
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: pranith-laptop:/tmp/3
Brick2: pranith-laptop:/tmp/1

pranith @ ~/workspace/1repo
15:28:10 :) $ showmount -e localhost
Export list for localhost:
/pranith *

pranith @ ~/workspace/1repo
15:28:19 :) $ sudo gluster volume replace-brick pranith pranith-laptop:/tmp/1 pranith-laptop:/tmp/2 start
replace-brick started successfully

pranith @ ~/workspace/1repo
15:29:38 :) $ sleep 5 ; sudo gluster volume replace-brick pranith pranith-laptop:/tmp/1 pranith-laptop:/tmp/2 status
Number of files migrated = 4000        Migration complete 

pranith @ ~/workspace/1repo
15:30:21 :) $ ls /tmp/2

pranith @ ~/workspace/1repo
15:30:26 :) $ git pull
Already up-to-date.

pranith @ ~/workspace/1repo
15:32:09 :) $ sleep 5 ; sudo gluster volume replace-brick pranith pranith-laptop:/tmp/1 pranith-laptop:/tmp/2 commit
replace-brick commit successful

pranith @ ~/workspace/1repo
15:34:02 :) $ ls /tmp/2

pranith @ ~/workspace/1repo
15:34:06 :) $ showmount -e localhost
Export list for localhost:
/pranith *

(In reply to comment #2)
> (In reply to comment #1)
> > A volume stop and start makes it working again.
> 
> hi Harsha,
>       I tried the steps provided in the bug description. It is not happening.
> Could you please let me know if I missed anything?.
> 
> pranith @ ~/workspace/1repo
> 15:28:03 :) $ sudo gluster volume info
> 

This has been working since 3.1.1 it was against 3.1.1qa5, if you not able to reproduce then it is relinquished to close.

I think Pranith's patch to de-register ports through glusterd fixed this problem.

(In reply to comment #4)
> I think Pranith's patch to de-register ports through glusterd fixed this
> problem.

Shehjar,
     that patch is not yet accepted. Its working fine even without that patch.

Pranith.