763515 – (GLUSTER-1783) kill glusterd and stop the cluster and start again - check nfs process

Bug 763515 (GLUSTER-1783) - kill glusterd and stop the cluster and start again - check nfs process

Summary: kill glusterd and stop the cluster and start again - check nfs process

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	GLUSTER-1783
Product:	GlusterFS
Classification:	Community
Component:	glusterd
Sub Component:
Version:	3.1-alpha
Hardware:	All
OS:	Linux
Priority:	low
Severity:	high
Target Milestone:	---
Assignee:	Pranith Kumar K
QA Contact:
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2010-10-03 14:30 UTC by Lakshmipathi G
Modified:	2015-12-01 16:45 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:
Regression:	RTP
Mount Type:	All
Documentation:	---
CRM:
Verified Versions:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description Lakshmipathi G 2010-10-03 14:30:40 UTC

started 2dht cluster  and killed glusterd and stop the cluster and started it again. but showmount says "showmount: RPC: Program not registered".


10.192.141.187# ps aux | grep gluster
root     19264  0.3  0.1  64704 11688 ?        Ssl  10:22   0:00 glusterd
root     19284  0.0  0.1  82748  9960 ?        Ssl  10:25   0:00 /usr/local/sbin/glusterfsd --xlator-option dht39-server.listen-port=6971 -s localhost --volfile-id dht39.10.192.141.187.mnt-oct3 -p /etc/glusterd/vols/dht39/run/10.192.141.187-mnt-oct3.pid --brick-name /mnt/oct3 --brick-port 6971 -l /etc/glusterd/logs/bricks/mnt-oct3.log
root     19288  1.6  0.7 126900 61544 ?        Ssl  10:25   0:00 /usr/local/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p /etc/glusterd/nfs/run/nfs.pid -l /etc/glusterd/logs/nfs.log


10.192.141.187#showmount -e localhost
Export list for localhost:
/dht39 *


10.192.141.187#killall glusterd


10.192.141.187# ps aux | grep gluster
root     19284  0.2  0.7 131844 59372 ?        Ssl  10:25   0:00 /usr/local/sbin/glusterfsd --xlator-option dht39-server.listen-port=6971 -s localhost --volfile-id dht39.10.192.141.187.mnt-oct3 -p /etc/glusterd/vols/dht39/run/10.192.141.187-mnt-oct3.pid --brick-name /mnt/oct3 --brick-port 6971 -l /etc/glusterd/logs/bricks/mnt-oct3.log
root     19288  0.2  0.8 128964 63784 ?        Ssl  10:25   0:00 /usr/local/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p /etc/glusterd/nfs/run/nfs.pid -l /etc/glusterd/logs/nfs.log

10.192.141.187#showmount -e localhost
Export list for localhost:
/dht39 *

10.192.141.187#glusterd
10.192.141.187#gluster volume stop dht39
Stopping volume will make its data inaccessible. Do you want to Continue? (y/n) y
Stopping volume dht39 has been successful

10.192.141.187#gluster volume start dht39
Starting volume dht39 has been successful

10.192.141.187# ps aux | grep gluster
root     19284  0.0  0.7 131844 59376 ?        Ssl  10:25   0:00 /usr/local/sbin/glusterfsd --xlator-option dht39-server.listen-port=6971 -s localhost --volfile-id dht39.10.192.141.187.mnt-oct3 -p /etc/glusterd/vols/dht39/run/10.192.141.187-mnt-oct3.pid --brick-name /mnt/oct3 --brick-port 6971 -l /etc/glusterd/logs/bricks/mnt-oct3.log
root     19303  1.9  0.1  64704 11712 ?        Ssl  10:25   0:00 glusterd
root     19326  1.2  0.7 126904 61548 ?        Ssl  10:25   0:00 /usr/local/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p /etc/glusterd/nfs/run/nfs.pid -l /etc/glusterd/logs/nfs.log
root     19331  0.0  0.0   6060   608 pts/0    S+   10:26   0:00 grep gluster


showmount: RPC: Program not registered
10.192.141.187#showmount -e localhost
showmount: RPC: Program not registered
10.192.141.187#showmount -e localhost        
showmount: RPC: Program not registered
10.192.141.187#showmount -e localhost
showmount: RPC: Program not registered
10.192.141.187#showmount -e localhost
showmount: RPC: Program not registered
10.192.141.187#showmount -e localhost
showmount: RPC: Program not registered
10.192.141.187#showmount -e localhost

============
#cat /etc/glusterd/logs/nfs.log
[2010-10-03 10:25:11.324443] I [nfs.c:769:init] nfs: NFS service started
[2010-10-03 10:25:11.324617] C [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-10-03 10:25:11.324635] C [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-10-03 10:25:11.334955] W [graph.c:120:_log_if_option_is_invalid] nfs-server: option 'nfs3.dht39.volume-id' is not recognized
Given volfile:
+------------------------------------------------------------------------------+
  1: volume dht39-client-0
  2:     type protocol/client
  3:     option remote-host 10.192.134.144
  4:     option remote-subvolume /mnt/oct3
  5:     option transport-type tcp
  6: end-volume
  7: 
  8: volume dht39-client-1
  9:     type protocol/client
 10:     option remote-host 10.192.141.187
 11:     option remote-subvolume /mnt/oct3
 12:     option transport-type tcp
 13: end-volume
 14: 
 15: volume dht39-dht
 16:     type cluster/distribute
 17:     subvolumes dht39-client-0 dht39-client-1
 18: end-volume
 19: 
 20: volume dht39-write-behind
 21:     type performance/write-behind
 22:     subvolumes dht39-dht
 23: end-volume
 24: 
 25: volume dht39-read-ahead
 26:     type performance/read-ahead
 27:     subvolumes dht39-write-behind
 28: end-volume
 29: 
 30: volume dht39-io-cache
 31:     type performance/io-cache
 32:     subvolumes dht39-read-ahead
 33: end-volume
 34: 
 35: volume dht39-quick-read
 36:     type performance/quick-read
 37:     subvolumes dht39-io-cache
 38: end-volume
 39: 
 40: volume dht39
 41:     type debug/io-stats
 42:     subvolumes dht39-quick-read
 43: end-volume
 44: 
 45: volume nfs-server
 46:     type nfs/server
 47:     option nfs.dynamic-volumes on
 48:     option rpc-auth.addr.dht39.allow *
 49:     option nfs3.dht39.volume-id 0c5ddadf-ff7e-4574-9166-c0ef0bc17f17
 50:     subvolumes dht39
 51: end-volume

+------------------------------------------------------------------------------+
[2010-10-03 10:25:12.2888] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-0: failed to get the port number for remote subvolume
[2010-10-03 10:25:14.340017] I [client-handshake.c:698:select_server_supported_programs] dht39-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-10-03 10:25:14.413888] I [client-handshake.c:534:client_setvolume_cbk] dht39-client-1: Connected to 10.192.141.187:6971, attached to remote volume '/mnt/oct3'.
[2010-10-03 10:25:18.348457] I [client-handshake.c:698:select_server_supported_programs] dht39-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-10-03 10:25:18.348999] I [client-handshake.c:534:client_setvolume_cbk] dht39-client-0: Connected to 10.192.134.144:6971, attached to remote volume '/mnt/oct3'.
[2010-10-03 10:25:18.360772] I [nfs.c:315:__nfs_subvolume_start] nfs: All exports up
[2010-10-03 10:25:44.484776] I [glusterfsd.c:668:cleanup_and_exit] glusterfsd: shutting down
[2010-10-03 10:25:44.485102] I [io-stats.c:1683:fini] dht39: io-stats translator unloaded
[2010-10-03 10:25:45.612066] I [nfs.c:769:init] nfs: NFS service started
[2010-10-03 10:25:45.612202] C [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-10-03 10:25:45.612220] C [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-10-03 10:25:45.631280] W [graph.c:120:_log_if_option_is_invalid] nfs-server: option 'nfs3.dht39.volume-id' is not recognized
Given volfile:
+------------------------------------------------------------------------------+
  1: volume dht39-client-0
  2:     type protocol/client
  3:     option remote-host 10.192.134.144
  4:     option remote-subvolume /mnt/oct3
  5:     option transport-type tcp
  6: end-volume
  7: 
  8: volume dht39-client-1
  9:     type protocol/client
 10:     option remote-host 10.192.141.187
 11:     option remote-subvolume /mnt/oct3
 12:     option transport-type tcp
 13: end-volume
 14: 
 15: volume dht39-dht
 16:     type cluster/distribute
 17:     subvolumes dht39-client-0 dht39-client-1
 18: end-volume
 19: 
 20: volume dht39-write-behind
 21:     type performance/write-behind
 22:     subvolumes dht39-dht
 23: end-volume
 24: 
 25: volume dht39-read-ahead
 26:     type performance/read-ahead
 27:     subvolumes dht39-write-behind
 28: end-volume
 29: 
 30: volume dht39-io-cache
 31:     type performance/io-cache
 32:     subvolumes dht39-read-ahead
 33: end-volume
 34: 
 35: volume dht39-quick-read
 36:     type performance/quick-read
 37:     subvolumes dht39-io-cache
 38: end-volume
 39: 
 40: volume dht39
 41:     type debug/io-stats
 42:     subvolumes dht39-quick-read
 43: end-volume
 44: 
 45: volume nfs-server
 46:     type nfs/server
 47:     option nfs.dynamic-volumes on
 48:     option rpc-auth.addr.dht39.allow *
 49:     option nfs3.dht39.volume-id 0c5ddadf-ff7e-4574-9166-c0ef0bc17f17
 50:     subvolumes dht39
 51: end-volume

+------------------------------------------------------------------------------+
[2010-10-03 10:25:45.632060] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:25:48.636896] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:25:48.639881] I [client-handshake.c:698:select_server_supported_programs] dht39-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-10-03 10:25:48.640452] I [client-handshake.c:534:client_setvolume_cbk] dht39-client-0: Connected to 10.192.134.144:6971, attached to remote volume '/mnt/oct3'.
[2010-10-03 10:25:50.169981] I [glusterfsd.c:668:cleanup_and_exit] glusterfsd: shutting down
[2010-10-03 10:25:50.170073] I [io-stats.c:1683:fini] dht39: io-stats translator unloaded
[2010-10-03 10:25:58.343791] I [nfs.c:769:init] nfs: NFS service started
[2010-10-03 10:25:58.343926] C [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-10-03 10:25:58.343944] C [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-10-03 10:25:58.354070] W [graph.c:120:_log_if_option_is_invalid] nfs-server: option 'nfs3.dht39.volume-id' is not recognized
Given volfile:
+------------------------------------------------------------------------------+
  1: volume dht39-client-0
  2:     type protocol/client
  3:     option remote-host 10.192.134.144
  4:     option remote-subvolume /mnt/oct3
  5:     option transport-type tcp
  6: end-volume
  7: 
  8: volume dht39-client-1
  9:     type protocol/client
 10:     option remote-host 10.192.141.187
 11:     option remote-subvolume /mnt/oct3
 12:     option transport-type tcp
 13: end-volume
 14: 
 15: volume dht39-dht
 16:     type cluster/distribute
 17:     subvolumes dht39-client-0 dht39-client-1
 18: end-volume
 19: 
 20: volume dht39-write-behind
 21:     type performance/write-behind
 22:     subvolumes dht39-dht
 23: end-volume
 24: 
 25: volume dht39-read-ahead
 26:     type performance/read-ahead
 27:     subvolumes dht39-write-behind
 28: end-volume
 29: 
 30: volume dht39-io-cache
 31:     type performance/io-cache
 32:     subvolumes dht39-read-ahead
 33: end-volume
 34: 
 35: volume dht39-quick-read
 36:     type performance/quick-read
 37:     subvolumes dht39-io-cache
 38: end-volume
 39: 
 40: volume dht39
 41:     type debug/io-stats
 42:     subvolumes dht39-quick-read
 43: end-volume
 44: 
 45: volume nfs-server
 46:     type nfs/server
 47:     option nfs.dynamic-volumes on
 48:     option rpc-auth.addr.dht39.allow *
 49:     option nfs3.dht39.volume-id 0c5ddadf-ff7e-4574-9166-c0ef0bc17f17
 50:     subvolumes dht39
 51: end-volume

+------------------------------------------------------------------------------+
[2010-10-03 10:25:58.359320] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:25:58.482345] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-0: failed to get the port number for remote subvolume
[2010-10-03 10:26:02.362737] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:05.367097] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:06.370953] I [client-handshake.c:698:select_server_supported_programs] dht39-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-10-03 10:26:06.371510] I [client-handshake.c:534:client_setvolume_cbk] dht39-client-0: Connected to 10.192.134.144:6971, attached to remote volume '/mnt/oct3'.
[2010-10-03 10:26:08.373789] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:11.378346] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:14.382875] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:17.387424] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:20.391937] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:23.396417] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:26.401106] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:29.405602] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:32.410172] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:35.414722] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:38.419225] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:41.423785] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:44.428160] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:47.432879] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:50.437482] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:53.441725] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:56.446549] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:59.451045] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:02.455605] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:05.460054] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:08.464647] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:11.469263] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:14.473757] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:17.478282] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:20.482884] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:23.487226] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:26.491936] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:29.496374] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:32.501022] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:35.505409] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:38.510246] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
10.192.141.187#

Comment 1 Anand Avati 2011-01-24 19:19:42 UTC

PATCH: http://patches.gluster.com/patch/5911 in master (glusterd: de-register nfs rpcs when it is stopped)

Comment 2 Pranith Kumar K 2011-01-25 00:57:56 UTC

added the code to de-register nfs rpcs as soon as the process is stopped.

Note You need to log in before you can comment on or make changes to this bug.