This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 763515 - (GLUSTER-1783) kill glusterd and stop the cluster and start again - check nfs process
kill glusterd and stop the cluster and start again - check nfs process
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: glusterd (Show other bugs)
3.1-alpha
All Linux
low Severity high
: ---
: ---
Assigned To: Pranith Kumar K
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-10-03 10:30 EDT by Lakshmipathi G
Modified: 2015-12-01 11:45 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: ---
Regression: RTP
Mount Type: All
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Lakshmipathi G 2010-10-03 10:30:40 EDT
started 2dht cluster  and killed glusterd and stop the cluster and started it again. but showmount says "showmount: RPC: Program not registered".


10.192.141.187# ps aux | grep gluster
root     19264  0.3  0.1  64704 11688 ?        Ssl  10:22   0:00 glusterd
root     19284  0.0  0.1  82748  9960 ?        Ssl  10:25   0:00 /usr/local/sbin/glusterfsd --xlator-option dht39-server.listen-port=6971 -s localhost --volfile-id dht39.10.192.141.187.mnt-oct3 -p /etc/glusterd/vols/dht39/run/10.192.141.187-mnt-oct3.pid --brick-name /mnt/oct3 --brick-port 6971 -l /etc/glusterd/logs/bricks/mnt-oct3.log
root     19288  1.6  0.7 126900 61544 ?        Ssl  10:25   0:00 /usr/local/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p /etc/glusterd/nfs/run/nfs.pid -l /etc/glusterd/logs/nfs.log


10.192.141.187#showmount -e localhost
Export list for localhost:
/dht39 *


10.192.141.187#killall glusterd


10.192.141.187# ps aux | grep gluster
root     19284  0.2  0.7 131844 59372 ?        Ssl  10:25   0:00 /usr/local/sbin/glusterfsd --xlator-option dht39-server.listen-port=6971 -s localhost --volfile-id dht39.10.192.141.187.mnt-oct3 -p /etc/glusterd/vols/dht39/run/10.192.141.187-mnt-oct3.pid --brick-name /mnt/oct3 --brick-port 6971 -l /etc/glusterd/logs/bricks/mnt-oct3.log
root     19288  0.2  0.8 128964 63784 ?        Ssl  10:25   0:00 /usr/local/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p /etc/glusterd/nfs/run/nfs.pid -l /etc/glusterd/logs/nfs.log

10.192.141.187#showmount -e localhost
Export list for localhost:
/dht39 *

10.192.141.187#glusterd
10.192.141.187#gluster volume stop dht39
Stopping volume will make its data inaccessible. Do you want to Continue? (y/n) y
Stopping volume dht39 has been successful

10.192.141.187#gluster volume start dht39
Starting volume dht39 has been successful

10.192.141.187# ps aux | grep gluster
root     19284  0.0  0.7 131844 59376 ?        Ssl  10:25   0:00 /usr/local/sbin/glusterfsd --xlator-option dht39-server.listen-port=6971 -s localhost --volfile-id dht39.10.192.141.187.mnt-oct3 -p /etc/glusterd/vols/dht39/run/10.192.141.187-mnt-oct3.pid --brick-name /mnt/oct3 --brick-port 6971 -l /etc/glusterd/logs/bricks/mnt-oct3.log
root     19303  1.9  0.1  64704 11712 ?        Ssl  10:25   0:00 glusterd
root     19326  1.2  0.7 126904 61548 ?        Ssl  10:25   0:00 /usr/local/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p /etc/glusterd/nfs/run/nfs.pid -l /etc/glusterd/logs/nfs.log
root     19331  0.0  0.0   6060   608 pts/0    S+   10:26   0:00 grep gluster


showmount: RPC: Program not registered
10.192.141.187#showmount -e localhost
showmount: RPC: Program not registered
10.192.141.187#showmount -e localhost        
showmount: RPC: Program not registered
10.192.141.187#showmount -e localhost
showmount: RPC: Program not registered
10.192.141.187#showmount -e localhost
showmount: RPC: Program not registered
10.192.141.187#showmount -e localhost
showmount: RPC: Program not registered
10.192.141.187#showmount -e localhost

============
#cat /etc/glusterd/logs/nfs.log
[2010-10-03 10:25:11.324443] I [nfs.c:769:init] nfs: NFS service started
[2010-10-03 10:25:11.324617] C [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-10-03 10:25:11.324635] C [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-10-03 10:25:11.334955] W [graph.c:120:_log_if_option_is_invalid] nfs-server: option 'nfs3.dht39.volume-id' is not recognized
Given volfile:
+------------------------------------------------------------------------------+
  1: volume dht39-client-0
  2:     type protocol/client
  3:     option remote-host 10.192.134.144
  4:     option remote-subvolume /mnt/oct3
  5:     option transport-type tcp
  6: end-volume
  7: 
  8: volume dht39-client-1
  9:     type protocol/client
 10:     option remote-host 10.192.141.187
 11:     option remote-subvolume /mnt/oct3
 12:     option transport-type tcp
 13: end-volume
 14: 
 15: volume dht39-dht
 16:     type cluster/distribute
 17:     subvolumes dht39-client-0 dht39-client-1
 18: end-volume
 19: 
 20: volume dht39-write-behind
 21:     type performance/write-behind
 22:     subvolumes dht39-dht
 23: end-volume
 24: 
 25: volume dht39-read-ahead
 26:     type performance/read-ahead
 27:     subvolumes dht39-write-behind
 28: end-volume
 29: 
 30: volume dht39-io-cache
 31:     type performance/io-cache
 32:     subvolumes dht39-read-ahead
 33: end-volume
 34: 
 35: volume dht39-quick-read
 36:     type performance/quick-read
 37:     subvolumes dht39-io-cache
 38: end-volume
 39: 
 40: volume dht39
 41:     type debug/io-stats
 42:     subvolumes dht39-quick-read
 43: end-volume
 44: 
 45: volume nfs-server
 46:     type nfs/server
 47:     option nfs.dynamic-volumes on
 48:     option rpc-auth.addr.dht39.allow *
 49:     option nfs3.dht39.volume-id 0c5ddadf-ff7e-4574-9166-c0ef0bc17f17
 50:     subvolumes dht39
 51: end-volume

+------------------------------------------------------------------------------+
[2010-10-03 10:25:12.2888] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-0: failed to get the port number for remote subvolume
[2010-10-03 10:25:14.340017] I [client-handshake.c:698:select_server_supported_programs] dht39-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-10-03 10:25:14.413888] I [client-handshake.c:534:client_setvolume_cbk] dht39-client-1: Connected to 10.192.141.187:6971, attached to remote volume '/mnt/oct3'.
[2010-10-03 10:25:18.348457] I [client-handshake.c:698:select_server_supported_programs] dht39-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-10-03 10:25:18.348999] I [client-handshake.c:534:client_setvolume_cbk] dht39-client-0: Connected to 10.192.134.144:6971, attached to remote volume '/mnt/oct3'.
[2010-10-03 10:25:18.360772] I [nfs.c:315:__nfs_subvolume_start] nfs: All exports up
[2010-10-03 10:25:44.484776] I [glusterfsd.c:668:cleanup_and_exit] glusterfsd: shutting down
[2010-10-03 10:25:44.485102] I [io-stats.c:1683:fini] dht39: io-stats translator unloaded
[2010-10-03 10:25:45.612066] I [nfs.c:769:init] nfs: NFS service started
[2010-10-03 10:25:45.612202] C [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-10-03 10:25:45.612220] C [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-10-03 10:25:45.631280] W [graph.c:120:_log_if_option_is_invalid] nfs-server: option 'nfs3.dht39.volume-id' is not recognized
Given volfile:
+------------------------------------------------------------------------------+
  1: volume dht39-client-0
  2:     type protocol/client
  3:     option remote-host 10.192.134.144
  4:     option remote-subvolume /mnt/oct3
  5:     option transport-type tcp
  6: end-volume
  7: 
  8: volume dht39-client-1
  9:     type protocol/client
 10:     option remote-host 10.192.141.187
 11:     option remote-subvolume /mnt/oct3
 12:     option transport-type tcp
 13: end-volume
 14: 
 15: volume dht39-dht
 16:     type cluster/distribute
 17:     subvolumes dht39-client-0 dht39-client-1
 18: end-volume
 19: 
 20: volume dht39-write-behind
 21:     type performance/write-behind
 22:     subvolumes dht39-dht
 23: end-volume
 24: 
 25: volume dht39-read-ahead
 26:     type performance/read-ahead
 27:     subvolumes dht39-write-behind
 28: end-volume
 29: 
 30: volume dht39-io-cache
 31:     type performance/io-cache
 32:     subvolumes dht39-read-ahead
 33: end-volume
 34: 
 35: volume dht39-quick-read
 36:     type performance/quick-read
 37:     subvolumes dht39-io-cache
 38: end-volume
 39: 
 40: volume dht39
 41:     type debug/io-stats
 42:     subvolumes dht39-quick-read
 43: end-volume
 44: 
 45: volume nfs-server
 46:     type nfs/server
 47:     option nfs.dynamic-volumes on
 48:     option rpc-auth.addr.dht39.allow *
 49:     option nfs3.dht39.volume-id 0c5ddadf-ff7e-4574-9166-c0ef0bc17f17
 50:     subvolumes dht39
 51: end-volume

+------------------------------------------------------------------------------+
[2010-10-03 10:25:45.632060] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:25:48.636896] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:25:48.639881] I [client-handshake.c:698:select_server_supported_programs] dht39-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-10-03 10:25:48.640452] I [client-handshake.c:534:client_setvolume_cbk] dht39-client-0: Connected to 10.192.134.144:6971, attached to remote volume '/mnt/oct3'.
[2010-10-03 10:25:50.169981] I [glusterfsd.c:668:cleanup_and_exit] glusterfsd: shutting down
[2010-10-03 10:25:50.170073] I [io-stats.c:1683:fini] dht39: io-stats translator unloaded
[2010-10-03 10:25:58.343791] I [nfs.c:769:init] nfs: NFS service started
[2010-10-03 10:25:58.343926] C [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-10-03 10:25:58.343944] C [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-10-03 10:25:58.354070] W [graph.c:120:_log_if_option_is_invalid] nfs-server: option 'nfs3.dht39.volume-id' is not recognized
Given volfile:
+------------------------------------------------------------------------------+
  1: volume dht39-client-0
  2:     type protocol/client
  3:     option remote-host 10.192.134.144
  4:     option remote-subvolume /mnt/oct3
  5:     option transport-type tcp
  6: end-volume
  7: 
  8: volume dht39-client-1
  9:     type protocol/client
 10:     option remote-host 10.192.141.187
 11:     option remote-subvolume /mnt/oct3
 12:     option transport-type tcp
 13: end-volume
 14: 
 15: volume dht39-dht
 16:     type cluster/distribute
 17:     subvolumes dht39-client-0 dht39-client-1
 18: end-volume
 19: 
 20: volume dht39-write-behind
 21:     type performance/write-behind
 22:     subvolumes dht39-dht
 23: end-volume
 24: 
 25: volume dht39-read-ahead
 26:     type performance/read-ahead
 27:     subvolumes dht39-write-behind
 28: end-volume
 29: 
 30: volume dht39-io-cache
 31:     type performance/io-cache
 32:     subvolumes dht39-read-ahead
 33: end-volume
 34: 
 35: volume dht39-quick-read
 36:     type performance/quick-read
 37:     subvolumes dht39-io-cache
 38: end-volume
 39: 
 40: volume dht39
 41:     type debug/io-stats
 42:     subvolumes dht39-quick-read
 43: end-volume
 44: 
 45: volume nfs-server
 46:     type nfs/server
 47:     option nfs.dynamic-volumes on
 48:     option rpc-auth.addr.dht39.allow *
 49:     option nfs3.dht39.volume-id 0c5ddadf-ff7e-4574-9166-c0ef0bc17f17
 50:     subvolumes dht39
 51: end-volume

+------------------------------------------------------------------------------+
[2010-10-03 10:25:58.359320] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:25:58.482345] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-0: failed to get the port number for remote subvolume
[2010-10-03 10:26:02.362737] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:05.367097] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:06.370953] I [client-handshake.c:698:select_server_supported_programs] dht39-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-10-03 10:26:06.371510] I [client-handshake.c:534:client_setvolume_cbk] dht39-client-0: Connected to 10.192.134.144:6971, attached to remote volume '/mnt/oct3'.
[2010-10-03 10:26:08.373789] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:11.378346] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:14.382875] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:17.387424] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:20.391937] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:23.396417] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:26.401106] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:29.405602] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:32.410172] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:35.414722] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:38.419225] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:41.423785] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:44.428160] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:47.432879] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:50.437482] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:53.441725] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:56.446549] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:26:59.451045] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:02.455605] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:05.460054] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:08.464647] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:11.469263] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:14.473757] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:17.478282] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:20.482884] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:23.487226] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:26.491936] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:29.496374] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:32.501022] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:35.505409] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
[2010-10-03 10:27:38.510246] E [client-handshake.c:772:client_query_portmap_cbk] dht39-client-1: failed to get the port number for remote subvolume
10.192.141.187#
Comment 1 Anand Avati 2011-01-24 14:19:42 EST
PATCH: http://patches.gluster.com/patch/5911 in master (glusterd: de-register nfs rpcs when it is stopped)
Comment 2 Pranith Kumar K 2011-01-24 19:57:56 EST
added the code to de-register nfs rpcs as soon as the process is stopped.

Note You need to log in before you can comment on or make changes to this bug.