763481 – (GLUSTER-1749) adding second brick causes iozone to exit

Bug 763481 (GLUSTER-1749) - adding second brick causes iozone to exit

Summary: adding second brick causes iozone to exit

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	GLUSTER-1749
Product:	GlusterFS
Classification:	Community
Component:	distribute
Sub Component:
Version:	3.1-alpha
Hardware:	All
OS:	Linux
Priority:	low
Severity:	medium
Target Milestone:	---
Assignee:	shishir gowda
QA Contact:
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2010-09-30 11:03 UTC by Lakshmipathi G
Modified:	2015-12-01 16:45 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:
Regression:	RTP
Mount Type:	nfs
Documentation:	---
CRM:
Verified Versions:
Dependent Products:

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Description Lakshmipathi G 2010-09-30 11:03:40 UTC

iozone running on nfsmount of 2dht setup. first added brick 10.214.231.112.
iozone worked fine.now added another brick-10.198.110.16. iozone failed with follow error message -

        32768    4096  962715  356568   352917   387695  440691  461455  354535   443128   382000   253940   295103  372672   394215
           32768    8192  841369  307591   337668   278419  427809  260872  394899   448139   410390   239400   311107  497192   410014
           32768   16384  834938  316067   656266   381649  260443  376925  347826   312340   285485   262290   418776  308972   388204
           65536      64  548225  393881   479877   509234  532328  413379  478462   474805   518850   398136   386394  447002   541974
           65536     128  580437  411598   428412   516796  435379  401773  517371   464215   538623   421610   405946  430430   460732
           65536     256  678743  414905   491296   497468  492707  394414  464481   524535   562347   351823   392737  456121   523856
           65536     512  631879  428555   460687   644550  645325  355671  495152   504674   572982   334724   376034  440990   639593
           65536    1024  546475  387906   441792   478462  441828  380764  484353   475139   735244fsync: Unknown error 526

iozone: interrupted

exiting iozone

[root@ip-10-245-210-193 mnt]# 


nfs-server log :
Given volfile:
+------------------------------------------------------------------------------+
  1: volume add1-client-0
  2:     type protocol/client
  3:     option transport-type tcp
  4:     option remote-host 10.192.141.187
  5:     option transport.socket.nodelay on
  6:     option remote-subvolume /mnt/add1
  7: end-volume
  8: 
  9: volume add1-client-1
 10:     type protocol/client
 11:     option transport-type tcp
 12:     option remote-host 10.192.134.144
 13:     option transport.socket.nodelay on
 14:     option remote-subvolume /mnt/add1
 15: end-volume
 16: 
 17: volume add1-client-2
 18:     type protocol/client
 19:     option transport-type tcp
 20:     option remote-host 10.214.231.112
 21:     option transport.socket.nodelay on
 22:     option remote-subvolume /mnt/add1
 23: end-volume
 24: 
 25: volume add1-dht
 26: type cluster/distribute
 27: #   option lookup-unhashed on
 28: #   option min-free-disk on
 29: #   option unhashed-sticky-bit on
 30:     subvolumes add1-client-0 add1-client-1 add1-client-2 
 31: end-volume
 32: 
 33: volume add1-write-behind
 34:     type performance/write-behind
 35: #   option flush-behind on
 36: #   option cache-size on
 37: #   option disable-for-first-nbytes on
 38: #   option enable-O_SYNC on
 39: #   option enable-trickling-writes on
 40:     subvolumes add1-dht
 41: end-volume
 42: 
 43: volume add1-read-ahead
 44:     type performance/read-ahead
 45: #   option force-atime-update on
 46: #   option page-count on
 47:     subvolumes add1-write-behind
 48: end-volume
 49: 
 50: volume add1-io-cache
 51:     type performance/io-cache
 52: #   option priority on
 53: #   option cache-timeout on
 54: #   option cache-size on
 55: #   option min-file-size on
 56: #   option max-file-size on
 57:     subvolumes add1-read-ahead
 58: end-volume
 59: 
 60: volume add1-quick-read
 61:     type performance/quick-read
 62: #   option priority on
 63: #   option cache-timeout on
 64: #   option cache-size on
 65: #   option max-file-size on
 66:     subvolumes add1-io-cache
 67: end-volume
 68: 
 69: volume add1-stat-prefetch
 70:     type performance/stat-prefetch
 71:     subvolumes add1-quick-read
 72: end-volume
 73: 
 74: volume add1
 75:     type debug/io-stats
 76:     option dump-fd-stats no
 77:     option latency-measurement no
 78:     subvolumes add1-stat-prefetch
 79: end-volume
 80: 
 81: volume nfs-server
 82: type nfs/server
 83: option rpc-auth.addr.add1.allow *
 84: option nfs.dynamic-volumes on
 85: option nfs3.add1.volume-id 25a2be99-d35b-464a-9155-a722a9aae4f1
 86: subvolumes  add1
 87: end-volume

+------------------------------------------------------------------------------+
[2010-09-30 06:54:37.599726] E [client-handshake.c:730:client_query_portmap_cbk] add1-client-2: failed to get the port number for remote subvolume
[2010-09-30 06:54:40.976995] I [client-handshake.c:660:select_server_supported_programs] add1-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-30 06:54:40.977254] I [client-handshake.c:496:client_setvolume_cbk] add1-client-0: Connected to 10.192.141.187:6971, attached to remote volume '/mnt/add1'.
[2010-09-30 06:54:41.808] I [client-handshake.c:660:select_server_supported_programs] add1-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-30 06:54:41.1408] I [client-handshake.c:496:client_setvolume_cbk] add1-client-1: Connected to 10.192.134.144:6971, attached to remote volume '/mnt/add1'.
[2010-09-30 06:54:43.987806] I [client-handshake.c:660:select_server_supported_programs] add1-client-2: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-30 06:54:43.988468] I [client-handshake.c:496:client_setvolume_cbk] add1-client-2: Connected to 10.214.231.112:6971, attached to remote volume '/mnt/add1'.
[2010-09-30 06:54:44.9833] I [nfs.c:315:__nfs_subvolume_start] nfs: All exports up
[2010-09-30 06:55:55.932922] I [glusterfsd.c:668:cleanup_and_exit] glusterfsd: shutting down
[2010-09-30 06:55:55.932951] I [glusterfsd.c:684:cleanup_and_exit] : Calling fini for nfs-server
[2010-09-30 06:55:55.933259] I [glusterfsd.c:686:cleanup_and_exit] : Done Calling fini for nfs-server
[2010-09-30 06:55:55.933277] I [glusterfsd.c:684:cleanup_and_exit] : Calling fini for add1
[2010-09-30 06:55:55.933293] I [io-stats.c:1680:fini] add1: io-stats translator unloaded
[2010-09-30 06:55:55.933308] I [glusterfsd.c:686:cleanup_and_exit] : Done Calling fini for add1
[2010-09-30 06:55:55.933322] I [glusterfsd.c:684:cleanup_and_exit] : Calling fini for add1-stat-prefetch
[2010-09-30 06:55:55.949389] I [glusterfsd.c:686:cleanup_and_exit] : Done Calling fini for add1-stat-prefetch
[2010-09-30 06:55:55.949454] I [glusterfsd.c:684:cleanup_and_exit] : Calling fini for add1-quick-read
[2010-09-30 06:55:55.949469] I [glusterfsd.c:686:cleanup_and_exit] : Done Calling fini for add1-quick-read
[2010-09-30 06:55:55.949484] I [glusterfsd.c:684:cleanup_and_exit] : Calling fini for add1-io-cache
[2010-09-30 06:55:55.949500] I [glusterfsd.c:686:cleanup_and_exit] : Done Calling fini for add1-io-cache
[2010-09-30 06:55:55.949514] I [glusterfsd.c:684:cleanup_and_exit] : Calling fini for add1-read-ahead
[2010-09-30 06:55:55.949529] I [glusterfsd.c:686:cleanup_and_exit] : Done Calling fini for add1-read-ahead
[2010-09-30 06:55:55.949572] I [glusterfsd.c:684:cleanup_and_exit] : Calling fini for add1-write-behind
[2010-09-30 06:55:55.949588] I [glusterfsd.c:686:cleanup_and_exit] : Done Calling fini for add1-write-behind
[2010-09-30 06:55:55.949602] I [glusterfsd.c:684:cleanup_and_exit] : Calling fini for add1-dht
[2010-09-30 06:55:55.949618] I [glusterfsd.c:686:cleanup_and_exit] : Done Calling fini for add1-dht
[2010-09-30 06:55:55.949632] I [glusterfsd.c:684:cleanup_and_exit] : Calling fini for add1-client-2
[2010-09-30 06:55:55.949722] I [glusterfsd.c:686:cleanup_and_exit] : Done Calling fini for add1-client-2
[2010-09-30 06:55:55.949750] I [glusterfsd.c:684:cleanup_and_exit] : Calling fini for add1-client-1
[2010-09-30 06:55:55.949923] I [glusterfsd.c:686:cleanup_and_exit] : Done Calling fini for add1-client-1
[2010-09-30 06:55:55.949948] I [glusterfsd.c:684:cleanup_and_exit] : Calling fini for add1-client-0
[2010-09-30 06:55:55.950294] E [rpc-clnt.c:335:saved_frames_unwind] rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(FSYNC(16)) called at 2010-09-30 06:55:55.471236
[2010-09-30 06:55:55.950407] I [glusterfsd.c:686:cleanup_and_exit] : Done Calling fini for add1-client-0
[2010-09-30 06:55:57.74227] I [nfs.c:769:init] nfs: NFS service started
[2010-09-30 06:55:57.74374] C [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-09-30 06:55:57.74392] C [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-09-30 06:55:57.104368] W [graph.c:120:_log_if_option_is_invalid] nfs-server: option 'nfs3.add1.volume-id' is not recognized
[2010-09-30 06:55:57.104428] W [graph.c:120:_log_if_option_is_invalid] add1-client-3: option 'transport.socket.nodelay' is not recognized
[2010-09-30 06:55:57.104453] W [graph.c:120:_log_if_option_is_invalid] add1-client-2: option 'transport.socket.nodelay' is not recognized
[2010-09-30 06:55:57.104470] W [graph.c:120:_log_if_option_is_invalid] add1-client-1: option 'transport.socket.nodelay' is not recognized
[2010-09-30 06:55:57.104487] W [graph.c:120:_log_if_option_is_invalid] add1-client-0: option 'transport.socket.nodelay' is not recognized
Given volfile:
+------------------------------------------------------------------------------+
  1: volume add1-client-0
  2:     type protocol/client
  3:     option transport-type tcp
  4:     option remote-host 10.192.141.187
  5:     option transport.socket.nodelay on
  6:     option remote-subvolume /mnt/add1
  7: end-volume
  8: 
  9: volume add1-client-1
 10:     type protocol/client
 11:     option transport-type tcp
 12:     option remote-host 10.192.134.144
 13:     option transport.socket.nodelay on
 14:     option remote-subvolume /mnt/add1
 15: end-volume
 16: 
 17: volume add1-client-2
 18:     type protocol/client
 19:     option transport-type tcp
 20:     option remote-host 10.214.231.112
 21:     option transport.socket.nodelay on
 22:     option remote-subvolume /mnt/add1
 23: end-volume
 24: 
 25: volume add1-client-3
 26:     type protocol/client
 27:     option transport-type tcp
 28:     option remote-host 10.198.110.16
 29:     option transport.socket.nodelay on
 30:     option remote-subvolume /mnt/add1
 31: end-volume
 32: 
 33: volume add1-dht
 34: type cluster/distribute
 35: #   option lookup-unhashed on
 36: #   option min-free-disk on
 37: #   option unhashed-sticky-bit on
 38:     subvolumes add1-client-0 add1-client-1 add1-client-2 add1-client-3 
 39: end-volume
 40: 
 41: volume add1-write-behind
 42:     type performance/write-behind
 43: #   option flush-behind on
 44: #   option cache-size on
 45: #   option disable-for-first-nbytes on
 46: #   option enable-O_SYNC on
 47: #   option enable-trickling-writes on
 48:     subvolumes add1-dht
 49: end-volume
 50: 
 51: volume add1-read-ahead
 52:     type performance/read-ahead
 53: #   option force-atime-update on
 54: #   option page-count on
 55:     subvolumes add1-write-behind
 56: end-volume
 57: 
 58: volume add1-io-cache
 59:     type performance/io-cache
 60: #   option priority on
 61: #   option cache-timeout on
 62: #   option cache-size on
 63: #   option min-file-size on
 64: #   option max-file-size on
 65:     subvolumes add1-read-ahead
 66: end-volume
 67: 
 68: volume add1-quick-read
 69:     type performance/quick-read
 70: #   option priority on
 71: #   option cache-timeout on
 72: #   option cache-size on
 73: #   option max-file-size on
 74:     subvolumes add1-io-cache
 75: end-volume
 76: 
 77: volume add1-stat-prefetch
 78:     type performance/stat-prefetch
 79:     subvolumes add1-quick-read
 80: end-volume
 81: 
 82: volume add1
 83:     type debug/io-stats
 84:     option dump-fd-stats no
 85:     option latency-measurement no
 86:     subvolumes add1-stat-prefetch
 87: end-volume
 88: 
 89: volume nfs-server
 90: type nfs/server
 91: option rpc-auth.addr.add1.allow *
 92: option nfs.dynamic-volumes on
 93: option nfs3.add1.volume-id 25a2be99-d35b-464a-9155-a722a9aae4f1
 94: subvolumes  add1
 95: end-volume

+------------------------------------------------------------------------------+
[2010-09-30 06:55:57.715687] E [client-handshake.c:730:client_query_portmap_cbk] add1-client-3: failed to get the port number for remote subvolume
[2010-09-30 06:56:00.108898] I [client-handshake.c:660:select_server_supported_programs] add1-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-30 06:56:00.109151] I [client-handshake.c:496:client_setvolume_cbk] add1-client-0: Connected to 10.192.141.187:6971, attached to remote volume '/mnt/add1'.
[2010-09-30 06:56:01.116451] I [client-handshake.c:660:select_server_supported_programs] add1-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-30 06:56:01.117121] I [client-handshake.c:496:client_setvolume_cbk] add1-client-1: Connected to 10.192.134.144:6971, attached to remote volume '/mnt/add1'.
[2010-09-30 06:56:01.128890] I [client-handshake.c:660:select_server_supported_programs] add1-client-2: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-30 06:56:01.129601] I [client-handshake.c:496:client_setvolume_cbk] add1-client-2: Connected to 10.214.231.112:6971, attached to remote volume '/mnt/add1'.
[2010-09-30 06:56:04.140323] I [client-handshake.c:660:select_server_supported_programs] add1-client-3: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-30 06:56:04.140990] I [client-handshake.c:496:client_setvolume_cbk] add1-client-3: Connected to 10.198.110.16:6971, attached to remote volume '/mnt/add1'.
[2010-09-30 06:56:04.142812] I [nfs.c:315:__nfs_subvolume_start] nfs: All exports up
===============
nfs-server log of newly added 2 brick
--
cat /etc/glusterd/logs/nfs.log
[2010-09-30 06:55:57.853569] I [nfs.c:769:init] nfs: NFS service started
[2010-09-30 06:55:57.853740] C [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-09-30 06:55:57.853758] C [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-09-30 06:55:57.873264] W [graph.c:120:_log_if_option_is_invalid] nfs-server: option 'nfs3.add1.volume-id' is not recognized
[2010-09-30 06:55:57.873320] W [graph.c:120:_log_if_option_is_invalid] add1-client-3: option 'transport.socket.nodelay' is not recognized
[2010-09-30 06:55:57.873340] W [graph.c:120:_log_if_option_is_invalid] add1-client-2: option 'transport.socket.nodelay' is not recognized
[2010-09-30 06:55:57.873357] W [graph.c:120:_log_if_option_is_invalid] add1-client-1: option 'transport.socket.nodelay' is not recognized
[2010-09-30 06:55:57.873373] W [graph.c:120:_log_if_option_is_invalid] add1-client-0: option 'transport.socket.nodelay' is not recognized
Given volfile:
+------------------------------------------------------------------------------+
  1: volume add1-client-0
  2:     type protocol/client
  3:     option transport-type tcp
  4:     option remote-host 10.192.141.187
  5:     option transport.socket.nodelay on
  6:     option remote-subvolume /mnt/add1
  7: end-volume
  8: 
  9: volume add1-client-1
 10:     type protocol/client
 11:     option transport-type tcp
 12:     option remote-host 10.192.134.144
 13:     option transport.socket.nodelay on
 14:     option remote-subvolume /mnt/add1
 15: end-volume
 16: 
 17: volume add1-client-2
 18:     type protocol/client
 19:     option transport-type tcp
 20:     option remote-host 10.214.231.112
 21:     option transport.socket.nodelay on
 22:     option remote-subvolume /mnt/add1
 23: end-volume
 24: 
 25: volume add1-client-3
 26:     type protocol/client
 27:     option transport-type tcp
 28:     option remote-host 10.198.110.16
 29:     option transport.socket.nodelay on
 30:     option remote-subvolume /mnt/add1
 31: end-volume
 32: 
 33: volume add1-dht
 34: type cluster/distribute
 35: #   option lookup-unhashed on
 36: #   option min-free-disk on
 37: #   option unhashed-sticky-bit on
 38:     subvolumes add1-client-0 add1-client-1 add1-client-2 add1-client-3 
 39: end-volume
 40: 
 41: volume add1-write-behind
 42:     type performance/write-behind
 43: #   option flush-behind on
 44: #   option cache-size on
 45: #   option disable-for-first-nbytes on
 46: #   option enable-O_SYNC on
 47: #   option enable-trickling-writes on
 48:     subvolumes add1-dht
 49: end-volume
 50: 
 51: volume add1-read-ahead
 52:     type performance/read-ahead
 53: #   option force-atime-update on
 54: #   option page-count on
 55:     subvolumes add1-write-behind
 56: end-volume
 57: 
 58: volume add1-io-cache
 59:     type performance/io-cache
 60: #   option priority on
 61: #   option cache-timeout on
 62: #   option cache-size on
 63: #   option min-file-size on
 64: #   option max-file-size on
 65:     subvolumes add1-read-ahead
 66: end-volume
 67: 
 68: volume add1-quick-read
 69:     type performance/quick-read
 70: #   option priority on
 71: #   option cache-timeout on
 72: #   option cache-size on
 73: #   option max-file-size on
 74:     subvolumes add1-io-cache
 75: end-volume
 76: 
 77: volume add1-stat-prefetch
 78:     type performance/stat-prefetch
 79:     subvolumes add1-quick-read
 80: end-volume
 81: 
 82: volume add1
 83:     type debug/io-stats
 84:     option dump-fd-stats no
 85:     option latency-measurement no
 86:     subvolumes add1-stat-prefetch
 87: end-volume
 88: 
 89: volume nfs-server
 90: type nfs/server
 91: option rpc-auth.addr.add1.allow *
 92: option nfs.dynamic-volumes on
 93: option nfs3.add1.volume-id 25a2be99-d35b-464a-9155-a722a9aae4f1
 94: subvolumes  add1
 95: end-volume

+------------------------------------------------------------------------------+
[2010-09-30 06:56:01.879993] I [client-handshake.c:660:select_server_supported_programs] add1-client-3: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-30 06:56:01.880339] I [client-handshake.c:496:client_setvolume_cbk] add1-client-3: Connected to 10.198.110.16:6971, attached to remote volume '/mnt/add1'.
[2010-09-30 06:56:01.881066] I [client-handshake.c:660:select_server_supported_programs] add1-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-30 06:56:01.881135] I [client-handshake.c:660:select_server_supported_programs] add1-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-30 06:56:01.881562] I [client-handshake.c:660:select_server_supported_programs] add1-client-2: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-30 06:56:01.881662] I [client-handshake.c:496:client_setvolume_cbk] add1-client-1: Connected to 10.192.134.144:6971, attached to remote volume '/mnt/add1'.
[2010-09-30 06:56:01.881778] I [client-handshake.c:496:client_setvolume_cbk] add1-client-0: Connected to 10.192.141.187:6971, attached to remote volume '/mnt/add1'.
[2010-09-30 06:56:01.882154] I [client-handshake.c:496:client_setvolume_cbk] add1-client-2: Connected to 10.214.231.112:6971, attached to remote volume '/mnt/add1'.
[2010-09-30 06:56:01.884300] I [nfs.c:315:__nfs_subvolume_start] nfs: All exports up

Comment 1 shishir gowda 2010-10-02 01:10:13 UTC

Not seen anymore in the latest git

Calculating checksums on source and destination
===============================================
Calculating checksum on directory /usr/src/linux-2.6.35.3/fs/ ...

Entry counts
Regular files   : 4474
Directories     : 80
Symbolic links  : 0
Other           : 0
Total           : 4554

Metadata checksums
Regular files   : cd9
Directories     : 3e9
Symbolic links  : 3e9
Other           : 3e9

Checksums
Regular files   : f856ad3e93cddaf4202131d7546458bb
Directories     : 1c62522c0a0e6146
Symbolic links  : 0
Other           : 0
Total           : c415cec5cda7e309
-------------------------------------

Calculating checksum on directory /mnt/afr ...

Entry counts
Regular files   : 4474
Directories     : 80
Symbolic links  : 0
Other           : 0
Total           : 4554

Metadata checksums
Regular files   : cd9
Directories     : cd9
Symbolic links  : 3e9
Other           : 3e9

Checksums
Regular files   : f856ad3e93cddaf4202131d7546458bb
Directories     : 1c62522c0a0e6146
Symbolic links  : 0
Other           : 0
Total           : c415cec5cda7e309
-------------------------------------

Comment 2 shishir gowda 2010-10-02 01:12:09 UTC

Ignore the last post

Comment 3 shishir gowda 2010-10-02 01:12:09 UTC

Ignore the last post

Comment 4 shishir gowda 2010-10-05 08:50:13 UTC

Not seen in latest git after patch for bug 763456

Note You need to log in before you can comment on or make changes to this bug.