Bug 763332 (GLUSTER-1600)

Summary: showmount works but unable to mount
Product: [Community] GlusterFS Reporter: Lakshmipathi G <lakshmipathi>
Component: nfsAssignee: Shehjar Tikoo <shehjart>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: low    
Version: 3.1-alphaCC: gluster-bugs, vijay
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: RTP Mount Type: nfs
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Lakshmipathi G 2010-09-15 04:25:20 UTC
server1 log file-
=========
Given volfile:
+------------------------------------------------------------------------------+
  1: volume simpledht-client-0
  2:     type protocol/client
  3:     option transport-type tcp
  4:     option remote-host 10.192.141.187
  5:     option transport.socket.nodelay on
  6:     option remote-subvolume /mnt/lf/
  7: end-volume
  8: 
  9: volume simpledht-client-1
 10:     type protocol/client
 11:     option transport-type tcp
 12:     option remote-host 10.192.134.144
 13:     option transport.socket.nodelay on
 14:     option remote-subvolume /mnt/lf
 15: end-volume
 16: 
 17: volume simpledht-client-2
 18:     type protocol/client
 19:     option transport-type tcp
 20:     option remote-host 10.214.231.112
 21:     option transport.socket.nodelay on
 22:     option remote-subvolume /mnt/lf
 23: end-volume
 24: 
 25: volume simpledht-client-3
 26:     type protocol/client
 27:     option transport-type tcp
 28:     option remote-host 10.198.110.16
 29:     option transport.socket.nodelay on
 30:     option remote-subvolume /mnt/lf
 31: end-volume
 32: 
 33: volume simpledht-dht
 34: type cluster/distribute
 35: #   option lookup-unhashed on
 36: #   option min-free-disk on
 37: #   option unhashed-sticky-bit on
 38:     subvolumes simpledht-client-0 simpledht-client-1 simpledht-client-2 simpledht-client-3 
 39: end-volume
 40: 
 41: volume simpledht-write-behind
 42:     type performance/write-behind
 43: #   option flush-behind on
 44: #   option cache-size on
 45: #   option disable-for-first-nbytes on
 46: #   option enable-O_SYNC on
 47: #   option enable-trickling-writes on
 48:     subvolumes simpledht-dht
 49: end-volume
 50: 
 51: volume simpledht-read-ahead
 52:     type performance/read-ahead
 53: #   option force-atime-update on
 54: #   option page-count on
 55:     subvolumes simpledht-write-behind
 56: end-volume
 57: 
 58: volume simpledht-io-cache
 59:     type performance/io-cache
 60: #   option priority on
 61: #   option cache-timeout on
 62: #   option cache-size on
 63: #   option min-file-size on
 64: #   option max-file-size on
 65:     subvolumes simpledht-read-ahead
 66: end-volume
 67: 
 68: volume simpledht-quick-read
 69:     type performance/quick-read
 70: #   option priority on
 71: #   option cache-timeout on
 72: #   option cache-size on
 73: #   option max-file-size on
 74:     subvolumes simpledht-io-cache
 75: end-volume
 76: 
 77: volume simpledht
 78:     type performance/stat-prefetch
 79:     subvolumes simpledht-quick-read
 80: end-volume
 81: 
 82: volume nfs-server
 83: type nfs/server
 84: option rpc-auth.addr.simpledht.allow *
 85: option nfs.dynamic-volumes on
 86: option nfs3.simpledht.volume-id 6ee4acb8-1d95-493d-a90d-dc9b076f9859
 87: subvolumes  simpledht
 88: end-volume

+------------------------------------------------------------------------------+
[2010-09-15 03:18:27.589181] E [client-handshake.c:730:client_query_portmap_cbk] simpledht-client-3: failed to get the port number for remote subvolume
[2010-09-15 03:18:27.628317] E [client-handshake.c:730:client_query_portmap_cbk] simpledht-client-2: failed to get the port number for remote subvolume
[2010-09-15 03:18:27.649204] E [client-handshake.c:730:client_query_portmap_cbk] simpledht-client-1: failed to get the port number for remote subvolume
[2010-09-15 03:18:30.10155] I [client-handshake.c:660:select_server_supported_programs] simpledht-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-15 03:18:30.87013] I [client-handshake.c:496:client_setvolume_cbk] simpledht-client-0: Connected to 10.192.141.187:6971, attached to remote volume '/mnt/lf/'.
[2010-09-15 03:18:34.23144] I [client-handshake.c:660:select_server_supported_programs] simpledht-client-3: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-15 03:18:34.23706] I [client-handshake.c:496:client_setvolume_cbk] simpledht-client-3: Connected to 10.198.110.16:6971, attached to remote volume '/mnt/lf'.
[2010-09-15 03:18:34.25463] I [client-handshake.c:660:select_server_supported_programs] simpledht-client-2: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-15 03:18:34.25999] I [client-handshake.c:496:client_setvolume_cbk] simpledht-client-2: Connected to 10.214.231.112:6971, attached to remote volume '/mnt/lf'.
[2010-09-15 03:18:34.27840] I [client-handshake.c:660:select_server_supported_programs] simpledht-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-15 03:18:34.28415] I [client-handshake.c:496:client_setvolume_cbk] simpledht-client-1: Connected to 10.192.134.144:6971, attached to remote volume '/mnt/lf'.
[2010-09-15 03:21:00.403425] E [rpcsvc.c:1249:nfs_rpcsvc_program_actor] nfsrpc: RPC program not available
==========================
server2:log
++++++++++++
+------------------------------------------------------------------------------+
  1: volume simpledht-client-0
  2:     type protocol/client
  3:     option transport-type tcp
  4:     option remote-host 10.192.141.187
  5:     option transport.socket.nodelay on
  6:     option remote-subvolume /mnt/lf/
  7: end-volume
  8: 
  9: volume simpledht-client-1
 10:     type protocol/client
 11:     option transport-type tcp
 12:     option remote-host 10.192.134.144
 13:     option transport.socket.nodelay on
 14:     option remote-subvolume /mnt/lf
 15: end-volume
 16: 
 17: volume simpledht-client-2
 18:     type protocol/client
 19:     option transport-type tcp
 20:     option remote-host 10.214.231.112
 21:     option transport.socket.nodelay on
 22:     option remote-subvolume /mnt/lf
 23: end-volume
 24: 
 25: volume simpledht-client-3
 26:     type protocol/client
 27:     option transport-type tcp
 28:     option remote-host 10.198.110.16
 29:     option transport.socket.nodelay on
 30:     option remote-subvolume /mnt/lf
 31: end-volume
 32: 
 33: volume simpledht-dht
 34: type cluster/distribute
 35: #   option lookup-unhashed on
 36: #   option min-free-disk on
 37: #   option unhashed-sticky-bit on
 38:     subvolumes simpledht-client-0 simpledht-client-1 simpledht-client-2 simpledht-client-3 
 39: end-volume
 40: 
 41: volume simpledht-write-behind
 42:     type performance/write-behind
 43: #   option flush-behind on
 44: #   option cache-size on
 45: #   option disable-for-first-nbytes on
 46: #   option enable-O_SYNC on
 47: #   option enable-trickling-writes on
 48:     subvolumes simpledht-dht
 49: end-volume
 50: 
 51: volume simpledht-read-ahead
 52:     type performance/read-ahead
 53: #   option force-atime-update on
 54: #   option page-count on
 55:     subvolumes simpledht-write-behind
 56: end-volume
 57: 
 58: volume simpledht-io-cache
 59:     type performance/io-cache
 60: #   option priority on
 61: #   option cache-timeout on
 62: #   option cache-size on
 63: #   option min-file-size on
 64: #   option max-file-size on
 65:     subvolumes simpledht-read-ahead
 66: end-volume
 67: 
 68: volume simpledht-quick-read
 69:     type performance/quick-read
 70: #   option priority on
 71: #   option cache-timeout on
 72: #   option cache-size on
 73: #   option max-file-size on
 74:     subvolumes simpledht-io-cache
 75: end-volume
 76: 
 77: volume simpledht
 78:     type performance/stat-prefetch
 79:     subvolumes simpledht-quick-read
 80: end-volume
 81: 
 82: volume nfs-server
 83: type nfs/server
 84: option rpc-auth.addr.simpledht.allow *
 85: option nfs.dynamic-volumes on
 86: option nfs3.simpledht.volume-id 6ee4acb8-1d95-493d-a90d-dc9b076f9859
 87: subvolumes  simpledht
 88: end-volume

+------------------------------------------------------------------------------+
[2010-09-15 03:18:31.764549] I [client-handshake.c:660:select_server_supported_programs] simpledht-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-15 03:18:31.764765] I [client-handshake.c:496:client_setvolume_cbk] simpledht-client-1: Connected to 10.192.134.144:6971, attached to remote volume '/mnt/lf'.
[2010-09-15 03:18:31.778125] I [client-handshake.c:660:select_server_supported_programs] simpledht-client-3: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-15 03:18:31.778219] I [client-handshake.c:660:select_server_supported_programs] simpledht-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-15 03:18:31.778767] I [client-handshake.c:496:client_setvolume_cbk] simpledht-client-0: Connected to 10.192.141.187:6971, attached to remote volume '/mnt/lf/'.
[2010-09-15 03:18:31.778857] I [client-handshake.c:496:client_setvolume_cbk] simpledht-client-3: Connected to 10.198.110.16:6971, attached to remote volume '/mnt/lf'.
[2010-09-15 03:18:31.779121] I [client-handshake.c:660:select_server_supported_programs] simpledht-client-2: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-15 03:18:31.779614] I [client-handshake.c:496:client_setvolume_cbk] simpledht-client-2: Connected to 10.214.231.112:6971, attached to remote volume '/mnt/lf'
===============
server3:log
+++++++++

Given volfile:
+------------------------------------------------------------------------------+
  1: volume simpledht-client-0
  2:     type protocol/client
  3:     option transport-type tcp
  4:     option remote-host 10.192.141.187
  5:     option transport.socket.nodelay on
  6:     option remote-subvolume /mnt/lf/
  7: end-volume
  8: 
  9: volume simpledht-client-1
 10:     type protocol/client
 11:     option transport-type tcp
 12:     option remote-host 10.192.134.144
 13:     option transport.socket.nodelay on
 14:     option remote-subvolume /mnt/lf
 15: end-volume
 16: 
 17: volume simpledht-client-2
 18:     type protocol/client
 19:     option transport-type tcp
 20:     option remote-host 10.214.231.112
 21:     option transport.socket.nodelay on
 22:     option remote-subvolume /mnt/lf
 23: end-volume
 24: 
 25: volume simpledht-client-3
 26:     type protocol/client
 27:     option transport-type tcp
 28:     option remote-host 10.198.110.16
 29:     option transport.socket.nodelay on
 30:     option remote-subvolume /mnt/lf
 31: end-volume
 32: 
 33: volume simpledht-dht
 34: type cluster/distribute
 35: #   option lookup-unhashed on
 36: #   option min-free-disk on
 37: #   option unhashed-sticky-bit on
 38:     subvolumes simpledht-client-0 simpledht-client-1 simpledht-client-2 simpledht-client-3 
 39: end-volume
 40: 
 41: volume simpledht-write-behind
 42:     type performance/write-behind
 43: #   option flush-behind on
 44: #   option cache-size on
 45: #   option disable-for-first-nbytes on
 46: #   option enable-O_SYNC on
 47: #   option enable-trickling-writes on
 48:     subvolumes simpledht-dht
 49: end-volume
 50: 
 51: volume simpledht-read-ahead
 52:     type performance/read-ahead
 53: #   option force-atime-update on
 54: #   option page-count on
 55:     subvolumes simpledht-write-behind
 56: end-volume
 57: 
 58: volume simpledht-io-cache
 59:     type performance/io-cache
 60: #   option priority on
 61: #   option cache-timeout on
 62: #   option cache-size on
 63: #   option min-file-size on
 64: #   option max-file-size on
 65:     subvolumes simpledht-read-ahead
 66: end-volume
 67: 
 68: volume simpledht-quick-read
 69:     type performance/quick-read
 70: #   option priority on
 71: #   option cache-timeout on
 72: #   option cache-size on
 73: #   option max-file-size on
 74:     subvolumes simpledht-io-cache
 75: end-volume
 76: 
 77: volume simpledht
 78:     type performance/stat-prefetch
 79:     subvolumes simpledht-quick-read
 80: end-volume
 81: 
 82: volume nfs-server
 83: type nfs/server
 84: option rpc-auth.addr.simpledht.allow *
 85: option nfs.dynamic-volumes on
 86: option nfs3.simpledht.volume-id 6ee4acb8-1d95-493d-a90d-dc9b076f9859
 87: subvolumes  simpledht
 88: end-volume

+------------------------------------------------------------------------------+
[2010-09-15 03:18:31.666167] I [client-handshake.c:660:select_server_supported_programs] simpledht-client-2: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-15 03:18:31.666465] I [client-handshake.c:496:client_setvolume_cbk] simpledht-client-2: Connected to 10.214.231.112:6971, attached to remote volume '/mnt/lf'.
[2010-09-15 03:18:31.669728] I [client-handshake.c:660:select_server_supported_programs] simpledht-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-15 03:18:31.669813] I [client-handshake.c:660:select_server_supported_programs] simpledht-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-15 03:18:31.669907] I [client-handshake.c:660:select_server_supported_programs] simpledht-client-3: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-15 03:18:31.670381] I [client-handshake.c:496:client_setvolume_cbk] simpledht-client-1: Connected to 10.192.134.144:6971, attached to remote volume '/mnt/lf'.
[2010-09-15 03:18:31.670453] I [client-handshake.c:496:client_setvolume_cbk] simpledht-client-0: Connected to 10.192.141.187:6971, attached to remote volume '/mnt/lf/'.
[2010-09-15 03:18:31.670534] I [client-handshake.c:496:client_setvolume_cbk] simpledht-client-3: Connected to 10.198.110.16:6971, attached to remote volume '/mnt/lf'.
========

Comment 1 Lakshmipathi G 2010-09-15 04:48:47 UTC
glusterd log from server2
#cat /etc/glusterd/logs/mnt-lf.log 
[2010-09-15 03:46:50.819438] W [graph.c:253:gf_add_cmdline_options] simpledht-server: adding option 'listen-port' for volume 'simpledht-server' with value '6971'
[2010-09-15 03:46:50.820178] W [rpc-transport.c:566:validate_volume_options] tcp.simpledht-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction
[2010-09-15 03:46:50.820527] I [client.c:1661:build_client_config] simpledht-replace-brick: option 'remote-subvolume' not given
[2010-09-15 03:46:50.820647] W [graph.c:120:_log_if_option_is_invalid] simpledht-replace-brick: option 'transport.socket.nodelay' is not recognized
[2010-09-15 03:46:50.820666] W [graph.c:120:_log_if_option_is_invalid] simpledht-replace-brick: option 'remote-port' is not recognized
[2010-09-15 03:46:50.820687] I [afr-common.c:2627:afr_notify] simpledht-pump: Subvolume 'simpledht-locks' came back up; going online.
Given volfile:
+------------------------------------------------------------------------------+
  1: volume simpledht-posix
  2:     type storage/posix
  3:     option directory /mnt/lf
  4: #   option o-direct on
  5: #   option export-statfs-size on
  6: #   option mandate-attribute on
  7: #   option span-devices 1
  8: #   option background-unlink on
  9: end-volume
 10: 
 11: volume simpledht-access-control
 12: type features/access-control
 13: subvolumes simpledht-posix
 14: end-volume
 15: volume simpledht-locks
 16:     type features/locks
 17: #   option trace on
 18: #   option mandatory on
 19:     subvolumes simpledht-access-control
 20: end-volume
 21: 
 22: volume simpledht-replace-brick
 23:     type protocol/client
 24:     option transport-type tcp
 25:     option remote-port 34034
 26:     option transport.socket.nodelay on
 27: end-volume
 28: 
 29: volume simpledht-pump
 30:     type cluster/pump
 31:     subvolumes simpledht-locks simpledht-replace-brick
 32: end-volume
 33: 
 34: volume /mnt/lf
 35:     type performance/io-threads
 36:     option thread-count 16
 37: #   option autoscaling on
 38: #   option min-threads on
 39: #   option max-threads on
 40:     subvolumes simpledht-pump
 41: end-volume
 42: 
 43: volume simpledht-server
 44:     type protocol/server
 45:     option transport-type tcp
 46:     option auth.addr./mnt/lf.allow *
 47:     option transport.socket.nodelay on
 48:     subvolumes /mnt/lf
 49: end-volume
 50: 

+------------------------------------------------------------------------------+
[2010-09-15 03:46:53.949776] I [server-handshake.c:523:server_setvolume] simpledht-server: accepted client from 10.192.134.144:1019
[2010-09-15 03:46:54.951671] I [server-handshake.c:523:server_setvolume] simpledht-server: accepted client from 10.198.110.16:1013
[2010-09-15 03:46:54.963688] I [server-handshake.c:523:server_setvolume] simpledht-server: accepted client from 10.214.231.112:1012
[2010-09-15 03:46:57.218481] I [server-handshake.c:523:server_setvolume] simpledht-server: accepted client from 10.192.141.187:1014

Comment 2 Vijay Bellur 2010-09-15 06:18:10 UTC
PATCH: http://patches.gluster.com/patch/4792 in master (nfs3: Set gfid[15] to 1 for root)

Comment 3 Lakshmipathi G 2010-09-15 06:47:28 UTC
While testing with 3.1.0qa22,created simple 4 dht cluster.
showmount from all servers and client is fine. 
while trying to mount it says following -
# mount 10.192.141.187:/simpledht /mnt/
mount.nfs: Stale NFS file handle


Here is the server log file
----------
#cat /usr/local/var/log/glusterfs/etc-glusterd-nfs-nfs-server.vol.log 
[2010-09-15 02:43:14.614504] I [nfs.c:576:init] nfs: NFS service started
[2010-09-15 02:43:14.614691] C [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-09-15 02:43:14.614710] C [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-09-15 02:43:14.644359] W [graph.c:120:_log_if_option_is_invalid] nfs-server: option 'nfs3.simpledht.volume-id' is not recognized
[2010-09-15 02:43:14.644419] W [graph.c:120:_log_if_option_is_invalid] simpledht-client-3: option 'transport.socket.nodelay' is not recognized
[2010-09-15 02:43:14.644438] W [graph.c:120:_log_if_option_is_invalid] simpledht-client-2: option 'transport.socket.nodelay' is not recognized
[2010-09-15 02:43:14.644454] W [graph.c:120:_log_if_option_is_invalid] simpledht-client-1: option 'transport.socket.nodelay' is not recognized
[2010-09-15 02:43:14.644471] W [graph.c:120:_log_if_option_is_invalid] simpledht-client-0: option 'transport.socket.nodelay' is not recognized
Given volfile:
+------------------------------------------------------------------------------+
  1: volume simpledht-client-0
  2:     type protocol/client
  3:     option transport-type tcp
  4:     option remote-host 10.192.141.187
  5:     option transport.socket.nodelay on
  6:     option remote-subvolume /mnt/lf/
  7: end-volume
  8: 
  9: volume simpledht-client-1
 10:     type protocol/client
 11:     option transport-type tcp
 12:     option remote-host 10.192.134.144
 13:     option transport.socket.nodelay on
 14:     option remote-subvolume /mnt/lf
 15: end-volume
 16: 
 17: volume simpledht-client-2
 18:     type protocol/client
 19:     option transport-type tcp
 20:     option remote-host 10.214.231.112
 21:     option transport.socket.nodelay on
 22:     option remote-subvolume /mnt/lf
 23: end-volume
 24: 
 25: volume simpledht-client-3
 26:     type protocol/client
 27:     option transport-type tcp
 28:     option remote-host 10.198.110.16
 29:     option transport.socket.nodelay on
 30:     option remote-subvolume /mnt/lf
 31: end-volume
 32: 
 33: volume simpledht-dht
 34: type cluster/distribute
 35: #   option lookup-unhashed on
 36: #   option min-free-disk on
 37: #   option unhashed-sticky-bit on
 38:     subvolumes simpledht-client-0 simpledht-client-1 simpledht-client-2 simpledht-client-3 
 39: end-volume
 40: 
 41: volume simpledht-write-behind
 42:     type performance/write-behind
 43: #   option flush-behind on
 44: #   option cache-size on
 45: #   option disable-for-first-nbytes on
 46: #   option enable-O_SYNC on
 47: #   option enable-trickling-writes on
 48:     subvolumes simpledht-dht
 49: end-volume
 50: 
 51: volume simpledht-read-ahead
 52:     type performance/read-ahead
 53: #   option force-atime-update on
 54: #   option page-count on
 55:     subvolumes simpledht-write-behind
 56: end-volume
 57: 
 58: volume simpledht-io-cache
 59:     type performance/io-cache
 60: #   option priority on
 61: #   option cache-timeout on
 62: #   option cache-size on
 63: #   option min-file-size on
 64: #   option max-file-size on
 65:     subvolumes simpledht-read-ahead
 66: end-volume
 67: 
 68: volume simpledht-quick-read
 69:     type performance/quick-read
 70: #   option priority on
 71: #   option cache-timeout on
 72: #   option cache-size on
 73: #   option max-file-size on
 74:     subvolumes simpledht-io-cache
 75: end-volume
 76: 
 77: volume simpledht
 78:     type performance/stat-prefetch
 79:     subvolumes simpledht-quick-read
 80: end-volume
 81: 
 82: volume nfs-server
 83: type nfs/server
 84: option rpc-auth.addr.simpledht.allow *
 85: option nfs.dynamic-volumes on
 86: option nfs3.simpledht.volume-id 72f58b54-fbc8-4af6-b6bb-76a354a9997b
 87: subvolumes  simpledht
 88: end-volume

+------------------------------------------------------------------------------+
[2010-09-15 02:43:14.806156] E [client-handshake.c:730:client_query_portmap_cbk] simpledht-client-2: failed to get the port number for remote subvolume
[2010-09-15 02:43:14.849179] E [client-handshake.c:730:client_query_portmap_cbk] simpledht-client-3: failed to get the port number for remote subvolume
[2010-09-15 02:43:18.646259] I [client-handshake.c:660:select_server_supported_programs] simpledht-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-15 02:43:18.646506] I [client-handshake.c:496:client_setvolume_cbk] simpledht-client-0: Connected to 10.192.141.187:6971, attached to remote volume '/mnt/lf/'.
[2010-09-15 02:43:18.653208] I [client-handshake.c:660:select_server_supported_programs] simpledht-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-15 02:43:18.653683] I [client-handshake.c:496:client_setvolume_cbk] simpledht-client-1: Connected to 10.192.134.144:6971, attached to remote volume '/mnt/lf'.

Comment 4 Vijay Bellur 2010-09-15 08:55:21 UTC
PATCH: http://patches.gluster.com/patch/4795 in master (nfs3: Copy deviceid from correct gfid start octet)

Comment 5 Lakshmipathi G 2010-09-17 04:02:15 UTC
verified with 3.1.0qa25,its fixed.