Bug 884411
Summary: | client logs after graph switch is misleading as an E | ||
---|---|---|---|
Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Rahul Hinduja <rhinduja> |
Component: | replicate | Assignee: | Anuradha <atalur> |
Status: | CLOSED WONTFIX | QA Contact: | Rahul Hinduja <rhinduja> |
Severity: | low | Docs Contact: | |
Priority: | low | ||
Version: | 2.0 | CC: | khoi.mai2008, kparthas, nsathyan, pkarampu, rhs-bugs, smohan, spandura, storage-qa-internal, vagarwal, vbellur |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | x86_64 | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | Bug Fix | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2015-03-23 07:37:18 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Rahul Hinduja
2012-12-06 07:33:10 UTC
*** Bug 921360 has been marked as a duplicate of this bug. *** vsomyaju, Could you add a comment with the review url for the patch(es) which fix this issue? I am moving the bug state to ASSIGNED. The necessary criteria for a bug to be in POST state is that patches that fix the bug needs to be merged into upstream master branch. Once you have added the review url and a summary of the root cause, you could move the bug to POST state. I believe this client log does it for every "volume set VOLUME FEATURE". I am using gluster3.4.1-3. see below [2013-12-12 20:24:05.919068] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 1-devstatic-client-0: changing port to 49153 (from 0) [2013-12-12 20:24:05.919118] W [socket.c:514:__socket_rwv] 1-devstatic-client-0: readv failed (No data available) [2013-12-12 20:24:05.923322] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 1-devstatic-client-2: changing port to 49153 (from 0) [2013-12-12 20:24:05.923354] W [socket.c:514:__socket_rwv] 1-devstatic-client-2: readv failed (No data available) [2013-12-12 20:24:05.927187] I [client-handshake.c:1658:select_server_supported_programs] 1-devstatic-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2013-12-12 20:24:05.927369] I [client-handshake.c:1658:select_server_supported_programs] 1-devstatic-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2013-12-12 20:24:05.927555] I [client-handshake.c:1456:client_setvolume_cbk] 1-devstatic-client-0: Connected to 167.132.88.124:49153, attached to remote volume '/static/content'. [2013-12-12 20:24:05.927574] I [client-handshake.c:1468:client_setvolume_cbk] 1-devstatic-client-0: Server and Client lk-version numbers are not same, reopening the fds [2013-12-12 20:24:05.927612] I [afr-common.c:3698:afr_notify] 1-devstatic-replicate-0: Subvolume 'devstatic-client-0' came back up; going online. [2013-12-12 20:24:05.927634] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 1-devstatic-client-3: changing port to 49153 (from 0) [2013-12-12 20:24:05.927662] W [socket.c:514:__socket_rwv] 1-devstatic-client-3: readv failed (No data available) [2013-12-12 20:24:05.931289] I [client-handshake.c:1456:client_setvolume_cbk] 1-devstatic-client-2: Connected to 167.132.88.125:49153, attached to remote volume '/static/content'. [2013-12-12 20:24:05.931309] I [client-handshake.c:1468:client_setvolume_cbk] 1-devstatic-client-2: Server and Client lk-version numbers are not same, reopening the fds [2013-12-12 20:24:05.931341] I [afr-common.c:3698:afr_notify] 1-devstatic-replicate-1: Subvolume 'devstatic-client-2' came back up; going online. [2013-12-12 20:24:05.931357] I [client-handshake.c:450:client_set_lk_version_cbk] 1-devstatic-client-0: Server lk version = 1 [2013-12-12 20:24:05.931600] I [client-handshake.c:450:client_set_lk_version_cbk] 1-devstatic-client-2: Server lk version = 1 [2013-12-12 20:24:05.932409] I [client-handshake.c:1658:select_server_supported_programs] 1-devstatic-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2013-12-12 20:24:05.933199] I [client-handshake.c:1456:client_setvolume_cbk] 1-devstatic-client-3: Connected to 69.58.224.73:49153, attached to remote volume '/static/content'. [2013-12-12 20:24:05.933229] I [client-handshake.c:1468:client_setvolume_cbk] 1-devstatic-client-3: Server and Client lk-version numbers are not same, reopening the fds [2013-12-12 20:24:05.933865] I [client-handshake.c:450:client_set_lk_version_cbk] 1-devstatic-client-3: Server lk version = 1 [2013-12-12 20:24:05.960219] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 1-devstatic-client-1: changing port to 49153 (from 0) [2013-12-12 20:24:05.960258] W [socket.c:514:__socket_rwv] 1-devstatic-client-1: readv failed (No data available) [2013-12-12 20:24:05.965304] I [client-handshake.c:1658:select_server_supported_programs] 1-devstatic-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2013-12-12 20:24:05.966392] I [client-handshake.c:1456:client_setvolume_cbk] 1-devstatic-client-1: Connected to 69.58.224.72:49153, attached to remote volume '/static/content'. [2013-12-12 20:24:05.966412] I [client-handshake.c:1468:client_setvolume_cbk] 1-devstatic-client-1: Server and Client lk-version numbers are not same, reopening the fds [2013-12-12 20:24:05.974637] I [fuse-bridge.c:4769:fuse_graph_setup] 0-fuse: switched to graph 1 [2013-12-12 20:24:05.974713] I [client-handshake.c:450:client_set_lk_version_cbk] 1-devstatic-client-1: Server lk version = 1 [2013-12-12 20:24:10.140771] I [afr-common.c:2057:afr_set_root_inode_on_first_lookup] 1-devstatic-replicate-0: added root inode [2013-12-12 20:24:10.141824] I [afr-common.c:2057:afr_set_root_inode_on_first_lookup] 1-devstatic-replicate-1: added root inode [2013-12-12 20:24:10.142557] I [client.c:2163:notify] 0-devstatic-client-0: current graph is no longer active, destroying rpc_client [2013-12-12 20:24:10.142610] I [client.c:2163:notify] 0-devstatic-client-1: current graph is no longer active, destroying rpc_client [2013-12-12 20:24:10.142628] I [client.c:2163:notify] 0-devstatic-client-2: current graph is no longer active, destroying rpc_client [2013-12-12 20:24:10.142641] I [client.c:2163:notify] 0-devstatic-client-3: current graph is no longer active, destroying rpc_client [2013-12-12 20:24:10.142638] W [socket.c:514:__socket_rwv] 0-devstatic-client-0: readv failed (No data available) [2013-12-12 20:24:10.142685] I [client.c:2097:client_rpc_notify] 0-devstatic-client-0: disconnected [2013-12-12 20:24:10.142707] W [socket.c:514:__socket_rwv] 0-devstatic-client-1: readv failed (No data available) [2013-12-12 20:24:10.142726] I [client.c:2097:client_rpc_notify] 0-devstatic-client-1: disconnected [2013-12-12 20:24:10.142734] E [afr-common.c:3735:afr_notify] 0-devstatic-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2013-12-12 20:24:10.142748] W [socket.c:514:__socket_rwv] 0-devstatic-client-2: readv failed (No data available) [2013-12-12 20:24:10.142763] I [client.c:2097:client_rpc_notify] 0-devstatic-client-2: disconnected [2013-12-12 20:24:10.142774] W [socket.c:514:__socket_rwv] 0-devstatic-client-3: readv failed (No data available) [2013-12-12 20:24:10.142793] I [client.c:2097:client_rpc_notify] 0-devstatic-client-3: disconnected [2013-12-12 20:24:10.142800] E [afr-common.c:3735:afr_notify] 0-devstatic-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. [2013-12-12 20:24:17.779518] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2013-12-12 20:24:17.802635] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2013-12-12 20:24:17.812883] I [quota.c:3051:quota_parse_limits] 2-devstatic-quota: /:751619276800 [2013-12-12 20:24:17.812898] I [quota.c:3083:quota_parse_limits] 2-devstatic-quota: /:751619276800 [2013-12-12 20:24:17.816915] I [socket.c:3480:socket_init] 2-devstatic-client-3: SSL support is NOT enabled [2013-12-12 20:24:17.816944] I [socket.c:3495:socket_init] 2-devstatic-client-3: using system polling thread [2013-12-12 20:24:17.817784] I [socket.c:3480:socket_init] 2-devstatic-client-2: SSL support is NOT enabled [2013-12-12 20:24:17.817801] I [socket.c:3495:socket_init] 2-devstatic-client-2: using system polling thread [2013-12-12 20:24:17.818640] I [socket.c:3480:socket_init] 2-devstatic-client-1: SSL support is NOT enabled [2013-12-12 20:24:17.818659] I [socket.c:3495:socket_init] 2-devstatic-client-1: using system polling thread [2013-12-12 20:24:17.819535] I [socket.c:3480:socket_init] 2-devstatic-client-0: SSL support is NOT enabled [2013-12-12 20:24:17.819557] I [socket.c:3495:socket_init] 2-devstatic-client-0: using system polling thread [2013-12-12 20:24:17.819588] I [client.c:2154:notify] 2-devstatic-client-0: parent translators are ready, attempting connect on transport [2013-12-12 20:24:17.823300] I [client.c:2154:notify] 2-devstatic-client-1: parent translators are ready, attempting connect on transport [2013-12-12 20:24:17.826752] I [client.c:2154:notify] 2-devstatic-client-2: parent translators are ready, attempting connect on transport [2013-12-12 20:24:17.830184] I [client.c:2154:notify] 2-devstatic-client-3: parent translators are ready, attempting connect on transport Given volfile: +------------------------------------------------------------------------------+ 1: volume devstatic-client-0 2: type protocol/client 3: option transport-type tcp 4: option remote-subvolume /static/content 5: option remote-host omhq1b4e 6: end-volume 7: 8: volume devstatic-client-1 9: type protocol/client 10: option transport-type tcp 11: option remote-subvolume /static/content 12: option remote-host omdx1b50 13: end-volume 14: 15: volume devstatic-client-2 16: type protocol/client 17: option transport-type tcp 18: option remote-subvolume /static/content 19: option remote-host omhq1b4f 20: end-volume 21: 22: volume devstatic-client-3 23: type protocol/client 24: option transport-type tcp 25: option remote-subvolume /static/content 26: option remote-host omdx1b51 27: end-volume 28: 29: volume devstatic-replicate-0 30: type cluster/replicate 31: option eager-lock off 32: subvolumes devstatic-client-0 devstatic-client-1 33: end-volume 34: 35: volume devstatic-replicate-1 36: type cluster/replicate 37: option eager-lock off 38: subvolumes devstatic-client-2 devstatic-client-3 39: end-volume 40: 41: volume devstatic-dht 42: type cluster/distribute 43: subvolumes devstatic-replicate-0 devstatic-replicate-1 44: end-volume 45: 46: volume devstatic-quota 47: type features/quota 48: option timeout 0 49: option limit-set /:700GB 50: subvolumes devstatic-dht 51: end-volume 52: 53: volume devstatic-write-behind 54: type performance/write-behind 55: subvolumes devstatic-quota 56: end-volume 57: 58: volume devstatic-read-ahead 59: type performance/read-ahead 60: subvolumes devstatic-write-behind 61: end-volume 62: 63: volume devstatic-io-cache 64: type performance/io-cache 65: subvolumes devstatic-read-ahead 66: end-volume 67: 68: volume devstatic-quick-read 69: type performance/quick-read 70: subvolumes devstatic-io-cache 71: end-volume 72: 73: volume devstatic-open-behind 74: type performance/open-behind 75: option lazy-open off 76: subvolumes devstatic-quick-read 77: end-volume 78: 79: volume devstatic-md-cache 80: type performance/md-cache 81: subvolumes devstatic-open-behind 82: end-volume 83: 84: volume devstatic 85: type debug/io-stats 86: option count-fop-hits off 87: option latency-measurement off 88: subvolumes devstatic-md-cache 89: end-volume +------------------------------------------------------------------------------+ [2013-12-12 20:24:17.833743] I [glusterfsd-mgmt.c:1584:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2013-12-12 20:24:17.834385] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 2-devstatic-client-2: changing port to 49153 (from 0) [2013-12-12 20:24:17.834427] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 2-devstatic-client-0: changing port to 49153 (from 0) [2013-12-12 20:24:17.834460] W [socket.c:514:__socket_rwv] 2-devstatic-client-2: readv failed (No data available) [2013-12-12 20:24:17.838111] W [socket.c:514:__socket_rwv] 2-devstatic-client-0: readv failed (No data available) [2013-12-12 20:24:17.842423] I [client-handshake.c:1658:select_server_supported_programs] 2-devstatic-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2013-12-12 20:24:17.842527] I [client-handshake.c:1658:select_server_supported_programs] 2-devstatic-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2013-12-12 20:24:17.842786] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 2-devstatic-client-3: changing port to 49153 (from 0) [2013-12-12 20:24:17.842845] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 2-devstatic-client-1: changing port to 49153 (from 0) [2013-12-12 20:24:17.842869] W [socket.c:514:__socket_rwv] 2-devstatic-client-3: readv failed (No data available) [2013-12-12 20:24:17.846802] W [socket.c:514:__socket_rwv] 2-devstatic-client-1: readv failed (No data available) [2013-12-12 20:24:17.850348] I [client-handshake.c:1456:client_setvolume_cbk] 2-devstatic-client-2: Connected to 167.132.88.125:49153, attached to remote volume '/static/content'. [2013-12-12 20:24:17.850368] I [client-handshake.c:1468:client_setvolume_cbk] 2-devstatic-client-2: Server and Client lk-version numbers are not same, reopening the fds [2013-12-12 20:24:17.850415] I [afr-common.c:3698:afr_notify] 2-devstatic-replicate-1: Subvolume 'devstatic-client-2' came back up; going online. [2013-12-12 20:24:17.850443] I [client-handshake.c:1456:client_setvolume_cbk] 2-devstatic-client-0: Connected to 167.132.88.124:49153, attached to remote volume '/static/content'. [2013-12-12 20:24:17.850452] I [client-handshake.c:1468:client_setvolume_cbk] 2-devstatic-client-0: Server and Client lk-version numbers are not same, reopening the fds [2013-12-12 20:24:17.850485] I [afr-common.c:3698:afr_notify] 2-devstatic-replicate-0: Subvolume 'devstatic-client-0' came back up; going online. [2013-12-12 20:24:17.850674] I [client-handshake.c:450:client_set_lk_version_cbk] 2-devstatic-client-2: Server lk version = 1 [2013-12-12 20:24:17.850734] I [client-handshake.c:450:client_set_lk_version_cbk] 2-devstatic-client-0: Server lk version = 1 [2013-12-12 20:24:17.851116] I [client-handshake.c:1658:select_server_supported_programs] 2-devstatic-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2013-12-12 20:24:17.851452] I [client-handshake.c:1658:select_server_supported_programs] 2-devstatic-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2013-12-12 20:24:17.851864] I [client-handshake.c:1456:client_setvolume_cbk] 2-devstatic-client-3: Connected to 69.58.224.73:49153, attached to remote volume '/static/content'. [2013-12-12 20:24:17.851884] I [client-handshake.c:1468:client_setvolume_cbk] 2-devstatic-client-3: Server and Client lk-version numbers are not same, reopening the fds [2013-12-12 20:24:17.852244] I [client-handshake.c:1456:client_setvolume_cbk] 2-devstatic-client-1: Connected to 69.58.224.72:49153, attached to remote volume '/static/content'. [2013-12-12 20:24:17.852267] I [client-handshake.c:1468:client_setvolume_cbk] 2-devstatic-client-1: Server and Client lk-version numbers are not same, reopening the fds [2013-12-12 20:24:17.859561] I [fuse-bridge.c:4769:fuse_graph_setup] 0-fuse: switched to graph 2 [2013-12-12 20:24:17.859662] I [client-handshake.c:450:client_set_lk_version_cbk] 2-devstatic-client-1: Server lk version = 1 [2013-12-12 20:24:17.859682] I [client-handshake.c:450:client_set_lk_version_cbk] 2-devstatic-client-3: Server lk version = 1 [2013-12-12 20:24:20.146200] I [afr-common.c:2057:afr_set_root_inode_on_first_lookup] 2-devstatic-replicate-0: added root inode [2013-12-12 20:24:20.147207] I [afr-common.c:2057:afr_set_root_inode_on_first_lookup] 2-devstatic-replicate-1: added root inode [2013-12-12 20:24:20.147874] I [client.c:2163:notify] 1-devstatic-client-0: current graph is no longer active, destroying rpc_client [2013-12-12 20:24:20.147923] I [client.c:2163:notify] 1-devstatic-client-1: current graph is no longer active, destroying rpc_client [2013-12-12 20:24:20.147954] I [client.c:2163:notify] 1-devstatic-client-2: current graph is no longer active, destroying rpc_client [2013-12-12 20:24:20.147953] W [socket.c:514:__socket_rwv] 1-devstatic-client-0: readv failed (No data available) [2013-12-12 20:24:20.147973] I [client.c:2163:notify] 1-devstatic-client-3: current graph is no longer active, destroying rpc_client [2013-12-12 20:24:20.148004] I [client.c:2097:client_rpc_notify] 1-devstatic-client-0: disconnected [2013-12-12 20:24:20.148029] W [socket.c:514:__socket_rwv] 1-devstatic-client-1: readv failed (No data available) [2013-12-12 20:24:20.148045] I [client.c:2097:client_rpc_notify] 1-devstatic-client-1: disconnected [2013-12-12 20:24:20.148052] E [afr-common.c:3735:afr_notify] 1-devstatic-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2013-12-12 20:24:20.148068] W [socket.c:514:__socket_rwv] 1-devstatic-client-2: readv failed (No data available) [2013-12-12 20:24:20.148092] I [client.c:2097:client_rpc_notify] 1-devstatic-client-2: disconnected [2013-12-12 20:24:20.148105] W [socket.c:514:__socket_rwv] 1-devstatic-client-3: readv failed (No data available) [2013-12-12 20:24:20.148121] I [client.c:2097:client_rpc_notify] 1-devstatic-client-3: disconnected [2013-12-12 20:24:20.148128] E [afr-common.c:3735:afr_notify] 1-devstatic-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. [2013-12-12 20:24:23.539011] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2013-12-12 20:24:24.590371] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2013-12-12 20:24:24.598105] I [quota.c:3051:quota_parse_limits] 2-devstatic-quota: /:751619276800 [2013-12-12 20:24:24.598121] I [quota.c:3083:quota_parse_limits] 2-devstatic-quota: /:751619276800 [2013-12-12 20:24:24.607835] I [glusterfsd-mgmt.c:1584:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing Thu Dec 12 14:25:01 CST 2013 Here is the same event on 1 of my 4 bricks. [2013-12-12 20:24:10.142360] I [server.c:762:server_rpc_notify] 0-devstatic-server: disconnecting connectionfrom omhq1ce6-13589-2013/12/12-20:19:23:115673-devstatic-client-0-0 [2013-12-12 20:24:10.142392] I [server-helpers.c:729:server_connection_put] 0-devstatic-server: Shutting down connection omhq1ce6-13589-2013/12/12-20:19:23:115673-devstatic-client-0-0 [2013-12-12 20:24:10.142417] I [server-helpers.c:617:server_connection_destroy] 0-devstatic-server: destroyed connection of omhq1ce6-13589-2013/12/12-20:19:23:115673-devstatic-client-0-0 [2013-12-12 20:24:17.842368] I [server-handshake.c:567:server_setvolume] 0-devstatic-server: accepted client from omhq1ce6-13589-2013/12/12-20:19:23:115673-devstatic-client-0-2 (version: 3.4.1) [2013-12-12 20:24:20.147666] I [server.c:762:server_rpc_notify] 0-devstatic-server: disconnecting connectionfrom omhq1ce6-13589-2013/12/12-20:19:23:115673-devstatic-client-0-1 [2013-12-12 20:24:20.147690] I [server-helpers.c:729:server_connection_put] 0-devstatic-server: Shutting down connection omhq1ce6-13589-2013/12/12-20:19:23:115673-devstatic-client-0-1 [2013-12-12 20:24:20.147708] I [server-helpers.c:617:server_connection_destroy] 0-devstatic-server: destroyed connection of omhq1ce6-13589-2013/12/12-20:19:23:115673-devstatic-client-0-1 The product version of Red Hat Storage on which this issue was reported has reached End Of Life (EOL) [1], hence this bug report is being closed. If the issue is still observed on a current version of Red Hat Storage, please file a new bug report on the current version. [1] https://rhn.redhat.com/errata/RHSA-2014-0821.html The product version of Red Hat Storage on which this issue was reported has reached End Of Life (EOL) [1], hence this bug report is being closed. If the issue is still observed on a current version of Red Hat Storage, please file a new bug report on the current version. [1] https://rhn.redhat.com/errata/RHSA-2014-0821.html The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |