+++ This bug was initially created as a clone of Bug #1544699 +++ +++ This bug was initially created as a clone of Bug #1544366 +++ Description of problem: Was trying to test https://review.gluster.org/#/c/19538/ when I found that rolling upgrade from glusterfs-3.13 to the yet to be released glusterfs 4.0 is broken. In a 2 node setup, when one node is upgraded, clients mounted on each node can only see the local bricks and not the ones on the other node. I see the following errors in the client logs.: E [MSGID: 114044] [client-handshake.c:1093:client_setvolume_cbk] 0-testvol-client-1: SETVOLUME on remote-host failed: lock state version not supplied [Invalid argument] Steps to Reproduce: 1. Create a 2 node 1x2 volume and mount locally on each node, all on glustefs-3.13. 2. Upgrade one of the nodes to 4.0 branch 3. Clients can see only local bricks. --- Additional comment from Anoop C S on 2018-02-12 17:12:17 IST --- I think I failed to handle the scenario where clients are upgraded before servers. RCA:(new clients[>=4.0] and old servers[<=3.13]) SETVOLUME request from client post-upgrade does not contain its lk-version in dictionary that is being passed onto server(https://review.gluster.org/#/c/12363/). This means that the server side check for "clnt-lk-version" inside the received dictionary would fail and error is returned back to new client. --- Additional comment from Ravishankar N on 2018-02-12 19:37:17 IST --- (In reply to Anoop C S from comment #1) > I think I failed to handle the scenario where clients are upgraded before > servers. > FWIW, it is not just the mount but things like self-heal-daemon and glfsheal (gfapi based program that is used to display 'heal info') were also affected.
REVIEW: https://review.gluster.org/19663 (protocol/server: Insert dummy clnt-lk-version to avoid upgrade failure) posted (#1) for review on release-4.0 by Shyamsundar Ranganathan
COMMIT: https://review.gluster.org/19663 committed in release-4.0 by "Shyamsundar Ranganathan" <srangana> with a commit message- protocol/server: Insert dummy clnt-lk-version to avoid upgrade failure This is required as we check for 'clnt-lk-version' in SETVOLUME callback with older clients in place against newer servers. Change is similar to what we have done via https://review.gluster.org/#/c/19560/. (cherry picked from commit fecb0fc748806d4e6d61bcbef976acf473e55c82) Change-Id: If333c20cf9503f40687ec926c44c7e50222c05b5 BUG: 1551112 Signed-off-by: Anoop C S <anoopcs>
There is a further bug in the upgrade process, post any server is upgraded, directory creation from clients fail with EIO, but the directories are created on the bricks. The issue was root caused to iatt corruption. The reason for the corruption is that, DHT requests and receives iatt information in the dictionary. This is done for calls in (f)setxattr, (f)removexattr and unlink. The server brick process encodes this into the xdata dictionary in the new iatt format, whereas the client maps this to the older iatt structure, thus mangling the actual values (as the structure definition is not backward compatible). A patch is posted against master at https://review.gluster.org/#/c/19690/1 that addresses this issue, by taking action in the protocol layer, to ensure we send back the right iatt format to the client based on the protocol dialect. The downside of this approach is that, at a future time, we cannot add an option that can force a client to speak the 3.x dialect (akin to a mount option that can force the client to use the 3.x protocol version), as the client interpretation of the structure at that point would become mangled.
REVIEW: https://review.gluster.org/19694 (protocol: Added iatt conversion to older format) posted (#1) for review on release-4.0 by Shyamsundar Ranganathan
COMMIT: https://review.gluster.org/19694 committed in release-4.0 by "Raghavendra G" <rgowdapp> with a commit message- protocol: Added iatt conversion to older format Added iatt conversion to an older format, when dealing with older RPC versions. This enables iatt structure conformance when dealing with older clients. This helps fix rolling upgrade from 3.x versions to 4.0 version of gluster by sending the right iatt in the dictionary when DHT requests the same. (cherry picked from commit b966c7790e35de353ae09ee48d4e2f55e0117f7e) Change-Id: Ieaf925f81f8c7798a8fba1e90a59fa9dec82856c BUG: 1551112 Signed-off-by: ShyamsundarR <srangana>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-4.0.0, please open a new bug report. glusterfs-4.0.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/announce/2018-March/000092.html [2] https://www.gluster.org/pipermail/gluster-users/