+++ This bug was initially created as a clone of Bug #1544366 +++ Description of problem: Was trying to test https://review.gluster.org/#/c/19538/ when I found that rolling upgrade from glusterfs-3.13 to the yet to be released glusterfs 4.0 is broken. In a 2 node setup, when one node is upgraded, clients mounted on each node can only see the local bricks and not the ones on the other node. I see the following errors in the client logs.: E [MSGID: 114044] [client-handshake.c:1093:client_setvolume_cbk] 0-testvol-client-1: SETVOLUME on remote-host failed: lock state version not supplied [Invalid argument] Steps to Reproduce: 1. Create a 2 node 1x2 volume and mount locally on each node, all on glustefs-3.13. 2. Upgrade one of the nodes to 4.0 branch 3. Clients can see only local bricks. --- Additional comment from Anoop C S on 2018-02-12 17:12:17 IST --- I think I failed to handle the scenario where clients are upgraded before servers. RCA:(new clients[>=4.0] and old servers[<=3.13]) SETVOLUME request from client post-upgrade does not contain its lk-version in dictionary that is being passed onto server(https://review.gluster.org/#/c/12363/). This means that the server side check for "clnt-lk-version" inside the received dictionary would fail and error is returned back to new client. --- Additional comment from Ravishankar N on 2018-02-12 19:37:17 IST --- (In reply to Anoop C S from comment #1) > I think I failed to handle the scenario where clients are upgraded before > servers. > FWIW, it is not just the mount but things like self-heal-daemon and glfsheal (gfapi based program that is used to display 'heal info') were also affected.
REVIEW: https://review.gluster.org/19560 (protcol/client: Insert dummy clnt-lk-version to avoid upgrade failure) posted (#2) for review on master by Anoop C S
COMMIT: https://review.gluster.org/19560 committed in master by "Amar Tumballi" <amarts> with a commit message- protcol/client: Insert dummy clnt-lk-version to avoid upgrade failure With https://review.gluster.org/#/c/12363/ being merged, we no longer send client's lk-version to server side and the corresponding check on server is also removed. But when clients are upgraded prior to servers, the check for lk-version at server side fails and is reported back to clients resulting in disconnection. Since we don't have lock-recovery (lk-version and grace-timeout) logic anymore in code base our best bet would be to add client's default lk-version i.e, 1, into the dictionary just to make server side check pass and continue with remaining SETVOLUME operations. Change-Id: I441b67bd271d1e9ba9a7c08703e651c7a6bd945b BUG: 1544699 Signed-off-by: Anoop C S <anoopcs>
REVIEW: https://review.gluster.org/19656 (protocol/server: Insert dummy clnt-lk-version to avoid upgrade failure) posted (#1) for review on master by Anoop C S
COMMIT: https://review.gluster.org/19656 committed in master by "Shyamsundar Ranganathan" <srangana> with a commit message- protocol/server: Insert dummy clnt-lk-version to avoid upgrade failure This is required as we check for 'clnt-lk-version' in SETVOLUME callback with older clients in place against newer servers. Change is similar to what we have done via https://review.gluster.org/#/c/19560/. Change-Id: If333c20cf9503f40687ec926c44c7e50222c05b5 BUG: 1544699 Signed-off-by: Anoop C S <anoopcs>
REVIEW: https://review.gluster.org/19690 (protocol: Added iatt conversion to older format) posted (#1) for review on master by Shyamsundar Ranganathan
There was a further bug in the upgrade process, post any server is upgraded, directory creation from clients fail with EIO, but the directories are created on the bricks. The issue was root caused to iatt corruption. The reason for the corruption is that, DHT requests and receives iatt information in the dictionary. This is done for calls in (f)setxattr, (f)removexattr and unlink. The server brick process encodes this into the xdata dictionary in the new iatt format, whereas the client maps this to the older iatt structure, thus mangling the actual values (as the structure definition is not backward compatible). The patch posted at comment #5 addresses this issue, by taking action in the protocol layer, to ensure we send back the right iatt format to the client based on the protocol dialect. The downside of this approach is that, at a future time, we cannot add an option that can force a client to speak the 3.x dialect (akin to a mount option that can force the client to use the 3.x protocol version), as the client interpretation of the structure at that point would become mangled.
COMMIT: https://review.gluster.org/19690 committed in master by "Raghavendra G" <rgowdapp> with a commit message- protocol: Added iatt conversion to older format Added iatt conversion to an older format, when dealing with older RPC versions. This enables iatt structure conformance when dealing with older clients. This helps fix rolling upgrade from 3.x versions to 4.0 version of gluster by sending the right iatt in the dictionary when DHT requests the same. Change-Id: Ieaf925f81f8c7798a8fba1e90a59fa9dec82856c BUG: 1544699 Signed-off-by: ShyamsundarR <srangana>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-v4.1.0, please open a new bug report. glusterfs-v4.1.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/announce/2018-June/000102.html [2] https://www.gluster.org/pipermail/gluster-users/