+++ This bug was initially created as a clone of Bug #1544366 +++
Description of problem:
Was trying to test https://review.gluster.org/#/c/19538/ when I found that rolling upgrade from glusterfs-3.13 to the yet to be released glusterfs 4.0 is broken. In a 2 node setup, when one node is upgraded, clients mounted on each node can only see the local bricks and not the ones on the other node. I see the following errors in the client logs.:
E [MSGID: 114044] [client-handshake.c:1093:client_setvolume_cbk] 0-testvol-client-1: SETVOLUME on remote-host failed: lock state version not supplied [Invalid argument]
Steps to Reproduce:
1. Create a 2 node 1x2 volume and mount locally on each node, all on glustefs-3.13.
2. Upgrade one of the nodes to 4.0 branch
3. Clients can see only local bricks.
--- Additional comment from Anoop C S on 2018-02-12 17:12:17 IST ---
I think I failed to handle the scenario where clients are upgraded before servers.
RCA:(new clients[>=4.0] and old servers[<=3.13])
SETVOLUME request from client post-upgrade does not contain its lk-version in dictionary that is being passed onto server(https://review.gluster.org/#/c/12363/). This means that the server side check for "clnt-lk-version" inside the received dictionary would fail and error is returned back to new client.
--- Additional comment from Ravishankar N on 2018-02-12 19:37:17 IST ---
(In reply to Anoop C S from comment #1)
> I think I failed to handle the scenario where clients are upgraded before
FWIW, it is not just the mount but things like self-heal-daemon and glfsheal (gfapi based program that is used to display 'heal info') were also affected.
REVIEW: https://review.gluster.org/19560 (protcol/client: Insert dummy clnt-lk-version to avoid upgrade failure) posted (#2) for review on master by Anoop C S
COMMIT: https://review.gluster.org/19560 committed in master by "Amar Tumballi" <email@example.com> with a commit message- protcol/client: Insert dummy clnt-lk-version to avoid upgrade failure
With https://review.gluster.org/#/c/12363/ being merged, we no longer
send client's lk-version to server side and the corresponding check on
server is also removed. But when clients are upgraded prior to servers,
the check for lk-version at server side fails and is reported back to
clients resulting in disconnection.
Since we don't have lock-recovery (lk-version and grace-timeout) logic
anymore in code base our best bet would be to add client's default
lk-version i.e, 1, into the dictionary just to make server side check
pass and continue with remaining SETVOLUME operations.
Signed-off-by: Anoop C S <firstname.lastname@example.org>
REVIEW: https://review.gluster.org/19656 (protocol/server: Insert dummy clnt-lk-version to avoid upgrade failure) posted (#1) for review on master by Anoop C S
COMMIT: https://review.gluster.org/19656 committed in master by "Shyamsundar Ranganathan" <email@example.com> with a commit message- protocol/server: Insert dummy clnt-lk-version to avoid upgrade failure
This is required as we check for 'clnt-lk-version' in SETVOLUME callback
with older clients in place against newer servers. Change is similar to
what we have done via https://review.gluster.org/#/c/19560/.
Signed-off-by: Anoop C S <firstname.lastname@example.org>
REVIEW: https://review.gluster.org/19690 (protocol: Added iatt conversion to older format) posted (#1) for review on master by Shyamsundar Ranganathan
There was a further bug in the upgrade process, post any server is upgraded, directory creation from clients fail with EIO, but the directories are created on the bricks.
The issue was root caused to iatt corruption.
The reason for the corruption is that, DHT requests and receives iatt information in the dictionary. This is done for calls in (f)setxattr, (f)removexattr and unlink. The server brick process encodes this into the xdata dictionary in the new iatt format, whereas the client maps this to the older iatt structure, thus mangling the actual values (as the structure definition is not backward compatible).
The patch posted at comment #5 addresses this issue, by taking action in the protocol layer, to ensure we send back the right iatt format to the client based on the protocol dialect.
The downside of this approach is that, at a future time, we cannot add an option that can force a client to speak the 3.x dialect (akin to a mount option that can force the client to use the 3.x protocol version), as the client interpretation of the structure at that point would become mangled.
COMMIT: https://review.gluster.org/19690 committed in master by "Raghavendra G" <email@example.com> with a commit message- protocol: Added iatt conversion to older format
Added iatt conversion to an older format, when dealing with
older RPC versions. This enables iatt structure conformance
when dealing with older clients.
This helps fix rolling upgrade from 3.x versions to 4.0 version
of gluster by sending the right iatt in the dictionary when DHT
requests the same.
Signed-off-by: ShyamsundarR <firstname.lastname@example.org>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-v4.1.0, please open a new bug report.
glusterfs-v4.1.0 has been announced on the Gluster mailinglists , packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist  and the update infrastructure for your distribution.