Red Hat Bugzilla – Bug 1202388
[SNAPSHOT]: After a volume which has quota enabled is restored to a snap, attaching another node to the cluster is not successful
Last modified: 2016-09-17 08:54:14 EDT
Description of problem: ======================= After quota is enabled on the volume, restore the volume to any snapshot and then attach another node to the cluster. CLI shows the command is succesful but gluster peer status shows the newly added node is in 'Rejected' state Version-Release number of selected component (if applicable): ============================================================ glusterfs 3.6.0.51 How reproducible: ================ 2/2 Steps to Reproduce: ==================== 1.Create a 6x2 dist rep volume and start it 2.Fuse and NFS mount the volume enable quota and set server and client threads to 4 and 7 3.Create some files and take snapshot and activate it. Repeat the same steps at few intervals 4.Stop the volume and restore it to a particular snapshot. 5.Start the volume and access the files from the mount point. 6.Now add another node to the cluster (snapshot12.lab.eng.blr.redhat.com) [2015-03-16 13:30:10.157903] : peer probe snapshot12.lab.eng.blr.redhat.com : SUCCESS 7. Check gluster peer status from the node where the probe command was executed gluster peer status Number of Peers: 5 Hostname: rhs-arch-srv2.lab.eng.blr.redhat.com Uuid: 1db55b58-30d0-4da1-8a91-3b795a430446 State: Peer in Cluster (Connected) Hostname: rhs-arch-srv3.lab.eng.blr.redhat.com Uuid: b3dcf788-7e08-45d3-8343-f032cd5b41fc State: Peer in Cluster (Connected) Hostname: rhs-arch-srv4.lab.eng.blr.redhat.com Uuid: 12a1f3a6-4f96-4345-9198-bdc540792572 State: Peer in Cluster (Connected) Hostname: snapshot11.lab.eng.blr.redhat.com Uuid: e4509823-08ad-4e11-90db-38024fae926c State: Peer in Cluster (Connected) Hostname: snapshot12.lab.eng.blr.redhat.com Uuid: 3ade540a-ff00-4334-b07f-8d43ce92b07d State: Peer Rejected (Connected) -----------------------> shows Rejected state 8.The other nodes in the cluster do not show the newly added node in the cluster [root@rhs-arch-srv2 .unsupported]# gluster peer status Number of Peers: 4 Hostname: 10.70.34.50 Uuid: 520168e7-b2c4-49c7-98f0-96d0a3acbbc8 State: Peer in Cluster (Connected) Hostname: rhs-arch-srv3.lab.eng.blr.redhat.com Uuid: b3dcf788-7e08-45d3-8343-f032cd5b41fc State: Peer in Cluster (Connected) Hostname: rhs-arch-srv4.lab.eng.blr.redhat.com Uuid: 12a1f3a6-4f96-4345-9198-bdc540792572 State: Peer in Cluster (Connected) Hostname: snapshot11.lab.eng.blr.redhat.com Uuid: e4509823-08ad-4e11-90db-38024fae926c State: Peer in Cluster (Connected) 9.Check the peer status on the newly added node [root@snapshot12 ~]# gluster peer status Number of Peers: 1 Hostname: 10.70.34.50 Uuid: 520168e7-b2c4-49c7-98f0-96d0a3acbbc8 State: Peer Rejected (Connected) -------------------Part of the log from 10.70.34.50-------------------- [2015-03-16 13:30:10.371769] I [glusterd-handler.c:2320:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 3ade540a-ff00-4334-b07f-8d43ce92b07d [2015-03-16 13:30:10.372065] E [MSGID: 106012] [glusterd-utils.c:3531:glusterd_compare_friend_volume] 0-management: Cksums of quota configuration of volume vol0 differ. local cksum = 0, remote cksum = 1271429248 on peer snapshot12.lab.e ng.blr.redhat.com [2015-03-16 13:30:10.372097] I [glusterd-handler.c:3372:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to snapshot12.lab.eng.blr.redhat.com (0), ret: 0 -------------------------------------------------------------------- Actual results: =============== After volume which has quota enabled is restored to a snapshot, attaching another node to the cluster is not successful Expected results: ================ Attaching the node should be successful Additional info: =============== [root@inception ~]# gluster v i vol0 Volume Name: vol0 Type: Distributed-Replicate Volume ID: b1ae2f52-4161-4200-a639-ba2c5aeeb8c5 Status: Started Snap Volume: no Number of Bricks: 6 x 2 = 12 Transport-type: tcp Bricks: Brick1: inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/64e656d56fac44dcaf1dda4ed0b10dc6/brick1/b1 Brick2: rhs-arch-srv2.lab.eng.blr.redhat.com:/var/run/gluster/snaps/64e656d56fac44dcaf1dda4ed0b10dc6/brick2/b1 Brick3: rhs-arch-srv3.lab.eng.blr.redhat.com:/var/run/gluster/snaps/64e656d56fac44dcaf1dda4ed0b10dc6/brick3/b1 Brick4: rhs-arch-srv4.lab.eng.blr.redhat.com:/var/run/gluster/snaps/64e656d56fac44dcaf1dda4ed0b10dc6/brick4/b1 Brick5: inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/64e656d56fac44dcaf1dda4ed0b10dc6/brick5/b2 Brick6: rhs-arch-srv2.lab.eng.blr.redhat.com:/var/run/gluster/snaps/64e656d56fac44dcaf1dda4ed0b10dc6/brick6/b2 Brick7: rhs-arch-srv3.lab.eng.blr.redhat.com:/var/run/gluster/snaps/64e656d56fac44dcaf1dda4ed0b10dc6/brick7/b2 Brick8: rhs-arch-srv4.lab.eng.blr.redhat.com:/var/run/gluster/snaps/64e656d56fac44dcaf1dda4ed0b10dc6/brick8/b2 Brick9: inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/64e656d56fac44dcaf1dda4ed0b10dc6/brick9/b3 Brick10: rhs-arch-srv2.lab.eng.blr.redhat.com:/var/run/gluster/snaps/64e656d56fac44dcaf1dda4ed0b10dc6/brick10/b3 Brick11: rhs-arch-srv3.lab.eng.blr.redhat.com:/var/run/gluster/snaps/64e656d56fac44dcaf1dda4ed0b10dc6/brick11/b3 Brick12: rhs-arch-srv4.lab.eng.blr.redhat.com:/var/run/gluster/snaps/64e656d56fac44dcaf1dda4ed0b10dc6/brick12/b3 Options Reconfigured: client.event-threads: 4 server.event-threads: 7 features.uss: enable features.quota: on performance.readdir-ahead: on auto-delete: disable snap-max-soft-limit: 90 snap-max-hard-limit: 256
Could we please have the RCA for this bug updated
If quota is enabled on the volume and if it is restored, then we cannot add another node to the cluster.
RCA : When we enable quota on a volume, the quota cksum file is created. and is present in /var/lib/glusterd/vols/<volname>/, and the in memory structure of volinfo is updated as well. When we take a snapshot we copy the quota cksum file and along with that we take a copy of in memory structure. During restore we are missing out to fill in the quota cksum value. Hence after we restore the snapshot the cksum is updated to 0. When we try to add a new node, the quota cksum mismatches and because of that we fail to add a new node.
While trying out a workaround for this problem. I have hit any interesting issue. The issue is as follows: 1) Let us say we have 2 nodes and a volume (vol1) is spread across these 2 nodes. 2) We take a snapshot 3) We set some options on the volume "vol1" [a] Volume options are persisted in /var/lib/glusterd/vols/<volname>/info [b] snap volume info is saved in /var/lib/glusterd/snaps/<snapvol>/info * Newly added option should be saved in [a] (mentioned above) 4) Peer probe to a new node. 5) Now check the "info file of snap volume (i.e mentioned in point 3.b) * Info file of snap volume contains the newly set option. Ideally the snap volume should not contain that option, as it is set after snapshot create. * Because of this there will be a mismatch of volume checksum too. * Until and unless the glusterd does not restart we are safe. Once glusterd is restarted on the new node, because of checksum mismatch the newly added node goes into a peer reject state.
There are 2 separate problems for this bug 1) After taking a snapshot if we set any volume option and if we add a new node, then ideally the new node should have the same "info" file of snap-volume compared to existing nodes. But, the problem here is along with existing "info" file contents, the new volume option is also being updated in "info" file of snap-volume, which should not be the case. Because of this checksum mismatch happens and peer goes into rejected state. * Workaround here would be that, after snapshot restore, we need to copy the /var/lib/glusterd/vols/<volname>/info file from existing node to new node and restart the glusterd on new node. 2) Let us imagine we have quota enabled on the volume and we take snapshot. During snapshot restore we fail to re-calculate the quota checksum and because of that quota checksum is initialized to zero. If we try to add a new node then quota_conf checksum mismatches and because of that peer will go into a rejected state * Workaround here would be that after snapshot restore, the glusterd needs to be restarted on existing nodes and then a peer probe has to be done on new node.
Hi Rajesh, I see this bug being added as a known issue for the 3.0.4 release. Please fill out the doc text.
Hi Rajesh, Can you please review the edited doc text for technical accuracy and sign off?
doc-text seems fine to me.
Workaround mentioned in Comment 7 is tested and works fine.
root@darkknightrises cron.d]# gluster snapshot create snap3 vol0 snapshot create: success: Snap snap3_GMT-2015.07.04-07.19.09 created successfully [root@darkknightrises cron.d]# gluster snapshot activate snap3_GMT-2015.07.04-07.19.09 Snapshot activate: snap3_GMT-2015.07.04-07.19.09: Snap activated successfully [root@darkknightrises cron.d]# gluster v stop vol0 Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y volume stop: vol0: success [root@darkknightrises cron.d]# gluster snapshot restore snap3_GMT-2015.07.04-07.19.09 Restore operation will replace the original volume with the snapshotted volume. Do you still want to continue? (y/n) y Snapshot restore: snap3_GMT-2015.07.04-07.19.09: Snap restored successfully [root@darkknightrises cron.d]# gluster v start vol0 volume start: vol0: success [root@darkknightrises cron.d]# gluster peer probe 10.70.47.151 peer probe: success. [root@darkknightrises cron.d]# gluster peer status Number of Peers: 4 Hostname: 10.70.33.219 Uuid: e91bd1ef-38d5-4389-8db8-2a6528ccbb17 State: Peer in Cluster (Connected) Hostname: 10.70.44.13 Uuid: 7e6e250c-b14d-4850-96c6-2a194a47b90e State: Peer in Cluster (Connected) Hostname: 10.70.33.225 Uuid: 0e2626da-55f5-4523-95c0-55eefb4e53a3 State: Peer in Cluster (Connected) Hostname: 10.70.47.151 Uuid: 6f5126cf-312a-4274-8a09-380dd6d465ad State: Peer in Cluster (Connected) Bug verified on build glusterfs-3.7.1-7.el6rhs.x86_64.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-1495.html