Bug 1044867

Summary:	[SNAPSHOT]: once all the gluster process are killed and restarted, the snapshot list for a volume shows double of the snaps taken
Product:	Red Hat Gluster Storage	Reporter:	Rahul Hinduja <rhinduja>
Component:	snapshot	Assignee:	rjoseph
Status:	CLOSED ERRATA	QA Contact:	Rahul Hinduja <rhinduja>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	rhgs-3.0	CC:	nsathyan, rhs-bugs, sdharane, senaik, spandit, ssamanta, storage-qa-internal
Target Milestone:	---
Target Release:	RHGS 3.0.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	SNAPSHOT
Fixed In Version:	glusterfs-3.4.0.snap.dec30.2013	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-09-22 19:30:50 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Rahul Hinduja 2013-12-19 07:10:21 UTC

Description of problem:
=======================

Initially 10 snaps were taken for a volume, but when all the gluster processes were killed using (killall glusterd glusterfs glusterfsd) and restarted the glusterd using "service glusterd start". The number of snaps taken for a volume is shown as 20 from a node where the gluster processes were killed, still shows 10 from other nodes in cluster.

Initial snapshot list for a volume:
===================================

[root@snapshot-10 ~]# gluster snapshot list vol-snap4

Volume Name : vol-snap4
Number of snaps taken : 10
Number of snaps available : 246

	Snap Name : snap11
	Snap Time : 2013-12-18 09:45:29
	Snap ID : 87a9ecd2-6e88-42f3-9e9d-b8d77662f1de

	Snap Name : snap12
	Snap Time : 2013-12-18 09:45:31
	Snap ID : f54ea806-51ef-42ad-a6a6-3ab014cc68f9

	Snap Name : snap13
	Snap Time : 2013-12-18 09:45:34
	Snap ID : 9de95f5a-93ac-48d9-afad-756f0302a681

	Snap Name : snap14
	Snap Time : 2013-12-18 09:45:36
	Snap ID : 7e670ee2-f450-4545-b4d7-42f4600c9e26

	Snap Name : snap15
	Snap Time : 2013-12-18 09:45:40
	Snap ID : 6df1143b-4b79-497a-b2e3-11a866039018

	Snap Name : snap16
	Snap Time : 2013-12-18 09:45:43
	Snap ID : 63e7efd3-9320-4abe-af5f-13aa9a72e64f

	Snap Name : snap17
	Snap Time : 2013-12-18 09:45:46
	Snap ID : 1db0f060-4563-4b6a-9588-b9a0f43106e1

	Snap Name : snap18
	Snap Time : 2013-12-18 09:45:49
	Snap ID : 8d3b658e-358e-44e3-ba0e-03e2f04ac06a

	Snap Name : snap19
	Snap Time : 2013-12-18 09:45:53
	Snap ID : a7d0cbcf-39ab-4515-bcd8-4a5c5ba887ac

	Snap Name : snap20
	Snap Time : 2013-12-18 09:45:56
	Snap ID : d0419a96-99c1-4d7d-86b2-f7e068f38425
[root@snapshot-10 ~]# 


Killed all the gluster processes:
=================================

[root@snapshot-10 ~]# killall glusterd glusterfsd glusterfs

Started the glusterd process:
=============================

[root@snapshot-10 ~]# service glusterd start
Starting glusterd:                                         [  OK  ]
[root@snapshot-10 ~]#

Now listing snapshot from the node shows snaps taken as 20 but in actual lists 10:
===============================================================================

[root@snapshot-10 ~]# gluster snapshot list vol-snap4

Volume Name : vol-snap4
Number of snaps taken : 20
Number of snaps available : 236

	Snap Name : snap11
	Snap Time : 1970-01-01 05:30:00
	Snap ID : 87a9ecd2-6e88-42f3-9e9d-b8d77662f1de

	Snap Name : snap12
	Snap Time : 1970-01-01 05:30:00
	Snap ID : f54ea806-51ef-42ad-a6a6-3ab014cc68f9

	Snap Name : snap13
	Snap Time : 1970-01-01 05:30:00
	Snap ID : 9de95f5a-93ac-48d9-afad-756f0302a681

	Snap Name : snap14
	Snap Time : 1970-01-01 05:30:00
	Snap ID : 7e670ee2-f450-4545-b4d7-42f4600c9e26

	Snap Name : snap15
	Snap Time : 1970-01-01 05:30:00
	Snap ID : 6df1143b-4b79-497a-b2e3-11a866039018

	Snap Name : snap16
	Snap Time : 1970-01-01 05:30:00
	Snap ID : 63e7efd3-9320-4abe-af5f-13aa9a72e64f

	Snap Name : snap17
	Snap Time : 1970-01-01 05:30:00
	Snap ID : 1db0f060-4563-4b6a-9588-b9a0f43106e1

	Snap Name : snap18
	Snap Time : 1970-01-01 05:30:00
	Snap ID : 8d3b658e-358e-44e3-ba0e-03e2f04ac06a

	Snap Name : snap19
	Snap Time : 1970-01-01 05:30:00
	Snap ID : a7d0cbcf-39ab-4515-bcd8-4a5c5ba887ac

	Snap Name : snap20
	Snap Time : 1970-01-01 05:30:00
	Snap ID : d0419a96-99c1-4d7d-86b2-f7e068f38425
[root@snapshot-10 ~]# 


From other cluster nodes it shows correct:
==========================================

[root@snapshot-09 ~]# gluster snapshot list vol-snap4

Volume Name : vol-snap4
Number of snaps taken : 10
Number of snaps available : 246

	Snap Name : snap11
	Snap Time : 2013-12-18 09:45:28
	Snap ID : 54b1f750-ce99-482a-84da-cdf78e786223

	Snap Name : snap12
	Snap Time : 2013-12-18 09:45:31
	Snap ID : be412ce6-eaaf-4c33-8948-002ba7126106

	Snap Name : snap13
	Snap Time : 2013-12-18 09:45:33
	Snap ID : af02dfce-a92f-4cb1-919e-969179ad5eac

	Snap Name : snap14
	Snap Time : 2013-12-18 09:45:36
	Snap ID : 26682c6b-92ea-4e13-a7e8-8632fed6db5d

	Snap Name : snap15
	Snap Time : 2013-12-18 09:45:39
	Snap ID : 2e6c7960-5e29-4c9c-a6e3-dea957177d67

	Snap Name : snap16
	Snap Time : 2013-12-18 09:45:42
	Snap ID : a866fcb1-44ed-4c87-8a7b-fa045fdcd996

	Snap Name : snap17
	Snap Time : 2013-12-18 09:45:45
	Snap ID : 30c6338c-826a-43a1-bdff-fb793f92ce04

	Snap Name : snap18
	Snap Time : 2013-12-18 09:45:49
	Snap ID : cfd40552-2e07-4f62-89e0-5a17f9ee67a7

	Snap Name : snap19
	Snap Time : 2013-12-18 09:45:52
	Snap ID : b41ee46b-4ecf-49a9-aa3a-53530a3011b9

	Snap Name : snap20
	Snap Time : 2013-12-18 09:45:56
	Snap ID : b2513ce7-c151-4ef0-bdd3-edca1c0fb417
[root@snapshot-09 ~]# 


Version-Release number of selected component (if applicable):
=============================================================

glusterfs-3.4.0.snap.dec03.2013git-1.el6.x86_64



Steps to Reproduce:
===================
1. Create snaps of a given volume
2. killall glusterd glusterfsd glusterfs
3. start glusterd service using "service glusterd start"
4. list the number of snaps using "gluster snapshot list vol-name"

Actual results:
===============

Volume Name : vol-snap4
Number of snaps taken : 20
Number of snaps available : 236

Expected results:
=================

Volume Name : vol-snap4
Number of snaps taken : 10
Number of snaps available : 246

Comment 2 Sachin Pandit 2013-12-30 09:58:57 UTC

The snapcount was updated twice during the restart, This
issue is fixed in the recently sent Build.

Comment 3 Rahul Hinduja 2013-12-30 12:17:48 UTC

Verified with the build: glusterfs-3.4.0.snap.dec30.2013git-1.el6.x86_64

Number of snaps shown is correct after glusterd restart.

[root@snapshot-09 ~]# gluster snapshot list | grep "Number of snaps taken"
Number of snaps taken : 10
Number of snaps taken : 10
Number of snaps taken : 10
Number of snaps taken : 10
Number of snaps taken : 10
[root@snapshot-09 ~]# gluster snapshot list | grep "Number of snaps available"
Number of snaps available : 246
Number of snaps available : 246
Number of snaps available : 246
Number of snaps available : 246
Number of snaps available : 246
[root@snapshot-09 ~]#


[root@snapshot-10 ~]# gluster snapshot list | grep "Number of snaps taken"
Number of snaps taken : 10
Number of snaps taken : 10
Number of snaps taken : 10
Number of snaps taken : 10
Number of snaps taken : 10
[root@snapshot-10 ~]# gluster snapshot list | grep "Number of snaps available"
Number of snaps available : 246
Number of snaps available : 246
Number of snaps available : 246
Number of snaps available : 246
Number of snaps available : 246
[root@snapshot-10 ~]# 


[root@snapshot-11 ~]# gluster snapshot list | grep "Number of snaps taken"
Number of snaps taken : 10
Number of snaps taken : 10
Number of snaps taken : 10
Number of snaps taken : 10
Number of snaps taken : 10
[root@snapshot-11 ~]#  gluster snapshot list | grep "Number of snaps available"
Number of snaps available : 246
Number of snaps available : 246
Number of snaps available : 246
Number of snaps available : 246
Number of snaps available : 246
[root@snapshot-11 ~]# 


[root@snapshot-12 ~]# gluster snapshot list | grep "Number of snaps taken"
Number of snaps taken : 10
Number of snaps taken : 10
Number of snaps taken : 10
Number of snaps taken : 10
Number of snaps taken : 10
[root@snapshot-12 ~]#  gluster snapshot list | grep "Number of snaps available"
Number of snaps available : 246
Number of snaps available : 246
Number of snaps available : 246
Number of snaps available : 246
Number of snaps available : 246
[root@snapshot-12 ~]# 

Moving the bug to verified state.

Comment 5 Nagaprasad Sathyanarayana 2014-04-21 06:17:30 UTC

Marking snapshot BZs to RHS 3.0.

Comment 6 Nagaprasad Sathyanarayana 2014-05-19 10:56:41 UTC

Setting flags required to add BZs to RHS 3.0 Errata

Comment 9 errata-xmlrpc 2014-09-22 19:30:50 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2014-1278.html