Bug 1076033

Summary:	Unknown Key: <bricks> are reported when the glusterd was restarted
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Rahul Hinduja <rhinduja>
Component:	glusterd	Assignee:	Kaushal <kaushal>
Status:	CLOSED ERRATA	QA Contact:	Byreddy <bsrirama>
Severity:	low	Docs Contact:
Priority:	low
Version:	rhgs-3.0	CC:	amukherj, asrivast, ggarg, kaushal, nlevinki, nsathyan, rcyriac, rhinduja, sasundar, vagarwal, vbellur, vmallika
Target Milestone:	---	Keywords:	ZStream
Target Release:	RHGS 3.1.1
Hardware:	x86_64
OS:	Linux
Whiteboard:	GlusterD
Fixed In Version:	glusterfs-3.7.1-12	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-10-05 07:07:34 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1251815

Description Rahul Hinduja 2014-03-13 12:08:01 UTC

Description of problem:
=======================

In a 2*2 clustered snapshot system, following errors were reported when the glusterd was restarted simultaneously on all the systems. 

[2014-03-13 04:58:46.748010] I [glusterd-store.c:2019:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 3
[2014-03-13 04:58:46.764780] E [glusterd-store.c:2567:glusterd_store_update_volinfo] 0-: Unknown key: brick-0
[2014-03-13 04:58:46.764844] E [glusterd-store.c:2567:glusterd_store_update_volinfo] 0-: Unknown key: brick-1
[2014-03-13 04:58:46.764873] E [glusterd-store.c:2567:glusterd_store_update_volinfo] 0-: Unknown key: brick-2
[2014-03-13 04:58:46.764897] E [glusterd-store.c:2567:glusterd_store_update_volinfo] 0-: Unknown key: brick-3
[2014-03-13 04:58:47.733494] E [glusterd-store.c:2567:glusterd_store_update_volinfo] 0-: Unknown key: brick-0
[2014-03-13 04:58:47.733595] E [glusterd-store.c:2567:glusterd_store_update_volinfo] 0-: Unknown key: brick-1
[2014-03-13 04:58:47.733622] E [glusterd-store.c:2567:glusterd_store_update_volinfo] 0-: Unknown key: brick-2
[2014-03-13 04:58:47.733644] E [glusterd-store.c:2567:glusterd_store_update_volinfo] 0-: Unknown key: brick-3
[2014-03-13 04:58:47.758419] E [glusterd-store.c:2567:glusterd_store_update_volinfo] 0-: Unknown key: brick-0
[2014-03-13 04:58:47.758459] E [glusterd-store.c:2567:glusterd_store_update_volinfo] 0-: Unknown key: brick-1
[2014-03-13 04:58:47.758490] E [glusterd-store.c:2567:glusterd_store_update_volinfo] 0-: Unknown key: brick-2
[2014-03-13 04:58:47.758514] E [glusterd-store.c:2567:glusterd_store_update_volinfo] 0-: Unknown key: brick-3
[2014-03-13 04:58:47.778857] E [glusterd-store.c:2567:glusterd_store_update_volinfo] 0-: Unknown key: brick-0
[2014-03-13 04:58:47.778899] E [glusterd-store.c:2567:glusterd_store_update_volinfo] 0-: Unknown key: brick-1
[2014-03-13 04:58:47.778926] E [glusterd-store.c:2567:glusterd_store_update_volinfo] 0-: Unknown key: brick-2

System had 188 snaps created from these bricks.



Version-Release number of selected component (if applicable):
=============================================================
glusterfs-3.4.1.2.snap.mar11.2014git-1.el6.x86_64


How reproducible:
=================
1/1


Steps to Reproduce:
1. Create multiple snaps for different volumes
2. Delete few snaps from different volume
3. Create the snaps with same name as deleted at step2
4. Restart the glusterd on all the machines simultaneously.

While the graphs are reloaded these error messages were logged.

Actual results:
===============

[2014-03-13 04:58:54.621217] E [glusterd-store.c:2567:glusterd_store_update_volinfo] 0-: Unknown key: brick-2
[2014-03-13 04:58:54.621244] E [glusterd-store.c:2567:glusterd_store_update_volinfo] 0-: Unknown key: brick-3
[2014-03-13 04:58:54.648320] E [glusterd-store.c:2567:glusterd_store_update_volinfo] 0-: Unknown key: brick-0
[2014-03-13 04:58:54.648436] E [glusterd-store.c:2567:glusterd_store_update_volinfo] 0-: Unknown key: brick-1



Expected results:
=================

No errors should be reported.

Comment 3 Vijaikumar Mallikarjuna 2014-03-14 12:16:15 UTC

This issue happens with regular volume as well, so it is not a snapshot bug

Comment 4 Kaushal 2014-03-21 05:31:58 UTC

This doesn't affect anything other than logs. This happens always when glusterd starts up if volumes are present. This is a bug with our volume options restore code. Since this doesn't affect the operation of glusterd, I say its of low severity and the priority could also be reduced.

Comment 6 Atin Mukherjee 2014-09-11 04:50:47 UTC

Marking priority as low as this bug doesn't affect any functionality

Comment 15 Atin Mukherjee 2015-08-12 04:37:19 UTC

Needinfo still applies as per comment 14

Comment 16 Gaurav Kumar Garg 2015-08-18 06:48:14 UTC

This issue was eventually fixed with some patch for RHGS 3.1.0 or much earlier. Hence moving this bug to ON_QA

Comment 17 Byreddy 2015-08-25 05:21:30 UTC

Verified this bug using the Version "glusterfs-3.7.1-12" with below steps.

Steps:
~~~~~~
1. Created multiple volumes (#4 Dis-Rep) using 2 node cluster.
2. Created multiple snaps for diff volumes.
3. Deleted fe snaps from diff volumes
4. Created snaps with same name as deleted as step-3
5. Restarted the glusterd on all the M/C's  simultaneously
6. Verified the glusterd log for the "unknow key" key // Not found

Fix is working. moving this Bug to verified state.

Comment 19 errata-xmlrpc 2015-10-05 07:07:34 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1845.html