Bug 1959985

Summary:	[GSS][mon] rook-operator scales mons to 4 after healthCheck timeout
Product:	[Red Hat Storage] Red Hat OpenShift Container Storage	Reporter:	Travis Nielsen <tnielsen>
Component:	rook	Assignee:	Travis Nielsen <tnielsen>
Status:	CLOSED ERRATA	QA Contact:	Shrivaibavi Raghaventhiran <sraghave>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.6	CC:	asriram, bkunal, ebenahar, madam, muagarwa, ocs-bugs, r.martinez, sapillai, tdesala
Target Milestone:	---	Keywords:	AutomationBackLog, ZStream
Target Release:	OCS 4.7.1
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	4.7.1-403.ci	Doc Type:	Bug Fix
Doc Text:	Earlier, if the operator was restarted during a mon failover, the operator could erroneously remove the new mon. Hence, the mon quorum was at risk when the operator removed the new mon. With this update, the operator will restore the state when the mon failover is in progress and properly complete the mon failover after the operator is restarted. Now, the mon quorum is more reliable in the node drains and mon failover scenarios.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-06-15 16:50:37 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1955831
Bug Blocks:

Comment 8 Shrivaibavi Raghaventhiran 2021-06-03 11:22:02 UTC

Platform:
----------

Vmware 3M 3W RHCOS cluster

Versions:
----------

OCP - 4.7.12
OCS - ocs-operator.v4.7.1-403.ci

Testcases Executed:
----------------------

1.  Perform single node drain and Restart rook-ceph operator during mon failover to happen
a. Wait for >= 20 mins and restart the rook-ceph operator
b. Uncordon drained node after 20 mins
c. Check for mons running in healthy state and no mons should be observed in pending state post recovery of node

2. Delete the mon deployment
a. Create the cluster, wait for it to be initially deployed
b. Scale down a mon (e.g. mon-a) so it falls out of quorum
c. Wait for the mon failover to be initiated (10 min)
d. As soon as the new mon is created and before the bad mon deployment
(rook-ceph-mon-a) is deleted, restart the operator
e. All 3 mons will be running

Restarted rook-ceph operator just during the failover
I did not see 4 mons, I always saw 3 mons, Its working as expected. we dont see 2 pending Mons

@santosh let me know if this BZ needs any more verifications or if it can be moved to verified state

Comment 9 Santosh Pillai 2021-06-03 15:02:41 UTC

(In reply to Shrivaibavi Raghaventhiran from comment #8)

> @santosh let me know if this BZ needs any more verifications or if it can be
> moved to verified state

Looks good to me. Can be moved to verified state.

Comment 10 Shrivaibavi Raghaventhiran 2021-06-03 16:04:22 UTC

Based on comment 8 and comment 9 moving the BZ to verified state

Comment 12 Travis Nielsen 2021-06-08 17:34:01 UTC

needs info separately answered between comments 7 and 8

Comment 16 errata-xmlrpc 2021-06-15 16:50:37 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenShift Container Storage 4.7.1 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2449