Bug 1816965

Summary:	Cluster state does not manage to reach 100% during an upgrade from 3.11.117 to 3.11.157
Product:	OpenShift Container Platform	Reporter:	Saurabh Sadhale <ssadhale>
Component:	Logging	Assignee:	Jeff Cantrill <jcantril>
Status:	CLOSED DUPLICATE	QA Contact:	Anping Li <anli>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	3.11.0	CC:	aos-bugs
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-03-25 12:48:30 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Saurabh Sadhale 2020-03-25 09:36:51 UTC

Description of problem:

After upgrading from 3.11.117 to 3.11.157 when attempting to upgrade the logging stack the cluster fails to reach the 100% active shards percentage. 

Memory of ES pod --> 55 GB per pod 
CPU of ES pod --> 4 cores per ES pod 
PRIMARY_SHARDS -- > 1
REPLICA_SHARDS -- > 2

The solution applied for solving the problem and achieve the 100% cluster state was to set all the shards which are in yellow state to one replica. This way there are no more shards to sync and the percentage goes back to 100% and the cluster turn to green state so the next member ( ES node ) can be redeployed by the playbook. There are no shards in RED state. 

Version-Release number of selected component (if applicable):


How reproducible:
Upgrade the EFK stack from 3.11.117 to 3.11.157

Steps to Reproduce:
1.
2.
3.

Actual results:
- The cluster percentage shows 99.92128197919595 
- Some shards remain in yellow state
Expected results:
- The cluster percentage should be 100%
- No shards remain in yellow state or RED state. 

Logs show that there is a version mismatch between nodes. 

~~~
"target node version [5.6.13] is older than the source node version [5.6.16]"
~~~


Additional info:

Comment 1 Jeff Cantrill 2020-03-25 12:48:30 UTC


*** This bug has been marked as a duplicate of bug 1816966 ***