Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1838153

Summary: The ZeroRedundancy is not honored during upgrade clo-first
Product: OpenShift Container Platform Reporter: Anping Li <anli>
Component: LoggingAssignee: Jeff Cantrill <jcantril>
Status: CLOSED ERRATA QA Contact: Anping Li <anli>
Severity: medium Docs Contact:
Priority: low    
Version: 4.5CC: aos-bugs, ewolinet, lvlcek
Target Milestone: ---   
Target Release: 4.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-27 16:00:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1843738    

Description Anping Li 2020-05-20 14:33:17 UTC 
Description of problem:
The ZeroRedundancy is not honored during the upgrade. Result in unassigned Replicas shards. If the nodeCount=1, the ES status will always be Yellow.

oc exec -c elasticsearch elasticsearch-cdm-6hyfcqb1-1-576c48f45-lm9dm -- es_util --query=_cat/shards
.kibana.647a750f1787408bf50088234ec0edd5a6a9b2ac                 0 p STARTED         3  37.5kb 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
.kibana                                                          0 p STARTED         1   3.2kb 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
project.logjsonx.f1301ba5-3a77-431e-b383-117619c14be9.2020.05.20 0 p STARTED      1658   1.9mb 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
project.logflatx.17244a69-6d18-4594-a99b-8af3efc55908.2020.05.20 0 p STARTED      1654   2.1mb 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
.searchguard                                                     0 p STARTED         5 145.7kb 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
app-write                                                        1 p STARTED         0    162b 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
app-write                                                        1 r UNASSIGNED                            
app-write                                                        2 p STARTED         0    162b 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
app-write                                                        2 r UNASSIGNED                            
app-write                                                        4 p STARTED         0    162b 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
app-write                                                        4 r UNASSIGNED                            
app-write                                                        3 p STARTED         0    162b 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
app-write                                                        3 r UNASSIGNED                            
app-write                                                        0 p STARTED         0    162b 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
app-write                                                        0 r UNASSIGNED                            
.operations.2020.05.20                                           0 p STARTED    562893 574.2mb 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
infra-write                                                      1 p STARTED         0    162b 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
infra-write                                                      1 r UNASSIGNED                            
infra-write                                                      2 p STARTED         0    162b 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
infra-write                                                      2 r UNASSIGNED                            
infra-write                                                      4 p STARTED         0    162b 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
infra-write                                                      4 r UNASSIGNED                            
infra-write                                                      3 p STARTED         0    162b 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
infra-write                                                      3 r UNASSIGNED                            
infra-write                                                      0 p STARTED         0    162b 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
infra-write                                                      0 r UNASSIGNED             

Version-Release number of selected component (if applicable):
4.4->4.5

How reproducible:
Always

Steps to Reproduce:
1. Install 4.4 using nodeCount=1, ZeroRedundancy
2. Upgrade CLO to 4.5
3. Waiting for a while, check the indices in Elaticsearch.
4. Upgrade EO to 4.5

Actual results:
UNASSIGNED Replica shards.

Expected results:

In clusterlogging-4.4.z, Add infra-write and infra-app into the index template.


Additional info:
Comment 1 Anping Li 2020-05-20 15:37:02 UTC 
I can reproduce it.  Tt is strange, when nodeCount=2, redundancyPolicy: "ZeroRedundancy",  No Replicas shards.
Comment 2 Anping Li 2020-05-23 12:26:33 UTC 
Interestingly, After upgrade,  the ES doesn't allocate replicas to the indices infra-write and app-write any more.

+ oc exec -c elasticsearch elasticsearch-cdm-xv9zo8gz-1-6fcc5d4998-lpb9g -- es_util --query=_cat/shards
project.logflatx.b1c056f0-4405-45bd-8cea-76338862d9ed.2020.05.23 0 p STARTED   746   543kb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
.security                                                        0 p STARTED     5  30.8kb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
project.logjsonx.9b993d19-0818-4232-8940-cf06a750e965.2020.05.23 0 p STARTED   746 485.3kb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
app-write                                                        1 p STARTED   182   414kb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
app-write                                                        4 p STARTED   166 269.1kb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
app-write                                                        2 p STARTED   171 317.8kb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
app-write                                                        3 p STARTED   165 331.2kb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
app-write                                                        0 p STARTED   176 259.9kb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
.kibana.647a750f1787408bf50088234ec0edd5a6a9b2ac                 0 p STARTED     4  64.2kb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
.operations.2020.05.23                                           0 p STARTED 31341  30.9mb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
.searchguard                                                     0 p STARTED     5  84.6kb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
infra-write                                                      1 p STARTED 18268  24.3mb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
infra-write                                                      4 p STARTED 18078    24mb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
infra-write                                                      2 p STARTED 18233  24.7mb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
infra-write                                                      3 p STARTED 18091  24.4mb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
infra-write                                                      0 p STARTED 18252  24.6mb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
.kibana                                                          0 p STARTED     1   3.2kb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
Comment 7 Jeff Cantrill 2020-06-04 11:47:04 UTC 
*** Bug 1843596 has been marked as a duplicate of this bug. ***
Comment 8 Anping Li 2020-06-08 13:10:28 UTC 
@Lukas, Move to verify as the indices replicas rollback after upgrade. and no function error.
Comment 10 errata-xmlrpc 2020-10-27 16:00:21 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196