Bug 1838153 - The ZeroRedundancy is not honored during upgrade clo-first
Summary: The ZeroRedundancy is not honored during upgrade clo-first
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 4.5
Hardware: Unspecified
OS: Unspecified
low
medium
Target Milestone: ---
: 4.6.0
Assignee: Jeff Cantrill
QA Contact: Anping Li
URL:
Whiteboard:
: 1843596 (view as bug list)
Depends On:
Blocks: 1843738
TreeView+ depends on / blocked
 
Reported: 2020-05-20 14:33 UTC by Anping Li
Modified: 2020-10-27 16:00 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-27 16:00:21 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin-aggregated-logging pull 1916 0 None closed Bug 1838153: Fix index patterns in index templates 2021-01-12 13:12:38 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:00:50 UTC

Description Anping Li 2020-05-20 14:33:17 UTC
Description of problem:
The ZeroRedundancy is not honored during the upgrade. Result in unassigned Replicas shards. If the nodeCount=1, the ES status will always be Yellow.

oc exec -c elasticsearch elasticsearch-cdm-6hyfcqb1-1-576c48f45-lm9dm -- es_util --query=_cat/shards
.kibana.647a750f1787408bf50088234ec0edd5a6a9b2ac                 0 p STARTED         3  37.5kb 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
.kibana                                                          0 p STARTED         1   3.2kb 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
project.logjsonx.f1301ba5-3a77-431e-b383-117619c14be9.2020.05.20 0 p STARTED      1658   1.9mb 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
project.logflatx.17244a69-6d18-4594-a99b-8af3efc55908.2020.05.20 0 p STARTED      1654   2.1mb 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
.searchguard                                                     0 p STARTED         5 145.7kb 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
app-write                                                        1 p STARTED         0    162b 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
app-write                                                        1 r UNASSIGNED                            
app-write                                                        2 p STARTED         0    162b 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
app-write                                                        2 r UNASSIGNED                            
app-write                                                        4 p STARTED         0    162b 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
app-write                                                        4 r UNASSIGNED                            
app-write                                                        3 p STARTED         0    162b 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
app-write                                                        3 r UNASSIGNED                            
app-write                                                        0 p STARTED         0    162b 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
app-write                                                        0 r UNASSIGNED                            
.operations.2020.05.20                                           0 p STARTED    562893 574.2mb 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
infra-write                                                      1 p STARTED         0    162b 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
infra-write                                                      1 r UNASSIGNED                            
infra-write                                                      2 p STARTED         0    162b 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
infra-write                                                      2 r UNASSIGNED                            
infra-write                                                      4 p STARTED         0    162b 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
infra-write                                                      4 r UNASSIGNED                            
infra-write                                                      3 p STARTED         0    162b 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
infra-write                                                      3 r UNASSIGNED                            
infra-write                                                      0 p STARTED         0    162b 10.128.2.53 elasticsearch-cdm-6hyfcqb1-1
infra-write                                                      0 r UNASSIGNED             

Version-Release number of selected component (if applicable):
4.4->4.5

How reproducible:
Always

Steps to Reproduce:
1. Install 4.4 using nodeCount=1, ZeroRedundancy
2. Upgrade CLO to 4.5
3. Waiting for a while, check the indices in Elaticsearch.
4. Upgrade EO to 4.5

Actual results:
UNASSIGNED Replica shards.

Expected results:

In clusterlogging-4.4.z, Add infra-write and infra-app into the index template.


Additional info:

Comment 1 Anping Li 2020-05-20 15:37:02 UTC
I can reproduce it.  Tt is strange, when nodeCount=2, redundancyPolicy: "ZeroRedundancy",  No Replicas shards.

Comment 2 Anping Li 2020-05-23 12:26:33 UTC
Interestingly, After upgrade,  the ES doesn't allocate replicas to the indices infra-write and app-write any more.

+ oc exec -c elasticsearch elasticsearch-cdm-xv9zo8gz-1-6fcc5d4998-lpb9g -- es_util --query=_cat/shards
project.logflatx.b1c056f0-4405-45bd-8cea-76338862d9ed.2020.05.23 0 p STARTED   746   543kb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
.security                                                        0 p STARTED     5  30.8kb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
project.logjsonx.9b993d19-0818-4232-8940-cf06a750e965.2020.05.23 0 p STARTED   746 485.3kb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
app-write                                                        1 p STARTED   182   414kb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
app-write                                                        4 p STARTED   166 269.1kb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
app-write                                                        2 p STARTED   171 317.8kb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
app-write                                                        3 p STARTED   165 331.2kb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
app-write                                                        0 p STARTED   176 259.9kb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
.kibana.647a750f1787408bf50088234ec0edd5a6a9b2ac                 0 p STARTED     4  64.2kb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
.operations.2020.05.23                                           0 p STARTED 31341  30.9mb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
.searchguard                                                     0 p STARTED     5  84.6kb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
infra-write                                                      1 p STARTED 18268  24.3mb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
infra-write                                                      4 p STARTED 18078    24mb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
infra-write                                                      2 p STARTED 18233  24.7mb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
infra-write                                                      3 p STARTED 18091  24.4mb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
infra-write                                                      0 p STARTED 18252  24.6mb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1
.kibana                                                          0 p STARTED     1   3.2kb 10.129.2.63 elasticsearch-cdm-xv9zo8gz-1

Comment 7 Jeff Cantrill 2020-06-04 11:47:04 UTC
*** Bug 1843596 has been marked as a duplicate of this bug. ***

Comment 8 Anping Li 2020-06-08 13:10:28 UTC
@Lukas, Move to verify as the indices replicas rollback after upgrade. and no function error.

Comment 10 errata-xmlrpc 2020-10-27 16:00:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.