Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1712955

Summary:	Can't scale up ES nodes from 3 to N (N>3) in clusterlogging CRD instance.
Product:	OpenShift Container Platform	Reporter:	ewolinet
Component:	Logging	Assignee:	ewolinet
Status:	CLOSED ERRATA	QA Contact:	Anping Li <anli>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.1.z	CC:	anli, aos-bugs, ewolinet, jcantril, pweil, qitang, rmeggins, vlaad
Target Milestone:	---
Target Release:	4.1.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:	1712721	Environment:
Last Closed:	2019-08-28 19:54:45 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1712721
Bug Blocks:

Description ewolinet 2019-05-22 15:11:13 UTC

+++ This bug was initially created as a clone of Bug #1712721 +++

Description of problem:
Deploy logging with 3 ES nodes, then wait until all pods running, change es node count to 4 in clusterlogging CRD instance, wait for about 10 minutes, the number of es node count is still 3 in the elasticsearch CRD instance. No logs in cluster-logging-operator pod.

Add ES nodes from 2 to 4 in the clusterlogging CRD instance, the ES node count can be changed to 4 in the elasticsearch CRD instance, and the ES pods could be scaled up. Find log `level=info msg="Elasticsearch node configuration change found, updating elasticsearch"` in cluster-logging-operator pod.

Version-Release number of selected component (if applicable):
image-registry.openshift-image-registry.svc:5000/openshift/ose-cluster-logging-operator:v4.1.0-201905191700


How reproducible:
Always

Steps to Reproduce:
1.Deploy logging via OLM, set es node count to 3 in the clusterlogging CRD instance
2.wait until all logging pods running, change es node count to 4 in clusterlogging CRD instance
3.check pods in `openshift-logging` namespace, and check the es node count in elasticsearch CRD instance and clusterlogging CRD instance

Actual results:


Expected results:


Additional info:

--- Additional comment from Qiaoling Tang on 2019-05-22 08:23:29 UTC ---

Actual results:
the es nodeCount in elasticsearch CRD instance isn't changed after changing es nodeCount from 3 to n (n>3) in the clusterlogging CRD instance

Expected results:
the es node count should be the same as it in the clusterlogging CRD instance.

Additional info:
Scaling up es nodes from 1 or 2 to n(n>=3), no issue.
Scaling up es nodes from 4 or 5 to 6, no issue.

This issue only happens when scaling up from 3 nodes to n(n > 3) nodes

The workaround is: 
1. change es nodeCount in clusterlogging CRD instance, 
2. use `oc delete elasticsearch elasticsearch -n openshift-logging` to delete elasticsearch CRD instance, then the elasticsearch would be recreated, and the nodeCount is what it set in the clusterlogging CRD instance.

--- Additional comment from Ben Parees on 2019-05-22 13:54:33 UTC ---

this should likely be cloned+backported to 4.1.z

Comment 3 Anping Li 2019-08-16 05:53:04 UTC

Test blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1741753

Comment 4 Anping Li 2019-08-19 06:59:36 UTC

Verified in v4.1.12

Comment 8 Anping Li 2019-08-23 05:32:54 UTC

Verified in 4.1.13, the ES can be scaled up from 3-4.
$ oc get pods
NAME                                            READY   STATUS      RESTARTS   AGE
cluster-logging-operator-54894bdc48-xn589       1/1     Running     0          34m
curator-1566538200-f2ddz                        0/1     Completed   0          118s
elasticsearch-cd-9ga2844i-1-7dc7f8d66-s4h5f     2/2     Running     0          2m2s
elasticsearch-cdm-80417wre-1-576558cfbb-wtgkf   2/2     Running     0          33m
elasticsearch-cdm-80417wre-2-78d4647ff5-sw58h   2/2     Running     0          33m
elasticsearch-cdm-80417wre-3-5cff54d779-g45vm   2/2     Running     0          33m

Comment 10 errata-xmlrpc 2019-08-28 19:54:45 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2547