Bug 1314564

Summary:	[DOCS] Scale the EFK pods when changes are made
Product:	OpenShift Container Platform	Reporter:	Eric Jones <erjones>
Component:	Documentation	Assignee:	Brandi Munilla <bmcelvee>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Anping Li <anli>
Severity:	low	Docs Contact:	Vikram Goyal <vigoyal>
Priority:	low
Version:	3.1.0	CC:	aos-bugs, bmcelvee, erjones, jokerman, mmccomas
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:	OpenShift Enterprise 3.1.1.6
Last Closed:	2018-03-08 15:04:14 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Eric Jones 2016-03-03 22:58:09 UTC

Document URL: 
https://access.redhat.com/documentation/en/openshift-enterprise/version-3.1/installation-and-configuration/#post-deployment-configuration

Section Number and Name: 
17.4. POST-DEPLOYMENT CONFIGURATION

Describe the issue: 
After working with some of the engineers for the EFK stack, it became apparent that when you make changes to any part of the EFK stack (specifically the FluentD or ElasticSearch) you should scale both of these down to zero, make the changes, and then scale them back up to the appropriate numbers, trying to scale FluentD not too long after scaling ElasticSearch.

Suggestions for improvement: 
Make a note about this so that anytime changes are made, the appropriate actions are taken.

Additional information: 
You can do this by running `oc scale --replicas=0 dc/<ELASTICSEARCH_DC> dc/<FLUENTD_DC> ` and then, after you have made the change (like attach a PVC) you can scale them back up using `oc scale --replicas=# dc/<ELASTICSEARCH_DC>` and `oc scale --replicas=# dc/<FLUENTD_DC>`

Comment 3 Brandi Munilla 2018-01-23 22:24:39 UTC

Hi Eric and Anping,

I opened the following PR with a note instructing readers to scale Elasticsearch and Fluentd as recommended:

https://github.com/openshift/openshift-docs/pull/7281

Comment 4 Anping Li 2018-01-24 08:10:47 UTC

Brandi,
Fluentd is deployed as daemonset. you couldn't scale down fluentd. The correct way is to make the fluentd don't match any nodes.

To scale to zero:
+----
+$ oc scale --replicas=0 dc/<ELASTICSEARCH_DC> 

 $ get the fluentd node selector  
 oc get ds logging-fluentd -o yaml |grep -A 1 Selector
      nodeSelector:
        logging-infra-fluentd: "true"
 $ Change nodeSelector in daemonset configure to match zero node  

+----
+
+To scale to back up from zero:
+----
+$ oc scale --replicas=# dc/<ELASTICSEARCH_DC>
+ Change the nodeSelector back in daemonset configure.

Comment 5 Anping Li 2018-01-24 08:21:48 UTC

You can use oc patch command to modify the daemonset nodeSelector.

oc patch ds logging-fluentd -p '{"spec":{"template":{"spec":{"nodeSelector":{"nonexistlabel":"true"}}}}}'

Comment 7 Anping Li 2018-01-30 07:19:55 UTC

Brandi,
 How about change as the following?

+Change nodeSelector in the daemonset configuration to match zero:
+----
$ get the fluentd node selector
$ oc get ds logging-fluentd -o yaml |grep -A 1 Selector
     nodeSelector:
       logging-infra-fluentd: "true"

Use the `oc patch` command to modify the daemonset nodeSelector:
oc patch ds logging-fluentd -p '{"spec":{"template":{"spec":{"nodeSelector":{"nonexistlabel":"true"}}}}}'

$ get the fluentd node selector
$ oc get ds logging-fluentd -o yaml |grep -A 1 Selector
     nodeSelector:
       "nonexistlabel: "true"

To scale Elastcsearch back up from zero:
----
$ oc scale --replicas=# dc/<ELASTICSEARCH_DC>
----
change nodeSelector in the daemonset configuration to back to  logging-infra-fluentd: "true

Use the `oc patch` command to modify the daemonset nodeSelector:
----
oc patch ds logging-fluentd -p '{"spec":{"template":{"spec":{"nodeSelector":{"logging-infra-fluentd":"true"}}}}}'

Comment 8 Brandi Munilla 2018-02-12 19:48:07 UTC

Thanks, Anping! I updated the PR: 
https://github.com/openshift/openshift-docs/pull/7281

Comment 9 Anping Li 2018-02-13 02:04:34 UTC

Looks good to me now.

Comment 12 openshift-github-bot 2018-03-07 20:49:01 UTC

Commits pushed to master at https://github.com/openshift/openshift-docs

https://github.com/openshift/openshift-docs/commit/ca24ef03bbd5d32000b35fa52537ec2a8e6ead81
Bug 1314564 Add Instructions to Scale EFK Pods When Changes are Made

https://github.com/openshift/openshift-docs/commit/cf6b546a9f96c9bfbce20cc5834eb8e0e2a81314
Merge pull request #7778 from bmcelvee/BZ1314564

Bug 1314564 Add Instructions to Scale EFK Pods When Changes are Made

Comment 13 Brandi Munilla 2018-03-08 15:04:14 UTC

Link to documentation on the Customer Portal: https://access.redhat.com/documentation/en-us/openshift_enterprise/3.1/html-single/installation_and_configuration/#fluentd