Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1545280

Summary:	[starter-ca-central-1] infinite loop in logging upgrade playbook
Product:	OpenShift Container Platform	Reporter:	Justin Pierce <jupierce>
Component:	Installer	Assignee:	ewolinet
Status:	CLOSED ERRATA	QA Contact:	Anping Li <anli>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	3.9.0	CC:	aos-bugs, jcantril, jokerman, mifiedle, mmccomas
Target Milestone:	---
Target Release:	3.9.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: Excessive log output in the logging role due to module calls. Consequence: With larger number of nodes we see exponential decrease in performance when using -vvv verbosity. Fix: Change two different modules to be action_plugins instead since they do not need to be run against a host and are just manipulating variables. Result: Decreased output and increased performance from before when using -vvv verbosity.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-03-28 14:28:16 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Justin Pierce 2018-02-14 14:26:20 UTC

Description of problem:
During an upgrade of logging components on starter-ca-central-1, the logging playbooks were executed but had to be forcibly terminated after running for over 8 hours and generating ~50G of log output.

Version-Release number of the following components:
Ansible 2.4.3
OCP v3.9.0-0.42.0

A link to a portion of the log output will be attached.

Comment 3 ewolinet 2018-02-15 14:59:13 UTC

If this merges in as well, it would save some more time/output https://github.com/openshift/openshift-ansible/pull/7150

Comment 5 Mike Fiedler 2018-03-02 19:47:24 UTC

Installed logging with -vvv on 85 node cluster with openshift-ansible.noarch        3.9.0-0.53.0.git.0.f8f01ef.el7


INSTALLER STATUS ****************************************************************************************
Initialization             : Complete (0:00:17)     
Logging Install            : Complete (0:06:21)     


real    6m42.417s         
user    2m14.022s         
sys     1m39.366s    

Output was 4.4MB

Comment 6 Mike Fiedler 2018-03-03 18:31:04 UTC

2nd -vvv install in 148 node cluster

INSTALLER STATUS ****************************************************************************************
Initialization             : Complete (0:00:38)     
Logging Install            : Complete (0:09:03)     


real    9m48.398s         
user    3m2.816s          
sys     2m55.410s 

6.3MB log

Comment 7 Anping Li 2018-03-05 02:25:12 UTC

@Mike, Many thanks. Have you try the logging redeploy/upgrade?

Comment 9 Anping Li 2018-03-06 02:43:19 UTC

@Mike Redeploy is ok. Move to verified.

Comment 12 errata-xmlrpc 2018-03-28 14:28:16 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489