Bug 1545280 – [starter-ca-central-1] infinite loop in logging upgrade playbook

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1545280 - [starter-ca-central-1] infinite loop in logging upgrade playbook

Summary:	[starter-ca-central-1] infinite loop in logging upgrade playbook

Status:	CLOSED ERRATA

Aliases:	None

Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Installer (Show other bugs)
Sub Component:
Version:	3.9.0
Hardware:	Unspecified Unspecified

Priority	unspecified Severity high
Target Milestone:	---
Target Release:	3.9.0
Assigned To:	ewolinet
QA Contact:	Anping Li
Docs Contact:

URL:
Whiteboard:
Keywords:

Depends On:
Blocks:
	Show dependency tree / graph

Reported:	2018-02-14 09:26 EST by Justin Pierce
Modified:	2018-03-28 10:28 EDT (History)
CC List:	5 users (show)

See Also:
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:	Cause: Excessive log output in the logging role due to module calls. Consequence: With larger number of nodes we see exponential decrease in performance when using -vvv verbosity. Fix: Change two different modules to be action_plugins instead since they do not need to be run against a host and are just manipulating variables. Result: Decreased output and increased performance from before when using -vvv verbosity.
Story Points:	---
Clone Of:
Environment:
Last Closed:	2018-03-28 10:28:16 EDT
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2018:0489	None	None	None	2018-03-28 10:28 EDT

Groups: None (edit)

Description Justin Pierce 2018-02-14 09:26:20 EST

Description of problem:
During an upgrade of logging components on starter-ca-central-1, the logging playbooks were executed but had to be forcibly terminated after running for over 8 hours and generating ~50G of log output.

Version-Release number of the following components:
Ansible 2.4.3
OCP v3.9.0-0.42.0

A link to a portion of the log output will be attached.

Comment 3 ewolinet 2018-02-15 09:59:13 EST

If this merges in as well, it would save some more time/output https://github.com/openshift/openshift-ansible/pull/7150

Comment 5 Mike Fiedler 2018-03-02 14:47:24 EST

Installed logging with -vvv on 85 node cluster with openshift-ansible.noarch        3.9.0-0.53.0.git.0.f8f01ef.el7


INSTALLER STATUS ****************************************************************************************
Initialization             : Complete (0:00:17)     
Logging Install            : Complete (0:06:21)     


real    6m42.417s         
user    2m14.022s         
sys     1m39.366s    

Output was 4.4MB

Comment 6 Mike Fiedler 2018-03-03 13:31:04 EST

2nd -vvv install in 148 node cluster

INSTALLER STATUS ****************************************************************************************
Initialization             : Complete (0:00:38)     
Logging Install            : Complete (0:09:03)     


real    9m48.398s         
user    3m2.816s          
sys     2m55.410s 

6.3MB log

Comment 7 Anping Li 2018-03-04 21:25:12 EST

@Mike, Many thanks. Have you try the logging redeploy/upgrade?

Comment 9 Anping Li 2018-03-05 21:43:19 EST

@Mike Redeploy is ok. Move to verified.

Comment 12 errata-xmlrpc 2018-03-28 10:28:16 EDT

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489

Note You need to log in before you can comment on or make changes to this bug.