Bug 1545280 - [starter-ca-central-1] infinite loop in logging upgrade playbook
Summary: [starter-ca-central-1] infinite loop in logging upgrade playbook
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 3.9.0
Assignee: ewolinet
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-02-14 14:26 UTC by Justin Pierce
Modified: 2018-03-28 14:28 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Excessive log output in the logging role due to module calls. Consequence: With larger number of nodes we see exponential decrease in performance when using -vvv verbosity. Fix: Change two different modules to be action_plugins instead since they do not need to be run against a host and are just manipulating variables. Result: Decreased output and increased performance from before when using -vvv verbosity.
Clone Of:
Environment:
Last Closed: 2018-03-28 14:28:16 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:0489 0 None None None 2018-03-28 14:28:54 UTC

Description Justin Pierce 2018-02-14 14:26:20 UTC
Description of problem:
During an upgrade of logging components on starter-ca-central-1, the logging playbooks were executed but had to be forcibly terminated after running for over 8 hours and generating ~50G of log output.

Version-Release number of the following components:
Ansible 2.4.3
OCP v3.9.0-0.42.0

A link to a portion of the log output will be attached.

Comment 3 ewolinet 2018-02-15 14:59:13 UTC
If this merges in as well, it would save some more time/output https://github.com/openshift/openshift-ansible/pull/7150

Comment 5 Mike Fiedler 2018-03-02 19:47:24 UTC
Installed logging with -vvv on 85 node cluster with openshift-ansible.noarch        3.9.0-0.53.0.git.0.f8f01ef.el7


INSTALLER STATUS ****************************************************************************************
Initialization             : Complete (0:00:17)     
Logging Install            : Complete (0:06:21)     


real    6m42.417s         
user    2m14.022s         
sys     1m39.366s    

Output was 4.4MB

Comment 6 Mike Fiedler 2018-03-03 18:31:04 UTC
2nd -vvv install in 148 node cluster

INSTALLER STATUS ****************************************************************************************
Initialization             : Complete (0:00:38)     
Logging Install            : Complete (0:09:03)     


real    9m48.398s         
user    3m2.816s          
sys     2m55.410s 

6.3MB log

Comment 7 Anping Li 2018-03-05 02:25:12 UTC
@Mike, Many thanks. Have you try the logging redeploy/upgrade?

Comment 9 Anping Li 2018-03-06 02:43:19 UTC
@Mike Redeploy is ok. Move to verified.

Comment 12 errata-xmlrpc 2018-03-28 14:28:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489


Note You need to log in before you can comment on or make changes to this bug.