Bug 1512028 - operations logs lost if no ops cluster and no @OUTPUT label
Summary: operations logs lost if no ops cluster and no @OUTPUT label
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.6.1
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 3.6.z
Assignee: Rich Megginson
QA Contact: Anping Li
URL:
Whiteboard:
Depends On: 1511719
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-10 17:21 UTC by Rich Megginson
Modified: 2017-12-07 06:49 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: When upgrading to a new version of logging, if the deployment does not use a separate OPS logging cluster, and there is no @OUTPUT label in the fluent.conf, fluentd does not know how to route the operations logs. Consequence: The operations logs are not logged. User will see no new operations (.operations.*) logs after the upgrade. Fix: Change the fluentd configuration to correctly handle the case both with and without the @OUTPUT label, and for the OPS and non-OPS cases. Result: Operations logs flow uninterrupted after upgrade.
Clone Of: 1511719
Environment:
Last Closed: 2017-12-07 06:49:06 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin-aggregated-logging pull 777 0 None None None 2017-11-10 17:21:52 UTC
Red Hat Product Errata RHBA-2017:3390 0 normal SHIPPED_LIVE OpenShift Container Platform 3.6, 3.5, and 3.4 images update 2017-12-07 11:46:04 UTC

Description Rich Megginson 2017-11-10 17:21:52 UTC
+++ This bug was initially created as a clone of Bug #1511719 +++

Description of problem:
Deploy logging with no ops cluster and based on 3.6 or an earlier 3.7 pre-release which does not have the @OUTPUT label in fluent.conf

update to a recent fluentd image

no operations logs are stored

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

--- Additional comment from Rich Megginson on 2017-11-10 12:20:45 EST ---

https://github.com/openshift/origin-aggregated-logging/commit/61d1196eeab4ac02b360b20f9338d3ab2b5d5eef

Comment 1 Rich Megginson 2017-11-11 00:34:28 UTC
koji_builds:
  https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=623860
repositories:
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:rhaos-3.6-rhel-7-docker-candidate-56826-20171111002326
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:latest
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6.173.0.73
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6.173.0.73-5
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6

Comment 3 Anping Li 2017-11-14 07:56:28 UTC
I can reproduced by deploy logging:3.6.173.0.49 using openshift-ansible-                       3.6.173.0.5.

Once we update to 3.6.173.0.63, no such issue.

logging-kibana/images/3.6.173.0.63-10
logging-elasticsearch/images/3.6.173.0.63-10
logging-auth-proxy/images/3.6.173.0.63-10
logging-fluentd/images/3.6.173.0.63-10
logging-curator/images/3.6.173.0.63-10

oc exec -c elasticsearch logging-es-data-master-foodc7mt-2-3bw56 -- curl -s -XGET --cacert /etc/elasticsearch/secret/admin-ca --cert /etc/elasticsearch/secret/admin-cert --key /etc/elasticsearch/secret/admin-key https://localhost:9200/.operations*/_count
{"count":421,"_shards":{"total":1,"successful":1,"failed":0}}

Comment 6 errata-xmlrpc 2017-12-07 06:49:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3390


Note You need to log in before you can comment on or make changes to this bug.