Bug 1520629 - [3.5] after upgrade to 3.6 from 3.5 log aggregation does not show recent logs
Summary: [3.5] after upgrade to 3.6 from 3.5 log aggregation does not show recent logs
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.4.0
Hardware: All
OS: Linux
high
urgent
Target Milestone: ---
: 3.5.z
Assignee: Jeff Cantrill
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On: 1494612
Blocks: 1525687
TreeView+ depends on / blocked
 
Reported: 2017-12-04 20:55 UTC by Ruchika K
Modified: 2018-04-30 05:01 UTC (History)
19 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Fluentd fails to properly process messages when it is unable to determine the namespace and pod uuids Consequence: The logging pipeline spews lots message and sometimes blocks log flow to elasticsearch Fix: Check for the missing fields and set orphan the record if needed. Result: Logs continue to flow and orphaned records end up in an orphaned namespace.
Clone Of: 1494612
: 1525687 (view as bug list)
Environment:
Last Closed: 2018-04-30 05:00:57 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:1235 None None None 2018-04-30 05:01:54 UTC
Github openshift origin-aggregated-logging pull 825 None None None 2017-12-07 20:29:18 UTC
Github openshift origin-aggregated-logging pull 851 None None None 2017-12-13 20:28:24 UTC
Red Hat Knowledge Base (Solution) 3211281 None None None 2017-12-04 20:55:42 UTC

Comment 2 Jeff Cantrill 2017-12-13 19:58:44 UTC
Moving this backed to assigned until we merge 3.5

Comment 7 Christian Stark 2018-01-05 14:40:41 UTC
please let us know if you have some update and/or eta we could share with the customer
Thank you
Christian

Comment 9 Christian Stark 2018-01-10 16:15:58 UTC
Hello Jeff,


Can you give us info which exact 3.5 image version will contain the fix?
We would recommend the customer the workaround mentioned in: https://access.redhat.com/solutions/3211281
but we need to be confident that it will work with 3.5 and with which exact Image version.

Thank you
Christian

Comment 16 Jeff Cantrill 2018-02-12 16:30:14 UTC
Please open an issue that is specific to what you are seeing as this fix is unrelated to what is reported in https://bugzilla.redhat.com/show_bug.cgi?id=1520629#c15

Comment 24 Junqi Zhao 2018-04-18 02:18:22 UTC
@Noriko
From
https://bugzilla.redhat.com/show_bug.cgi?id=1494612
I see Orphan index is added for logging 3.6

But I did not see Orphan index created for logging 3.5

Are my verification steps right to verify this defect?
1. Deploy logging 3.4 with pv and note down some project logs.
2. Upgrade logging 3.4 to 3.5, and check if the previous logs could be found.
3. Create new project to see if logs could be collected

Comment 25 Noriko Hosoi 2018-04-18 18:46:11 UTC
(In reply to Junqi Zhao from comment #24)
> @Noriko
> From
> https://bugzilla.redhat.com/show_bug.cgi?id=1494612
> I see Orphan index is added for logging 3.6
> 
> But I did not see Orphan index created for logging 3.5
> 
> Are my verification steps right to verify this defect?
> 1. Deploy logging 3.4 with pv and note down some project logs.
> 2. Upgrade logging 3.4 to 3.5, and check if the previous logs could be found.
> 3. Create new project to see if logs could be collected

Hi @Junqi,

The implementation for the orphan index for 3.5 is different from 3.6+.  The timing for the index creation might be also different from each other.

When you say "I see Orphan index is added for logging 3.6", the index is empty or it has some data in it?
  oc exec $EPOD -- curl -s -k --cert /etc/elasticsearch/secret/admin-cert \
       --key /etc/elasticsearch/secret/admin-key \
       "https://localhost:9200/_cat/indices?v"

When a log is missing kubernetes or kubernetes.namespace_id, it's indexed in the .orphaned index.  Could it be possible to mimic such a situation and check if the orphaned logs are indexed in the .orphaned index or not?  Thanks!

Comment 26 Junqi Zhao 2018-04-19 01:56:04 UTC
(In reply to Noriko Hosoi from comment #25)

> When you say "I see Orphan index is added for logging 3.6", the index is
> empty or it has some data in it?
>   oc exec $EPOD -- curl -s -k --cert /etc/elasticsearch/secret/admin-cert \
>        --key /etc/elasticsearch/secret/admin-key \
>        "https://localhost:9200/_cat/indices?v"

Orphan index is only applicable for logging 3.6 and above, 3.5 does not have Orphan index

Comment 27 Junqi Zhao 2018-04-19 02:51:48 UTC
After upgrade from logging 3.4 to 3.5, recent logs and new project logs could be shown.

# openshift version
openshift v3.5.5.31.67
kubernetes v1.5.2+43a9be4
etcd 3.1.0

logging-elasticsearch/images/v3.5.5.31.67-2
logging-curator/images/v3.5.5.31.67-4
logging-fluentd/images/v3.5.5.31.67-2
logging-kibana/images/v3.5.5.31.67-2
logging-auth-proxy/images/v3.5.5.31.67-2

Comment 31 Junqi Zhao 2018-04-23 10:07:48 UTC
After upgrade from logging 3.4 to 3.5, recent logs and new project logs could be shown.

# openshift version
openshift v3.5.5.31.67
kubernetes v1.5.2+43a9be4
etcd 3.1.0

logging-auth-proxy/images/v3.5.5.31.67-1
logging-curator/images/v3.5.5.31.67-3
logging-elasticsearch/images/v3.5.5.31.67-1
logging-kibana/images/v3.5.5.31.67-1
logging-fluentd/images/v3.5.5.31.67-1

Comment 34 errata-xmlrpc 2018-04-30 05:00:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:1235


Note You need to log in before you can comment on or make changes to this bug.