Bug 1418407 - OpenShift - Log archive link broken
Summary: OpenShift - Log archive link broken
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.4.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Jeff Cantrill
QA Contact: Xia Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-02-01 18:45 UTC by Frederic Giloux
Modified: 2017-02-08 14:26 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-02-08 14:26:05 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Frederic Giloux 2017-02-01 18:45:02 UTC
Description of problem:
The link in the OpenShift web UI to display the logs of a specific container in Kibana is broken 

Version-Release number of selected component (if applicable): tested with logging deployer v3.4 and 3.3.0 on OpenShift 3.4

How reproducible:
Always

Steps to Reproduce:
1. install the logging components
2. klick on the archive link on the top right corner of the container log page 

Actual results:
No log entries are displayed in Kibana

Expected results:
To see the container logs

Additional info:
It seems that dots "." are populated when underscores "_" are expected.
Link example:
ttps://kibana.apps.ocp.com/#/discover?_g=(time:(from:now-1w,mode:relative,to:now))&_a=(columns:!(kubernetes.container_name,message),index:.all,interval:auto,query:(query_string:(analyze_wildcard:!t,query:%27kubernetes.pod_name:%22jdg-app-7-bid8f%22%20AND%20kubernetes.namespace_name:%22jdg-datagrid%22%27)),sort:!(time,desc))

this matches the search parameters:
kubernetes.pod_name:"jdg-app-7-bid8f" AND kubernetes.namespace_name: "jdg-datagrid"
changing the query to kubernetes_pod_name:"jdg-app-7-bid8f" AND kubernetes_namespace_name: "jdg-datagrid" displayed the expected log entries

Comment 1 Jeff Cantrill 2017-02-01 20:40:49 UTC
@Frederic,

Can you verify on what version of OCP you are finding this issue? Did you:

1. Install 3.3
2. Install 3.3 logging
3. Upgrade to 3.4
4. Upgrade to 3.4 logging

And then determined the link is broken?

Comment 2 Frederic Giloux 2017-02-01 22:00:23 UTC
@Jeff,

it was OCP 3.4, a new installation (no upgrade) done on Monday. Logging installation done today. I can provide the exact version tomorrow.

Comment 3 Frederic Giloux 2017-02-02 12:56:55 UTC
Exact version of OCP is 3.4.40. 
We used logging-deployer: v3.4 with image digest 9fc6bb66, which matched v3.4.1.2-2

Comment 4 Jeff Cantrill 2017-02-02 20:14:41 UTC
I am unable to replicate the issue using:

ose: 3.4.1.2
fluentd:
  Image: registry.ops.openshift.com/openshift3/logging-fluentd:3.4.1
 Image ID: sha256:752eaf28738244e3b81d5afb9b166e2dca30433f7ccc45446e7aa9cc1d896f27

Fluentd is where the underscore/dot construct is introduced. At one point in the 3.4 development lifecycle we were required to flatten dot delimited fields by replacing them underscores.  This was temporary as we moved to a different version of ES that did not restrict us to this format.  

I am wondering if you are working from a "fresh" deployment that was attached to existing data.

Comment 5 Jeff Cantrill 2017-02-02 20:15:09 UTC
Xia is it possible for you to verify

Comment 7 Jeff Cantrill 2017-02-03 13:41:45 UTC
Xia,

Are you able to confirm that when you visit Kibana using the archive link that it correctly displays data?

Comment 8 Frederic Giloux 2017-02-03 18:01:17 UTC
I checked the fluentd version that has been deployed:
v3.4
registry.access.redhat.com/openshift3/logging-fluentd@sha256:9c5b0e43f9035b9f84ec1dd27d026ebc95b7a0d5de82d8d1c477d1268f692aa8

I see in Xia's link that he also has the dots. Are the dots or the underscores expected?

For what it is worth, we had first used the deployer 3.3.0 as it was the example in the 3.4 documentation. We then undeployed everything using the command available for this purpose in the documentation and started the process with v3.4 from start again.

Comment 9 Jeff Cantrill 2017-02-03 19:27:58 UTC
Should be something like kubernetes.namespace_name: https://github.com/openshift/origin-aggregated-logging/blob/release-1.4/fluentd/configs.d/openshift/output-es-config.conf#L6

Comment 10 Xia Zhao 2017-02-04 02:30:38 UTC
(In reply to Jeff Cantrill from comment #7)
> Xia,
> 
> Are you able to confirm that when you visit Kibana using the archive link
> that it correctly displays data?

Yes, I confirm. -- Also switched between different indices there, log entries are displayed well.

Comment 11 Jeff Cantrill 2017-02-08 01:04:09 UTC
@Frederic,

Please provide additional information on how to reproduce or close accordingly based on the previous comments.  I suspect you may have been working with older images that were not officially released.

Comment 12 Frederic Giloux 2017-02-08 04:56:11 UTC
@Jeff,

I am not at the customer any more. We did use officially released images. The installation was done with the deployer pointing to v3.4. It is possible that the fluentd pods from 3.3.0 did not get properly deleted without us noticing it. As I am not able to follow up on the case you can close it

Comment 13 Jeff Cantrill 2017-02-08 14:26:05 UTC
Unable to reproduce.


Note You need to log in before you can comment on or make changes to this bug.