Description of problem:
The link in the OpenShift web UI to display the logs of a specific container in Kibana is broken
Version-Release number of selected component (if applicable): tested with logging deployer v3.4 and 3.3.0 on OpenShift 3.4
Steps to Reproduce:
1. install the logging components
2. klick on the archive link on the top right corner of the container log page
No log entries are displayed in Kibana
To see the container logs
It seems that dots "." are populated when underscores "_" are expected.
this matches the search parameters:
kubernetes.pod_name:"jdg-app-7-bid8f" AND kubernetes.namespace_name: "jdg-datagrid"
changing the query to kubernetes_pod_name:"jdg-app-7-bid8f" AND kubernetes_namespace_name: "jdg-datagrid" displayed the expected log entries
Can you verify on what version of OCP you are finding this issue? Did you:
1. Install 3.3
2. Install 3.3 logging
3. Upgrade to 3.4
4. Upgrade to 3.4 logging
And then determined the link is broken?
it was OCP 3.4, a new installation (no upgrade) done on Monday. Logging installation done today. I can provide the exact version tomorrow.
Exact version of OCP is 3.4.40.
We used logging-deployer: v3.4 with image digest 9fc6bb66, which matched v126.96.36.199-2
I am unable to replicate the issue using:
Image ID: sha256:752eaf28738244e3b81d5afb9b166e2dca30433f7ccc45446e7aa9cc1d896f27
Fluentd is where the underscore/dot construct is introduced. At one point in the 3.4 development lifecycle we were required to flatten dot delimited fields by replacing them underscores. This was temporary as we moved to a different version of ES that did not restrict us to this format.
I am wondering if you are working from a "fresh" deployment that was attached to existing data.
Xia is it possible for you to verify
Are you able to confirm that when you visit Kibana using the archive link that it correctly displays data?
I checked the fluentd version that has been deployed:
I see in Xia's link that he also has the dots. Are the dots or the underscores expected?
For what it is worth, we had first used the deployer 3.3.0 as it was the example in the 3.4 documentation. We then undeployed everything using the command available for this purpose in the documentation and started the process with v3.4 from start again.
Should be something like kubernetes.namespace_name: https://github.com/openshift/origin-aggregated-logging/blob/release-1.4/fluentd/configs.d/openshift/output-es-config.conf#L6
(In reply to Jeff Cantrill from comment #7)
> Are you able to confirm that when you visit Kibana using the archive link
> that it correctly displays data?
Yes, I confirm. -- Also switched between different indices there, log entries are displayed well.
Please provide additional information on how to reproduce or close accordingly based on the previous comments. I suspect you may have been working with older images that were not officially released.
I am not at the customer any more. We did use officially released images. The installation was done with the deployer pointing to v3.4. It is possible that the fluentd pods from 3.3.0 did not get properly deleted without us noticing it. As I am not able to follow up on the case you can close it
Unable to reproduce.