Bug 1467963 - There are no Kibana dashboards for container admins
There are no Kibana dashboards for container admins
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging (Show other bugs)
3.6.0
Unspecified Unspecified
urgent Severity urgent
: ---
: 3.7.0
Assigned To: Rich Megginson
Xia Zhao
:
Depends On:
Blocks: 1502992
  Show dependency treegraph
 
Reported: 2017-07-05 12:03 EDT by Rich Megginson
Modified: 2017-11-28 17:00 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Feature: Kibana visualizations and dashboard for monitoring container/pod logs. These visualizations allow administrator users (cluster-admin or cluster-reader) to view logs by deployment, namespace, pod, and container. The script es_load_kibana_ui_objects is used to load dashboards and other Kibana UI objects for the given user. Usage: oc exec $espod -- es_load_kibana_ui_objects user-name That is, it exists inside the Elasticsearch and ES-OPS pod, and must be run inside those pods. Additionally, it requires some indices and other objects set up by the OpenShift Elasticsearch plugin, so the user must login to Kibana (or Elasticsearch) before using this script. This will also add an index pattern for `project.*` and load the necessary index pattern file. Reason: Give administrators an easier way to view Kubernetes/OpenShift related logs in the cluster. Result: Admin users have some graphs and a dashboard to use to view logs from Kubernetes/OpenShift pods and containers.
Story Points: ---
Clone Of:
: 1502992 (view as bug list)
Environment:
Last Closed: 2017-11-28 17:00:15 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Github openshift/origin-aggregated-logging/pull/518 None None None 2017-07-05 22:06 EDT

  None (edit)
Description Rich Megginson 2017-07-05 12:03:10 EDT
Description of problem:
There are no dashboards for container admins.  We must have some sort of Kibana visualizations for 3.6.  This is a release blocker.  There were some dashboards developed during the logging hackfest in June 2017 that must go into 3.6.

Version-Release number of selected component (if applicable):
3.6

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 openshift-github-bot 2017-07-06 23:12:44 EDT
Commit pushed to master at https://github.com/openshift/origin-aggregated-logging

https://github.com/openshift/origin-aggregated-logging/commit/b0044ec9023bc6472d125df0a1dee6746ad27b18
Bug 1467963 - There are no Kibana dashboards for container admins

https://bugzilla.redhat.com/show_bug.cgi?id=1467963

The script es_load_kibana_ui_objects is used to load dashboards and
other Kibana UI objects for the given user.  Usage:

oc exec $espod -- es_load_kibana_ui_objects user-name

That is, it exists inside the Elasticsearch and ES-OPS pod, and must
be run inside those pods.  Additionally, it requires some indices and
other objects set up by the OpenShift Elasticsearch plugin, so the user
must login to Kibana (or Elasticsearch) before using this script.

This will also add an index pattern for `project.*` and load the
necessary index pattern file.

This relies on some additions to the viaq schema which is provided
by https://github.com/ViaQ/elasticsearch-templates/pull/50
Comment 2 Rich Megginson 2017-07-11 13:39:56 EDT
new build - openshift3/logging-elasticsearch:v3.6.140-2
Comment 3 Xia Zhao 2017-07-12 05:16:31 EDT
Blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1469918
Comment 4 Xia Zhao 2017-07-13 05:39:21 EDT
Hi Rich,

* During testing, when I tried to load dashboards and other Kibana UI objects for cluster-admin user xiazhao (I followed the guide to log in to kibana UI with her before trying this), meet this error:

$ oc get po
NAME                                      READY     STATUS    RESTARTS   AGE
logging-curator-1-6gj7l                   1/1       Running   0          1h
logging-es-data-master-5ywqx64o-1-p2rzb   1/1       Running   0          1h
logging-fluentd-gvjd7                     1/1       Running   0          1h
logging-fluentd-pczkz                     1/1       Running   0          1h
logging-kibana-1-f95tp                    2/2       Running   0          1h

$ oc exec logging-es-data-master-5ywqx64o-1-p2rzb -- es_load_kibana_ui_objects xiazhao
rpc error: code = 13 desc = invalid header field value "oci runtime error: exec failed: container_linux.go:247: starting container process caused \"exec: \\\"es_load_kibana_ui_objects\\\": executable file not found in $PATH\"\n"

* After oc rsh into es pod, I can't find the script " es_load_kibana_ui_objects":
# oc rsh logging-es-data-master-5ywqx64o-1-p2rzb
sh-4.2$ find . -name es_load_kibana_ui_objects
sh-4.2$


* From your PR I found the script here, but don't know how should it run:
$ curl -O https://raw.githubusercontent.com/openshift/origin-aggregated-logging/master/elasticsearch/utils/es_load_kibana_ui_objects

$  oc exec logging-es-data-master-5ywqx64o-1-p2rzb -- es_load_kibana_ui_objects xiazhao
rpc error: code = 13 desc = invalid header field value "oci runtime error: exec failed: container_linux.go:247: starting container process caused \"exec: \\\"es_load_kibana_ui_objects\\\": executable file not found in $PATH\"\n"
command terminated with exit code 126

Could you please help point out what else steps should I do? 

Thanks,
Xia
Comment 5 Xia Zhao 2017-07-13 21:51:52 EDT
Confirmed that dev is rebuilding the elasticsearch image, set back to assigned, thanks!
Comment 6 Rich Megginson 2017-07-17 14:34:17 EDT
koji_builds:
  https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=574393
repositories:
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:rhaos-3.6-rhel-7-docker-candidate-85872-20170717180328
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:latest
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6.152.0
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6.152.0-2
Comment 7 Xia Zhao 2017-07-18 05:47:41 EDT
Tested with the latest v3.6 es image which is 12 hours newer than v3.6.152.0-2, issue is reproduced:
# docker images | grep elasticsearch
${brew-registry}/openshift3/logging-elasticsearch   v3.6                9b0f4b948e79        3 hours ago         404.2 MB
${brew-registry}/openshift3/logging-elasticsearch   v3.6.152.0-2        1e210744feb3        15 hours ago        404.7 MB

Test steps in detail:
1. Login kibana with user "xiazhao" who is cluster-admin

2. When executing the adding dashboard script, get this error:
# oc exec logging-es-data-master-x3pw9820-1-v7k4p -- es_load_kibana_ui_objects xiazhao
[2017-07-18 09:15:32,266][INFO ][container.run            ] Adding Kibana dashboards and other UI objects for user xiazhao index .kibana.f7724d98466ed7391e970202dc54a6460046aadb
[2017-07-18 09:15:32,267][INFO ][container.run            ] Adding the index pattern for project.* . . .
cat: /usr/share/java/elasticsearch/index_patterns/com.redhat.viaq-openshift.index-pattern.json: No such file or directory
command terminated with exit code 1

3. Relogin kibana with user "xiazhao", went into the "dashboard" tab, and searched for the kibana visualizations with pattern "*", get 0 foundings

@Rich, please help review and point out if any of the above test steps were wrong, thanks!
Comment 8 Jeff Cantrill 2017-07-18 08:07:03 EDT
(In reply to Xia Zhao from comment #7)
> Tested with the latest v3.6 es image which is 12 hours newer than
> v3.6.152.0-2, 

@Xia can you explain this comment?  If you tested an image is 12 hours newer then it is not the same image as:

issue is reproduced:
> # docker images | grep elasticsearch
> ${brew-registry}/openshift3/logging-elasticsearch   v3.6               
> 9b0f4b948e79        3 hours ago         404.2 MB
> ${brew-registry}/openshift3/logging-elasticsearch   v3.6.152.0-2       
> 1e210744feb3        15 hours ago        404.7 MB
>

Or are you stating you tested 2 sets of images:

1. 12 hours newer
2.  v3.6.152.0-2
Comment 9 Rich Megginson 2017-07-18 11:30:41 EDT
koji_builds:
  https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=574625
repositories:
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:rhaos-3.6-rhel-7-docker-candidate-57374-20170718145154
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:latest
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6.153
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6.153-2
Comment 10 Xia Zhao 2017-07-18 21:58:37 EDT
(In reply to Jeff Cantrill from comment #8)
> (In reply to Xia Zhao from comment #7)
> > Tested with the latest v3.6 es image which is 12 hours newer than
> > v3.6.152.0-2, 
> 
> @Xia can you explain this comment?  If you tested an image is 12 hours newer
> then it is not the same image as:
> 
> issue is reproduced:
> > # docker images | grep elasticsearch
> > ${brew-registry}/openshift3/logging-elasticsearch   v3.6               
> > 9b0f4b948e79        3 hours ago         404.2 MB
> > ${brew-registry}/openshift3/logging-elasticsearch   v3.6.152.0-2       
> > 1e210744feb3        15 hours ago        404.7 MB
> >
> 
> Or are you stating you tested 2 sets of images:
> 
> 1. 12 hours newer
> 2.  v3.6.152.0-2

Hi Jeff,

By specifying openshift_logging_image_version=v3.6 in the inventory file when deploy the logging tsacks, I tested with this image:
> > ${brew-registry}/openshift3/logging-elasticsearch   v3.6               
> > 9b0f4b948e79        3 hours ago         404.2 MB

Thanks,
Xia
Comment 11 Xia Zhao 2017-07-19 02:42:37 EDT
Verified with elasticsearch image v3.6.153-2, get this error message while loading the kibana dashboards, and checked on kibana UI, 0 visualizations was found under the dashboard tab for cluster-admin user:

# oc exec logging-es-data-master-odlf264i-1-bc62n -- es_load_kibana_ui_objects xiazhao
[2017-07-19 06:35:54,131][INFO ][container.run            ] Adding Kibana dashboards and other UI objects for user xiazhao index .kibana.f7724d98466ed7391e970202dc54a6460046aadb
[2017-07-19 06:35:54,132][INFO ][container.run            ] Adding the index pattern for project.* . . .
[2017-07-19 06:35:54,306][INFO ][container.run            ] Adding the Kibana UI objects . . .
cat: /usr/share/java/elasticsearch/kibana_ui_objects/*.json: No such file or directory
Traceback (most recent call last):
  File "<string>", line 4, in <module>
  File "/usr/lib64/python2.7/json/__init__.py", line 290, in load
    **kw)
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
command terminated with exit code 1
Comment 12 openshift-github-bot 2017-07-19 15:00:01 EDT
Commit pushed to master at https://github.com/openshift/origin-aggregated-logging

https://github.com/openshift/origin-aggregated-logging/commit/a4dd522cd1b55a3303343cdf6ab7694ab5954dae
Bug 1467963 There are no Kibana dashboards for container admins

https://bugzilla.redhat.com/show_bug.cgi?id=1467963
Additional fixes:
* Add kibana_ui_objects directory to downstream Dockerfile
* Other fixes to sync upstream and downstream Dockerfiles
* use ES_HOME instead of hardcoded path
* check for missing files and directories
Comment 13 Rich Megginson 2017-07-19 18:34:39 EDT
koji_builds:
  https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=575170
repositories:
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:rhaos-3.6-rhel-7-docker-candidate-80341-20170719194850
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:latest
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6.153
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6.153-3
Comment 14 Xia Zhao 2017-07-20 03:06:06 EDT
It's fixed: 

1. The adding script can now be run successfully:
# oc exec logging-es-data-master-ccham3fm-1-tx33k -- es_load_kibana_ui_objects xiazhao
[2017-07-20 06:48:36,553][INFO ][container.run            ] Adding Kibana dashboards and other UI objects for user xiazhao index .kibana.f7724d98466ed7391e970202dc54a6460046aadb
[2017-07-20 06:48:36,553][INFO ][container.run            ] Adding the index pattern for project.* . . .
[2017-07-20 06:48:36,701][INFO ][container.run            ] Adding the Kibana UI objects . . .
[2017-07-20 06:48:36,899][INFO ][container.run            ] Success

2. Login kibana after step 1) is done, found there are 2 pre-saved visualizations for cluster-admin user, and normal user not able to view them:

    Kubernetes Logs by Namespace, DC, Pod, Container
    Kubernetes Logs over Time w/ Container Name 

Image tested with: elasticsearch v3.6.153-3
# docker images | grep elasticsearch
${brew-registry}/openshift3/logging-elasticsearch      v3.6                f6e798e43e8f        10 hours ago        434.5 MB
${brew-registry}/openshift3/logging-elasticsearch      v3.6.153-3          f6e798e43e8f        10 hours ago        434.5 MB
Comment 20 errata-xmlrpc 2017-11-28 17:00:15 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188

Note You need to log in before you can comment on or make changes to this bug.