Description of problem: There are no dashboards for container admins. We must have some sort of Kibana visualizations for 3.6. This is a release blocker. There were some dashboards developed during the logging hackfest in June 2017 that must go into 3.6. Version-Release number of selected component (if applicable): 3.6 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Commit pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/b0044ec9023bc6472d125df0a1dee6746ad27b18 Bug 1467963 - There are no Kibana dashboards for container admins https://bugzilla.redhat.com/show_bug.cgi?id=1467963 The script es_load_kibana_ui_objects is used to load dashboards and other Kibana UI objects for the given user. Usage: oc exec $espod -- es_load_kibana_ui_objects user-name That is, it exists inside the Elasticsearch and ES-OPS pod, and must be run inside those pods. Additionally, it requires some indices and other objects set up by the OpenShift Elasticsearch plugin, so the user must login to Kibana (or Elasticsearch) before using this script. This will also add an index pattern for `project.*` and load the necessary index pattern file. This relies on some additions to the viaq schema which is provided by https://github.com/ViaQ/elasticsearch-templates/pull/50
new build - openshift3/logging-elasticsearch:v3.6.140-2
Blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1469918
Hi Rich, * During testing, when I tried to load dashboards and other Kibana UI objects for cluster-admin user xiazhao (I followed the guide to log in to kibana UI with her before trying this), meet this error: $ oc get po NAME READY STATUS RESTARTS AGE logging-curator-1-6gj7l 1/1 Running 0 1h logging-es-data-master-5ywqx64o-1-p2rzb 1/1 Running 0 1h logging-fluentd-gvjd7 1/1 Running 0 1h logging-fluentd-pczkz 1/1 Running 0 1h logging-kibana-1-f95tp 2/2 Running 0 1h $ oc exec logging-es-data-master-5ywqx64o-1-p2rzb -- es_load_kibana_ui_objects xiazhao rpc error: code = 13 desc = invalid header field value "oci runtime error: exec failed: container_linux.go:247: starting container process caused \"exec: \\\"es_load_kibana_ui_objects\\\": executable file not found in $PATH\"\n" * After oc rsh into es pod, I can't find the script " es_load_kibana_ui_objects": # oc rsh logging-es-data-master-5ywqx64o-1-p2rzb sh-4.2$ find . -name es_load_kibana_ui_objects sh-4.2$ * From your PR I found the script here, but don't know how should it run: $ curl -O https://raw.githubusercontent.com/openshift/origin-aggregated-logging/master/elasticsearch/utils/es_load_kibana_ui_objects $ oc exec logging-es-data-master-5ywqx64o-1-p2rzb -- es_load_kibana_ui_objects xiazhao rpc error: code = 13 desc = invalid header field value "oci runtime error: exec failed: container_linux.go:247: starting container process caused \"exec: \\\"es_load_kibana_ui_objects\\\": executable file not found in $PATH\"\n" command terminated with exit code 126 Could you please help point out what else steps should I do? Thanks, Xia
Confirmed that dev is rebuilding the elasticsearch image, set back to assigned, thanks!
koji_builds: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=574393 repositories: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:rhaos-3.6-rhel-7-docker-candidate-85872-20170717180328 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:latest brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6.152.0 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6.152.0-2
Tested with the latest v3.6 es image which is 12 hours newer than v3.6.152.0-2, issue is reproduced: # docker images | grep elasticsearch ${brew-registry}/openshift3/logging-elasticsearch v3.6 9b0f4b948e79 3 hours ago 404.2 MB ${brew-registry}/openshift3/logging-elasticsearch v3.6.152.0-2 1e210744feb3 15 hours ago 404.7 MB Test steps in detail: 1. Login kibana with user "xiazhao" who is cluster-admin 2. When executing the adding dashboard script, get this error: # oc exec logging-es-data-master-x3pw9820-1-v7k4p -- es_load_kibana_ui_objects xiazhao [2017-07-18 09:15:32,266][INFO ][container.run ] Adding Kibana dashboards and other UI objects for user xiazhao index .kibana.f7724d98466ed7391e970202dc54a6460046aadb [2017-07-18 09:15:32,267][INFO ][container.run ] Adding the index pattern for project.* . . . cat: /usr/share/java/elasticsearch/index_patterns/com.redhat.viaq-openshift.index-pattern.json: No such file or directory command terminated with exit code 1 3. Relogin kibana with user "xiazhao", went into the "dashboard" tab, and searched for the kibana visualizations with pattern "*", get 0 foundings @Rich, please help review and point out if any of the above test steps were wrong, thanks!
(In reply to Xia Zhao from comment #7) > Tested with the latest v3.6 es image which is 12 hours newer than > v3.6.152.0-2, @Xia can you explain this comment? If you tested an image is 12 hours newer then it is not the same image as: issue is reproduced: > # docker images | grep elasticsearch > ${brew-registry}/openshift3/logging-elasticsearch v3.6 > 9b0f4b948e79 3 hours ago 404.2 MB > ${brew-registry}/openshift3/logging-elasticsearch v3.6.152.0-2 > 1e210744feb3 15 hours ago 404.7 MB > Or are you stating you tested 2 sets of images: 1. 12 hours newer 2. v3.6.152.0-2
koji_builds: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=574625 repositories: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:rhaos-3.6-rhel-7-docker-candidate-57374-20170718145154 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:latest brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6.153 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6.153-2
(In reply to Jeff Cantrill from comment #8) > (In reply to Xia Zhao from comment #7) > > Tested with the latest v3.6 es image which is 12 hours newer than > > v3.6.152.0-2, > > @Xia can you explain this comment? If you tested an image is 12 hours newer > then it is not the same image as: > > issue is reproduced: > > # docker images | grep elasticsearch > > ${brew-registry}/openshift3/logging-elasticsearch v3.6 > > 9b0f4b948e79 3 hours ago 404.2 MB > > ${brew-registry}/openshift3/logging-elasticsearch v3.6.152.0-2 > > 1e210744feb3 15 hours ago 404.7 MB > > > > Or are you stating you tested 2 sets of images: > > 1. 12 hours newer > 2. v3.6.152.0-2 Hi Jeff, By specifying openshift_logging_image_version=v3.6 in the inventory file when deploy the logging tsacks, I tested with this image: > > ${brew-registry}/openshift3/logging-elasticsearch v3.6 > > 9b0f4b948e79 3 hours ago 404.2 MB Thanks, Xia
Verified with elasticsearch image v3.6.153-2, get this error message while loading the kibana dashboards, and checked on kibana UI, 0 visualizations was found under the dashboard tab for cluster-admin user: # oc exec logging-es-data-master-odlf264i-1-bc62n -- es_load_kibana_ui_objects xiazhao [2017-07-19 06:35:54,131][INFO ][container.run ] Adding Kibana dashboards and other UI objects for user xiazhao index .kibana.f7724d98466ed7391e970202dc54a6460046aadb [2017-07-19 06:35:54,132][INFO ][container.run ] Adding the index pattern for project.* . . . [2017-07-19 06:35:54,306][INFO ][container.run ] Adding the Kibana UI objects . . . cat: /usr/share/java/elasticsearch/kibana_ui_objects/*.json: No such file or directory Traceback (most recent call last): File "<string>", line 4, in <module> File "/usr/lib64/python2.7/json/__init__.py", line 290, in load **kw) File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads return _default_decoder.decode(s) File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded command terminated with exit code 1
Commit pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/a4dd522cd1b55a3303343cdf6ab7694ab5954dae Bug 1467963 There are no Kibana dashboards for container admins https://bugzilla.redhat.com/show_bug.cgi?id=1467963 Additional fixes: * Add kibana_ui_objects directory to downstream Dockerfile * Other fixes to sync upstream and downstream Dockerfiles * use ES_HOME instead of hardcoded path * check for missing files and directories
koji_builds: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=575170 repositories: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:rhaos-3.6-rhel-7-docker-candidate-80341-20170719194850 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:latest brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6.153 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6.153-3
It's fixed: 1. The adding script can now be run successfully: # oc exec logging-es-data-master-ccham3fm-1-tx33k -- es_load_kibana_ui_objects xiazhao [2017-07-20 06:48:36,553][INFO ][container.run ] Adding Kibana dashboards and other UI objects for user xiazhao index .kibana.f7724d98466ed7391e970202dc54a6460046aadb [2017-07-20 06:48:36,553][INFO ][container.run ] Adding the index pattern for project.* . . . [2017-07-20 06:48:36,701][INFO ][container.run ] Adding the Kibana UI objects . . . [2017-07-20 06:48:36,899][INFO ][container.run ] Success 2. Login kibana after step 1) is done, found there are 2 pre-saved visualizations for cluster-admin user, and normal user not able to view them: Kubernetes Logs by Namespace, DC, Pod, Container Kubernetes Logs over Time w/ Container Name Image tested with: elasticsearch v3.6.153-3 # docker images | grep elasticsearch ${brew-registry}/openshift3/logging-elasticsearch v3.6 f6e798e43e8f 10 hours ago 434.5 MB ${brew-registry}/openshift3/logging-elasticsearch v3.6.153-3 f6e798e43e8f 10 hours ago 434.5 MB
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:3188