Description of problem: In 3.11, logging into Kibana gives 2 errors: Elasticsearch plugin is red Unable to connect to Elasticsearch at http://localhost:9200. The kibana pod logs are full of: {"type":"log","@timestamp":"2018-07-30T17:29:04Z","tags":["warning","elasticsearch","admin"],"pid":226,"message":"Unable to revive connection: http://localhost:9200/"} {"type":"log","@timestamp":"2018-07-30T17:29:04Z","tags":["warning","elasticsearch","admin"],"pid":226,"message":"No living connections"} {"type":"log","@timestamp":"2018-07-30T17:29:07Z","tags":["warning","elasticsearch","admin"],"pid":226,"message":"Unable to revive connection: http://localhost:9200/"} {"type":"log","@timestamp":"2018-07-30T17:29:07Z","tags":["warning","elasticsearch","admin"],"pid":226,"message":"No living connections"} The env vars all look ok - no use of localhost: ELASTICSEARCH_URL=https://logging-es:9200 KIBANA_BIN=/usr/share/kibana/bin/kibana KIBANA_CONF_DIR=/etc/kibana KIBANA_HOME=/usr/share/kibana KIBANA_MEMORY_LIMIT=771751936 KIBANA_VER=5.6.10 KUBERNETES_PORT=tcp://172.24.0.1:443 KUBERNETES_PORT_443_TCP=tcp://172.24.0.1:443 KUBERNETES_PORT_443_TCP_ADDR=172.24.0.1 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_PORT_53_TCP=tcp://172.24.0.1:53 KUBERNETES_PORT_53_TCP_ADDR=172.24.0.1 KUBERNETES_PORT_53_TCP_PORT=53 KUBERNETES_PORT_53_TCP_PROTO=tcp KUBERNETES_PORT_53_UDP=udp://172.24.0.1:53 KUBERNETES_PORT_53_UDP_ADDR=172.24.0.1 KUBERNETES_PORT_53_UDP_PORT=53 KUBERNETES_PORT_53_UDP_PROTO=udp KUBERNETES_SERVICE_HOST=172.24.0.1 KUBERNETES_SERVICE_PORT=443 KUBERNETES_SERVICE_PORT_DNS=53 KUBERNETES_SERVICE_PORT_DNS_TCP=53 KUBERNETES_SERVICE_PORT_HTTPS=443 LOGGING_ES_PORT=tcp://172.26.233.225:9200 LOGGING_ES_PORT_9200_TCP=tcp://172.26.233.225:9200 LOGGING_ES_PORT_9200_TCP_ADDR=172.26.233.225 LOGGING_ES_PORT_9200_TCP_PORT=9200 LOGGING_ES_PORT_9200_TCP_PROTO=tcp LOGGING_ES_PROMETHEUS_PORT=tcp://172.26.28.112:443 LOGGING_ES_PROMETHEUS_PORT_443_TCP=tcp://172.26.28.112:443 LOGGING_ES_PROMETHEUS_PORT_443_TCP_ADDR=172.26.28.112 LOGGING_ES_PROMETHEUS_PORT_443_TCP_PORT=443 LOGGING_ES_PROMETHEUS_PORT_443_TCP_PROTO=tcp LOGGING_ES_PROMETHEUS_SERVICE_HOST=172.26.28.112 LOGGING_ES_PROMETHEUS_SERVICE_PORT=443 LOGGING_ES_PROMETHEUS_SERVICE_PORT_PROXY=443 LOGGING_ES_SERVICE_HOST=172.26.233.225 LOGGING_ES_SERVICE_PORT=9200 LOGGING_KIBANA_PORT=tcp://172.27.107.118:443 LOGGING_KIBANA_PORT_443_TCP=tcp://172.27.107.118:443 LOGGING_KIBANA_PORT_443_TCP_ADDR=172.27.107.118 LOGGING_KIBANA_PORT_443_TCP_PORT=443 LOGGING_KIBANA_PORT_443_TCP_PROTO=tcp LOGGING_KIBANA_SERVICE_HOST=172.27.107.118 LOGGING_KIBANA_SERVICE_PORT=443 Version-Release number of selected component (if applicable): 3.11.0-0.10.0 How reproducible: Always Steps to Reproduce: 1. Install logging 3.11.0-0.10.0 2. Attempt to login to kibana Additional info: Inventory: [OSEv3:children] masters etcd [masters] ip-172-31-3-24 [etcd] ip-172-31-3-24 [OSEv3:vars] deployment_type=openshift-enterprise openshift_deployment_type=openshift-enterprise openshift_release=v3.11 openshift_docker_additional_registries=registry.reg-aws.openshift.com openshift_logging_master_url=https://ec2-35-165-251-149.us-west-2.compute.amazonaws.com:8443 openshift_logging_master_public_url=https://ec2-35-165-251-149.us-west-2.compute.amazonaws.com:8443 openshift_logging_kibana_hostname=kibana.apps.0730-z7d.qe.rhcloud.com oreg_url=registry.reg-aws.openshift.com:443/openshift3/ose-${component}:${version} openshift_logging_image_version=v3.11 openshift_logging_install_logging=true openshift_logging_es_cluster_size=1 openshift_logging_es_pvc_dynamic=true openshift_logging_es_pvc_size=50Gi openshift_logging_es_pvc_storage_class_name=gp2 openshift_logging_fluentd_read_from_head=false openshift_logging_es_memory_limit=8Gi openshift_logging_curator_nodeselector={"node-role.kubernetes.io/infra": "true"} openshift_logging_kibana_nodeselector={"node-role.kubernetes.io/infra": "true"} openshift_logging_es_nodeselector={"node-role.kubernetes.io/infra": "true"} openshift_logging_install_eventrouter=true
This PR fixes Kibana not being able to connect to Elasticsearch but in the course of investigation found issues with permissions and seeding which I will investigate and address in subsequent BZ/PR https://github.com/openshift/origin-aggregated-logging/pull/1269
It is using the correct router name With openshift3/ose-logging-kibana5/images/v3.11.0-0.11.0.0. {"type":"log","@timestamp":"2018-08-06T08:50:36Z","tags":["warning","elasticsearch","admin"],"pid":226,"message":"Unable to revive connection: https://logging-es:9200/"} {"type":"log","@timestamp":"2018-08-06T08:50:36Z","tags":["warning","elasticsearch","admin"],"pid":226,"message":"No living connections"} Note that, you will hit https://bugzilla.redhat.com/show_bug.cgi?id=1595956.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2652