Bug 1456139 - logging-es-data-node pods are not deployed
Summary: logging-es-data-node pods are not deployed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: ewolinet
QA Contact: Xia Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-05-27 08:25 UTC by Xia Zhao
Modified: 2017-12-18 13:04 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
As part of 3.6 with logging we introduced a readiness probe but the timeout threshold wasn't high enough. This bumped it up
Clone Of:
Environment:
Last Closed: 2017-08-10 05:26:47 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1716 0 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.6 RPM Release Advisory 2017-08-10 09:02:50 UTC

Description Xia Zhao 2017-05-27 08:25:41 UTC
Description of problem:
Deploy logging v3.6 on OCP, after CEK got running, the logging-es-data-master pod can''t be ready because of Readiness probe failure:
Readiness probe failed: Elasticsearch node is not ready to accept HTTP requests yet [response code: 000]

Version-Release number of selected component (if applicable):
openshift-ansible-playbooks-3.6.85-1.git.0.109a54e.el7.noarch

# openshift version
openshift v3.6.85
kubernetes v1.6.1+5115d708d7
etcd 3.1.0

How reproducible:
Always

Steps to Reproduce:
1. Deploy logging v3.6 on OCP 
2. Check logging components
3. Describe es pod

Actual results:
2. the logging-es-data-node pods were not deployed
3. the logging-es-data-master pod can''t be ready because of Readiness probe failure:
Readiness probe failed: Elasticsearch node is not ready to accept HTTP requests yet [response code: 000]

Expected results:
2. logging-es-data-node pod and able to interact with masters
3. es-master node should be container ready

Additional info:
1. env attached
2. logging components dump:
# oc get po 
NAME                                      READY     STATUS    RESTARTS   AGE
logging-curator-1-hnd6v                   1/1       Running   1          6h
logging-es-data-master-xpehwc99-1-qg5l8   0/1       Running   0          6h
logging-fluentd-k2nnr                     1/1       Running   2          6h
logging-fluentd-zswx9                     1/1       Running   2          6h
logging-kibana-1-deploy                   0/1       Error     0          6h

Comment 2 Xia Zhao 2017-05-27 08:45:10 UTC
keyword "TestBlocker" added because the disable of es cluster will block logging tests for QE.

Comment 6 Xia Zhao 2017-06-06 08:37:39 UTC
Checked with openshift-ansible-playbooks-3.6.95-1.git.0.dbd2cf1.el7.noarch, the bug fix not yet included in:

# oc get po
NAME                                      READY     STATUS    RESTARTS   AGE
logging-curator-1-9jmxd                   1/1       Running   0          15h
logging-es-data-master-hqad8ai3-1-k81wn   1/1       Running   0          15h
logging-fluentd-6knml                     1/1       Running   0          15h
logging-fluentd-z43f8                     1/1       Running   0          15h
logging-kibana-1-deploy                   0/1       Error     0          15h

Comment 9 Xia Zhao 2017-06-09 09:41:40 UTC
@Eric,

Thanks, I see the es cluster is working fine, see "Cluster health is in [GREEN], create_mapping [com.redhat.viaq.common], update_mapping [com.redhat.viaq.common] in es log. Just one question to double confirm:

--Do we expect a sperate pod with name "logging-es-data-node" to be deployed to openshift system? 

If not, then it's safe enough to set this bz to "verified" status.

Tested with ansible version:
openshift-ansible-playbooks-3.6.98-1.git.0.e651d65.el7.noarch

Test env:
# openshift version
openshift v3.6.101
kubernetes v1.6.1+5115d708d7
etcd 3.2.0-rc.1

Images tested with:
openshift3/logging-elasticsearch   v3.6                                                     5655f1a59ed6        2 hours ago         404.5 MB

Comment 10 Xia Zhao 2017-06-12 04:01:04 UTC
@Eric,

Could you please be so kind to help confirm this question? Thanks in advance!
--Do we expect a sperate pod with name "logging-es-data-node" to be deployed to openshift system?

Comment 11 ewolinet 2017-06-12 15:24:21 UTC
@Xia

You would only see a data-node if you configured a non-master data node with the new openshift_logging_elasticsearch role. Currently there is not a way to do it using the role/playbook structure.

Comment 12 Xia Zhao 2017-06-13 02:00:20 UTC
Thanks Eric for the explanation. Setting this bz to verified according to test result in comment #9.

Comment 14 errata-xmlrpc 2017-08-10 05:26:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716


Note You need to log in before you can comment on or make changes to this bug.