Bug 1456139

Summary:	logging-es-data-node pods are not deployed
Product:	OpenShift Container Platform	Reporter:	Xia Zhao <xiazhao>
Component:	Installer	Assignee:	ewolinet
Status:	CLOSED ERRATA	QA Contact:	Xia Zhao <xiazhao>
Severity:	high	Docs Contact:
Priority:	high
Version:	3.6.0	CC:	aos-bugs, eminguez, ewolinet, jokerman, mmccomas, rkant, rmeggins, wsun
Target Milestone:	---	Keywords:	TestBlocker
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:	As part of 3.6 with logging we introduced a readiness probe but the timeout threshold wasn't high enough. This bumped it up	Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-08-10 05:26:47 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Xia Zhao 2017-05-27 08:25:41 UTC

Description of problem:
Deploy logging v3.6 on OCP, after CEK got running, the logging-es-data-master pod can''t be ready because of Readiness probe failure:
Readiness probe failed: Elasticsearch node is not ready to accept HTTP requests yet [response code: 000]

Version-Release number of selected component (if applicable):
openshift-ansible-playbooks-3.6.85-1.git.0.109a54e.el7.noarch

# openshift version
openshift v3.6.85
kubernetes v1.6.1+5115d708d7
etcd 3.1.0

How reproducible:
Always

Steps to Reproduce:
1. Deploy logging v3.6 on OCP
2. Check logging components
3. Describe es pod

Actual results:
2. the logging-es-data-node pods were not deployed
3. the logging-es-data-master pod can''t be ready because of Readiness probe failure:
Readiness probe failed: Elasticsearch node is not ready to accept HTTP requests yet [response code: 000]

Expected results:
2. logging-es-data-node pod and able to interact with masters
3. es-master node should be container ready

Additional info:
1. env attached
2. logging components dump:
# oc get po
NAME READY STATUS RESTARTS AGE
logging-curator-1-hnd6v 1/1 Running 1 6h
logging-es-data-master-xpehwc99-1-qg5l8 0/1 Running 0 6h
logging-fluentd-k2nnr 1/1 Running 2 6h
logging-fluentd-zswx9 1/1 Running 2 6h
logging-kibana-1-deploy 0/1 Error 0 6h

Comment 2 Xia Zhao 2017-05-27 08:45:10 UTC

keyword "TestBlocker" added because the disable of es cluster will block logging tests for QE.

Comment 6 Xia Zhao 2017-06-06 08:37:39 UTC

Checked with openshift-ansible-playbooks-3.6.95-1.git.0.dbd2cf1.el7.noarch, the bug fix not yet included in:

# oc get po
NAME                                      READY     STATUS    RESTARTS   AGE
logging-curator-1-9jmxd                   1/1       Running   0          15h
logging-es-data-master-hqad8ai3-1-k81wn   1/1       Running   0          15h
logging-fluentd-6knml                     1/1       Running   0          15h
logging-fluentd-z43f8                     1/1       Running   0          15h
logging-kibana-1-deploy                   0/1       Error     0          15h

Comment 9 Xia Zhao 2017-06-09 09:41:40 UTC

@Eric,

Thanks, I see the es cluster is working fine, see "Cluster health is in [GREEN], create_mapping [com.redhat.viaq.common], update_mapping [com.redhat.viaq.common] in es log. Just one question to double confirm:

--Do we expect a sperate pod with name "logging-es-data-node" to be deployed to openshift system? 

If not, then it's safe enough to set this bz to "verified" status.

Tested with ansible version:
openshift-ansible-playbooks-3.6.98-1.git.0.e651d65.el7.noarch

Test env:
# openshift version
openshift v3.6.101
kubernetes v1.6.1+5115d708d7
etcd 3.2.0-rc.1

Images tested with:
openshift3/logging-elasticsearch   v3.6                                                     5655f1a59ed6        2 hours ago         404.5 MB

Comment 10 Xia Zhao 2017-06-12 04:01:04 UTC

@Eric,

Could you please be so kind to help confirm this question? Thanks in advance!
--Do we expect a sperate pod with name "logging-es-data-node" to be deployed to openshift system?

Comment 11 ewolinet 2017-06-12 15:24:21 UTC

@Xia

You would only see a data-node if you configured a non-master data node with the new openshift_logging_elasticsearch role. Currently there is not a way to do it using the role/playbook structure.

Comment 12 Xia Zhao 2017-06-13 02:00:20 UTC

Thanks Eric for the explanation. Setting this bz to verified according to test result in comment #9.

Comment 14 errata-xmlrpc 2017-08-10 05:26:47 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716