Bug 1691230

Summary:	Too many wrong logs "Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-x-x" in EO pod log.
Product:	OpenShift Container Platform	Reporter:	Qiaoling Tang <qitang>
Component:	Logging	Assignee:	ewolinet
Status:	CLOSED ERRATA	QA Contact:	Anping Li <anli>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.1.0	CC:	aos-bugs, ewolinet, rmeggins
Target Milestone:	---
Target Release:	4.1.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-06-04 10:46:16 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Qiaoling Tang 2019-03-21 08:26:20 UTC

Description of problem:

There are too many "Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-x-x" in EO pod log when pvc enabled. These logs should only generated when the es pods redeployed, but now the logs generated frequently, and in fact none of the es pods is redeployed.

$ oc get pod -n openshift-operators
NAME                                      READY   STATUS    RESTARTS   AGE
elasticsearch-operator-5b4977987d-d4jxt   1/1     Running   0          7m58s

$ oc logs -n openshift-operators elasticsearch-operator-5b4977987d-d4jxt
time="2019-03-21T05:38:16Z" level=info msg="Go Version: go1.10.3"
time="2019-03-21T05:38:16Z" level=info msg="Go OS/Arch: linux/amd64"
time="2019-03-21T05:38:16Z" level=info msg="operator-sdk Version: 0.0.7"
time="2019-03-21T05:38:16Z" level=info msg="Watching logging.openshift.io/v1alpha1, Elasticsearch, , 5000000000"
time="2019-03-21T05:39:51Z" level=info msg="Creating new PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:39:51Z" level=info msg="Creating new PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:40:24Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:40:25Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:40:49Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:40:49Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:41:10Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:41:10Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:41:34Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:41:34Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:41:45Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:41:45Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:41:55Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:41:55Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:42:05Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:42:05Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:42:15Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:42:15Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:42:24Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:42:25Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:42:34Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:42:34Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:42:44Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:42:44Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:42:54Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:42:54Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:43:04Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:43:04Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:43:13Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:43:14Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:43:23Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:43:23Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:43:33Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:43:33Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:43:43Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:43:43Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:43:53Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:43:53Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:44:02Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:44:03Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:44:12Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:44:12Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:44:22Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:44:22Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:44:32Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:44:32Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:44:42Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:44:42Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:44:51Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:44:52Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:45:01Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:45:01Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:45:11Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:45:11Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:45:21Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:45:21Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:45:31Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:45:31Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:45:40Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:45:41Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:45:50Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:45:50Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:46:00Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:46:00Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"

$ oc get pod
NAME                                                  READY   STATUS    RESTARTS   AGE
cluster-logging-operator-78595b68c7-mkrmx             1/1     Running   0          6m38s
elasticsearch-clientdatamaster-0-1-6db4f44f6f-ngkrz   2/2     Running   0          6m20s
elasticsearch-clientdatamaster-0-2-665b59bdc9-6pndv   2/2     Running   0          6m5s
fluentd-578n2                                         1/1     Running   0          6m14s

$ oc get pvc
NAME                                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
elasticsearch-elasticsearch-clientdatamaster-0-1   Bound    pvc-bf558bad-4b9b-11e9-91cf-02b29c8e3c7e   10Gi       RWO            gp2            7m48s
elasticsearch-elasticsearch-clientdatamaster-0-2   Bound    pvc-bf91c0de-4b9b-11e9-91cf-02b29c8e3c7e   10Gi       RWO            gp2            7m48s

Version-Release number of selected component (if applicable):
image: quay.io/openshift/origin-elasticsearch-operator:latest
imageID: quay.io/openshift/origin-elasticsearch-operator@sha256:c30715c23f4dc6f04c22040df5547a738c13f42c6e79a27225f597ce6012a2a3
4.0.0-0.nightly-2019-03-20-153904

How reproducible:
Always

Steps to Reproduce:
1.Deploy logging via community-operators with pvc enabled
2.check EO pod logs
3.

Actual results:


Expected results:


Additional info:

Comment 1 ewolinet 2019-03-21 14:04:17 UTC

https://github.com/openshift/elasticsearch-operator/pull/105

Comment 2 Qiaoling Tang 2019-03-26 05:23:25 UTC

Verified in quay.io/openshift/origin-elasticsearch-operator@sha256:8e7a748802fc284162f5dadf7cdfd9ae2adfb72b6f9682d3b91ee87024aa0a76

Comment 4 errata-xmlrpc 2019-06-04 10:46:16 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758