1691230 – Too many wrong logs "Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-x-x" in EO pod log.

Bug 1691230 - Too many wrong logs "Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-x-x" in EO pod log.

Summary: Too many wrong logs "Reusing existing PVC: elasticsearch-elasticsearch-client...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Logging
Sub Component:
Version:	4.1.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	medium
Target Milestone:	---
Target Release:	4.1.0
Assignee:	ewolinet
QA Contact:	Anping Li
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-03-21 08:26 UTC by Qiaoling Tang
Modified:	2019-06-04 10:46 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-06-04 10:46:16 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2019:0758	0	None	None	None	2019-06-04 10:46:23 UTC

Description Qiaoling Tang 2019-03-21 08:26:20 UTC

Description of problem:

There are too many "Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-x-x" in EO pod log when pvc enabled. These logs should only generated when the es pods redeployed, but now the logs generated frequently, and in fact none of the es pods is redeployed.

$ oc get pod -n openshift-operators
NAME                                      READY   STATUS    RESTARTS   AGE
elasticsearch-operator-5b4977987d-d4jxt   1/1     Running   0          7m58s

$ oc logs -n openshift-operators elasticsearch-operator-5b4977987d-d4jxt
time="2019-03-21T05:38:16Z" level=info msg="Go Version: go1.10.3"
time="2019-03-21T05:38:16Z" level=info msg="Go OS/Arch: linux/amd64"
time="2019-03-21T05:38:16Z" level=info msg="operator-sdk Version: 0.0.7"
time="2019-03-21T05:38:16Z" level=info msg="Watching logging.openshift.io/v1alpha1, Elasticsearch, , 5000000000"
time="2019-03-21T05:39:51Z" level=info msg="Creating new PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:39:51Z" level=info msg="Creating new PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:40:24Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:40:25Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:40:49Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:40:49Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:41:10Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:41:10Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:41:34Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:41:34Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:41:45Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:41:45Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:41:55Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:41:55Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:42:05Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:42:05Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:42:15Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:42:15Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:42:24Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:42:25Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:42:34Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:42:34Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:42:44Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:42:44Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:42:54Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:42:54Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:43:04Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:43:04Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:43:13Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:43:14Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:43:23Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:43:23Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:43:33Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:43:33Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:43:43Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:43:43Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:43:53Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:43:53Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:44:02Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:44:03Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:44:12Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:44:12Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:44:22Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:44:22Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:44:32Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:44:32Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:44:42Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:44:42Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:44:51Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:44:52Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:45:01Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:45:01Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:45:11Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:45:11Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:45:21Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:45:21Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:45:31Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:45:31Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:45:40Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:45:41Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:45:50Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:45:50Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"
time="2019-03-21T05:46:00Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-1"
time="2019-03-21T05:46:00Z" level=info msg="Reusing existing PVC: elasticsearch-elasticsearch-clientdatamaster-0-2"

$ oc get pod
NAME                                                  READY   STATUS    RESTARTS   AGE
cluster-logging-operator-78595b68c7-mkrmx             1/1     Running   0          6m38s
elasticsearch-clientdatamaster-0-1-6db4f44f6f-ngkrz   2/2     Running   0          6m20s
elasticsearch-clientdatamaster-0-2-665b59bdc9-6pndv   2/2     Running   0          6m5s
fluentd-578n2                                         1/1     Running   0          6m14s

$ oc get pvc
NAME                                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
elasticsearch-elasticsearch-clientdatamaster-0-1   Bound    pvc-bf558bad-4b9b-11e9-91cf-02b29c8e3c7e   10Gi       RWO            gp2            7m48s
elasticsearch-elasticsearch-clientdatamaster-0-2   Bound    pvc-bf91c0de-4b9b-11e9-91cf-02b29c8e3c7e   10Gi       RWO            gp2            7m48s

Version-Release number of selected component (if applicable):
image: quay.io/openshift/origin-elasticsearch-operator:latest
imageID: quay.io/openshift/origin-elasticsearch-operator@sha256:c30715c23f4dc6f04c22040df5547a738c13f42c6e79a27225f597ce6012a2a3
4.0.0-0.nightly-2019-03-20-153904

How reproducible:
Always

Steps to Reproduce:
1.Deploy logging via community-operators with pvc enabled
2.check EO pod logs
3.

Actual results:


Expected results:


Additional info:

Comment 1 ewolinet 2019-03-21 14:04:17 UTC

https://github.com/openshift/elasticsearch-operator/pull/105

Comment 2 Qiaoling Tang 2019-03-26 05:23:25 UTC

Verified in quay.io/openshift/origin-elasticsearch-operator@sha256:8e7a748802fc284162f5dadf7cdfd9ae2adfb72b6f9682d3b91ee87024aa0a76

Comment 4 errata-xmlrpc 2019-06-04 10:46:16 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758

Note You need to log in before you can comment on or make changes to this bug.