Bug 1508150

Summary:	If an existing PVC used by an Elasticsearch DC is not labeled with logging-infra=support a new PVC is created
Product:	OpenShift Container Platform	Reporter:	Peter Portante <pportant>
Component:	Logging	Assignee:	ewolinet
Status:	CLOSED ERRATA	QA Contact:	Anping Li <anli>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	3.6.1	CC:	aos-bugs, ewolinet, jcantril, pdwyer, pportant, rmeggins
Target Milestone:	---	Keywords:	OpsBlocker
Target Release:	3.6.z
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:	undefined	Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-12-14 21:02:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Peter Portante 2017-10-31 21:26:23 UTC

If an existing PVC used by an Elasticsearch DC is not labeled with logging-infra=support a new PVC is created.

If we start with existing PVCs:

[root@engint-master-39a9e ~]# oc get pvc --show-labels
NAME                  STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS   AGE       LABELS
logging-es-ek5tsdfc   Bound     pvc-618dd908-0da6-11e7-9c90-126ee6b1d97f   187Gi      RWO           ebs            225d      <none>
logging-es-tdxh1mh5   Bound     pvc-623b50a4-0da6-11e7-afc1-123dee44c1c5   187Gi      RWO           ebs            225d      <none>

After running the logging playbook on an existing cluster, the old PVCs are no longer used, and new PVCs are created:

[root@engint-master-39a9e ~]# oc get pvc --show-labels
NAME                  STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS   AGE       LABELS
logging-es-0          Bound     pvc-746e794c-bb4b-11e7-b58e-126ee6b1d97f   200Gi      RWO           gp2            4d        logging-infra=support
logging-es-1          Bound     pvc-7d3d6054-bb4b-11e7-a660-123dee44c1c5   200Gi      RWO           gp2            4d        logging-infra=support
logging-es-ek5tsdfc   Bound     pvc-618dd908-0da6-11e7-9c90-126ee6b1d97f   187Gi      RWO           ebs            225d      <none>
logging-es-tdxh1mh5   Bound     pvc-623b50a4-0da6-11e7-afc1-123dee44c1c5   187Gi      RWO           ebs            225d      <none>

Comment 1 Peter Portante 2017-10-31 21:27:15 UTC

Using openshift-ansible v3.6.173.0.59.

Comment 2 ewolinet 2017-10-31 21:45:39 UTC

This is stemming from the fact that in 3.5 we pulled this information from the DC to populate the PVC name, in 3.6 we try to abstract that using logging_facts which requires a label instead of walking the DC to get the PVC name.

Comment 3 ewolinet 2017-11-01 20:12:25 UTC

https://github.com/openshift/openshift-ansible/pull/5980

Comment 5 Anping Li 2017-11-16 11:17:03 UTC

@peter,  How could we create a pvc named like logging-es-ek5tsdfc?  What are the openshift-ansible version and inventory variables?

Comment 6 Anping Li 2017-11-22 10:09:07 UTC

Verified and pass with openshift-ansible-3.7.9-1.git.4.d445616.el7.noarch
1. Deploy logging
2. Create pvc named 
oc get pvc -o name
persistentvolumeclaims/logging-es-ek5tsdfc
persistentvolumeclaims/logging-es-tdxh1mh5

3. Attached the pvc to ES pods
 oc set volume dc/logging-es-data-master-stfcyny3 --add --overwrite --type=pvc --name=elasticsearch-storage --claim-name=logging-es-ek5tsdfc
oc set volume dc/logging-es-ops-data-master-gfy26cki --add --overwrite --type=pvc --name=elasticsearch-storage --claim-name=logging-es-tdxh1mh5
4. Redeployed logging by playbook
openshift_logging_es_pvc_dynamic=True 
openshift_logging_es_pvc_prefix=logging-es
openshift_logging_es_ops_pvc_prefix=logging-es-ops

5. Check the pvc and dc
   The playbook wasn't create new pvc, the playbook is using the existing pvc.
   #oc get pvc -o name
     persistentvolumeclaims/logging-es-ek5tsdfc
     persistentvolumeclaims/logging-es-tdxh1mh5
   ## oc get dc --selector logging-infra=elasticsearch -o yaml |grep -A 1 persistentVolumeClaim
          persistentVolumeClaim:
            claimName: logging-es-ek5tsdfc
--
          persistentVolumeClaim:
            claimName: logging-es-tdxh1mh5

Comment 9 errata-xmlrpc 2017-12-14 21:02:32 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3438