Bug 1473600 - Failed to deploy metrics with dynamic pv along with OCP
Summary: Failed to deploy metrics with dynamic pv along with OCP
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 3.6.z
Assignee: Kenny Woodson
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-07-21 09:25 UTC by Junqi Zhao
Modified: 2017-07-24 03:10 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-07-24 03:10:18 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Junqi Zhao 2017-07-21 09:25:06 UTC
Description of problem:

Failed to deploy metrics with dynamic pv along with OCP, covered AWS which enabled cloud provider.
openshift_ansible_vars:
  openshift_hosted_metrics_deploy: true
  openshift_hosted_metrics_storage_kind: dynamic
  openshift_hosted_metrics_deployer_version: v3.6
  openshift_hosted_metrics_deployer_prefix: ${IMAGE_PREFIX}
  openshift_hosted_metrics_storage_volume_size: 10Gi


Version-Release number of selected component (if applicable):
ansible packages
openshift-ansible-3.6.162-1.git.0.50e29bd.el7.noarch.rpm
openshift-ansible-callback-plugins-3.6.162-1.git.0.50e29bd.el7.noarch.rpm
openshift-ansible-docs-3.6.162-1.git.0.50e29bd.el7.noarch.rpm
openshift-ansible-filter-plugins-3.6.162-1.git.0.50e29bd.el7.noarch.rpm
openshift-ansible-lookup-plugins-3.6.162-1.git.0.50e29bd.el7.noarch.rpm
openshift-ansible-playbooks-3.6.162-1.git.0.50e29bd.el7.noarch.rpm
openshift-ansible-roles-3.6.162-1.git.0.50e29bd.el7.noarch.rpm

How reproducible:
Always

Steps to Reproduce:
1. Deploy metrics 3.6 with dynamic pv along with OCP
2.
3.

Actual results:
all pods are not ready, no pv is created and pvc is in pending status
# oc get po
NAME                         READY     STATUS    RESTARTS   AGE
hawkular-cassandra-1-9rt4b   0/1       Pending   0          1h
hawkular-metrics-81z0l       0/1       Running   8          1h
heapster-v7vdj               0/1       Running   7          1h

# oc get pvc
NAME                  STATUS    VOLUME    CAPACITY   ACCESSMODES   STORAGECLASS   AGE
metrics-cassandra-1   Pending                                                     46m

# oc describe pvc metrics-cassandra-1
Name:		metrics-cassandra-1
Namespace:	openshift-infra
StorageClass:	
Status:		Pending
Volume:		
Labels:		metrics-infra=hawkular-cassandra
Annotations:	kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"labels":{"metrics-infra":"hawkular-cassandra"},"name":"metrics-cassandr...
Capacity:	
Access Modes:	
Events:
  FirstSeen	LastSeen	Count	From				SubObjectPath	Type		Reason		Message
  ---------	--------	-----	----				-------------	--------	------		-------
  46m		46m		3	persistentvolume-controller			Normal		FailedBinding	no persistent volumes available for this claim and no storage class is set
  46m		54s		182	persistentvolume-controller			Normal		FailedBinding	no persistent volumes available for this claim and no storage class is set

# oc get storageclass
NAME            TYPE
gp2 (default)   kubernetes.io/aws-ebs   

# oc get storageclass gp2 -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
  creationTimestamp: 2017-07-21T07:15:38Z
  name: gp2
  resourceVersion: "1893"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/gp2
  uid: 65c951ef-6de4-11e7-bd54-0ec8954bea1a
parameters:
  type: gp2
provisioner: kubernetes.io/aws-ebs

*********************************************************************
Discussed with wehe, this is because pvc is created before  default storageclass. We have default storageclass created on AWS and  GCE, but in installation, the default storageclass is created after  metric pvc created.

Expected results:
pv should be created

Additional info:


Description of problem:

Version-Release number of the following components:
rpm -q openshift-ansible
rpm -q ansible
ansible --version

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:
Please include the entire output from the last TASK line through the end of output if an error is generated

Expected results:

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 2 Kenny Woodson 2017-07-21 19:24:28 UTC
I was unable to reproduce this.  Was there any issue with the cloud-provider settings?

My PVC's come up and are instantly bound and notice the storageclass is gp2.

NAMESPACE         NAME                  STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS   AGE
openshift-infra   metrics-cassandra-1   Bound     pvc-b2493044-6e49-11e7-b6b4-0ed8afbf4ec0   100Gi	RWO           gp2            59s
openshift-infra   metrics-cassandra-2   Bound     pvc-b3690c61-6e49-11e7-b2f4-0e63380b0bd8   100Gi	RWO           gp2            58s
logging           logging-es-0          Bound     pvc-e7fcb8cb-6e49-11e7-ad66-0e68060efc74   200Gi	RWO           gp2            9s



NAME            TYPE
gp2 (default)   kubernetes.io/aws-ebs   
[root@ip-172-31-50-42 yum.repos.d]# oc get sc gp2 -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
  creationTimestamp: 2017-07-21T19:15:33Z
  name: gp2
  resourceVersion: "1855"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/gp2
  uid: f82d9d70-6e48-11e7-b2f4-0e63380b0bd8
parameters:
  encrypted: "false"
  kmsKeyId: ""
  type: gp2
provisioner: kubernetes.io/aws-ebs

Comment 3 Junqi Zhao 2017-07-24 03:10:18 UTC
Tested, it is successful to deploy metrics with dynamic pv along with OCP, close this defect as NOTABUG
# oc get pv
NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                                 STORAGECLASS   REASON    AGE
pvc-fc1c3bec-7014-11e7-bdc3-0eb506dd8202   10Gi       RWO           Delete          Bound     openshift-infra/metrics-cassandra-1   gp2                      54m

# oc get pvc
NAME                  STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS   AGE
metrics-cassandra-1   Bound     pvc-fc1c3bec-7014-11e7-bdc3-0eb506dd8202   10Gi       RWO           gp2            54m

# oc get sc gp2 -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
  creationTimestamp: 2017-07-24T01:45:55Z
  name: gp2
  resourceVersion: "1344"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/gp2
  uid: d5885a30-7011-11e7-bdc3-0eb506dd8202
parameters:
  encrypted: "false"
  kmsKeyId: ""
  type: gp2
provisioner: kubernetes.io/aws-ebs


Note You need to log in before you can comment on or make changes to this bug.