Bug 1465247 - hawkular-cassandra pod in pending status, since dynamic pv is not created: storageclass.storage.k8s.io "dynamic" not found
hawkular-cassandra pod in pending status, since dynamic pv is not created: st...
Status: CLOSED NEXTRELEASE
Product: OpenShift Container Platform
Classification: Red Hat
Component: Metrics (Show other bugs)
3.6.0
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Matt Wringe
Junqi Zhao
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-26 23:26 EDT by Junqi Zhao
Modified: 2017-08-16 15 EDT (History)
4 users (show)

See Also:
Fixed In Version: 3.6.0
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-14 14:42:20 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Junqi Zhao 2017-06-26 23:26:52 EDT
Description of problem:
Cluster is on openstack and the cloud provider is configured correctly.
Set openshift_metrics_cassandra_storage_type=dynamic in inventory file, after the deployer completes the pvc is not bound and no pv is created.
Describe pvc, error shows: storageclass.storage.k8s.io "dynamic" not found

# oc get pvc
NAME                  STATUS    VOLUME    CAPACITY   ACCESSMODES   STORAGECLASS   AGE
metrics-cassandra-1   Pending                                      dynamic        23m

# oc describe pvc metrics-cassandra-1
Name:		metrics-cassandra-1
Namespace:	openshift-infra
StorageClass:	dynamic
Status:		Pending
Volume:		
Labels:		metrics-infra=hawkular-cassandra
Annotations:	kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{"volume.beta.kubernetes.io/storage-class":"dynamic"},"labels":{"metrics-in...
		volume.beta.kubernetes.io/storage-class=dynamic
Capacity:	
Access Modes:	
Events:
  FirstSeen	LastSeen	Count	From				SubObjectPath	Type		Reason			Message
  ---------	--------	-----	----				-------------	--------	------			-------
  24m		4s		98	persistent-volume-controller			Warning		ProvisioningFailed	storageclass.storage.k8s.io "dynamic" not found


Version-Release number of selected component (if applicable):
# openshift version
openshift v3.6.122
kubernetes v1.6.1+5115d708d7
etcd 3.2.0

Images from brew registry
metrics-hawkular-metrics   v3.6                d8b79436734f        4 hours ago         1.293 GB
metrics-cassandra          v3.6                d36ecb9f528f        4 hours ago         572.9 MB
metrics-heapster           v3.6                ea352c93f79f        4 hours ago         274.3 MB

How reproducible:
Always

Steps to Reproduce:
1. Create an inventory file for metrics (see below)
2. Check pvc
3.

Actual results:
pvc is not bound.hawkular-cassandra pod is in pending status
# oc get po
NAME                         READY     STATUS    RESTARTS   AGE
hawkular-cassandra-1-69jhv   0/1       Pending   0          33m
hawkular-metrics-vc7g4       0/1       Running   4          33m
heapster-8gzjl               0/1       Running   3          33m


Expected results:
pvc is bound and all pods are running well

Additional info:
Inventory file:
[OSEv3:children]
masters

[masters]
${MASTER} openshift_public_hostname=${MASTER}

[OSEv3:vars]
ansible_ssh_user=root
ansible_ssh_private_key_file="~/libra.pem"
deployment_type=openshift-enterprise


# Metrics
openshift_metrics_install_metrics=true
openshift_metrics_hawkular_hostname=hawkular-metrics.${SUB_DOMAIN}
openshift_metrics_project=openshift-infra
openshift_metrics_image_prefix=${IMAGE_PREFIX}
openshift_metrics_image_version=v3.6
openshift_metrics_cassandra_storage_type=dynamic
Comment 2 Jeff Cantrill 2017-06-28 13:53:28 EDT
Moving this to ON_QA because it looks like storage class updates to metrics are available in openshift-4.6.120-1 or greater from https://github.com/openshift/openshift-ansible/pull/4532  What version of openshift-ansible did you test?  Master shows no ref to annotation of 'storageclass.storage.k8s.io'
Comment 3 Junqi Zhao 2017-06-28 23:02:54 EDT
Tested, issue is fixed, dynamic pv is created successfully and all pods are running well.

# oc get pv
NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                                 STORAGECLASS        REASON    AGE
pvc-44fe57a5-5c75-11e7-9584-fa163e36764b   10Gi       RWO           Delete          Bound     openshift-infra/metrics-cassandra-1   cinderprovisioner             5m

# oc get pvc
NAME                  STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS        AGE
metrics-cassandra-1   Bound     pvc-44fe57a5-5c75-11e7-9584-fa163e36764b   10Gi       RWO           cinderprovisioner   5m


# oc get po
NAME                         READY     STATUS    RESTARTS   AGE
hawkular-cassandra-1-4s65c   1/1       Running   0          14m
hawkular-metrics-285zk       1/1       Running   0          14m
heapster-ndd4m               1/1       Running   0          14m


# rpm -qa | grep openshift-ansible
openshift-ansible-docs-3.6.123.1003-1.git.0.002ceeb.el7.noarch
openshift-ansible-roles-3.6.123.1003-1.git.0.002ceeb.el7.noarch
openshift-ansible-3.6.123.1003-1.git.0.002ceeb.el7.noarch
openshift-ansible-filter-plugins-3.6.123.1003-1.git.0.002ceeb.el7.noarch
openshift-ansible-lookup-plugins-3.6.123.1003-1.git.0.002ceeb.el7.noarch
openshift-ansible-playbooks-3.6.123.1003-1.git.0.002ceeb.el7.noarch
openshift-ansible-callback-plugins-3.6.123.1003-1.git.0.002ceeb.el7.noarch

Note You need to log in before you can comment on or make changes to this bug.