Bug 1463577 - Elasticsearch not starting - dynamic pv not created: storageclass.storage.k8s.io "dynamic" not found
Elasticsearch not starting - dynamic pv not created: storageclass.storage.k8s...
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer (Show other bugs)
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: Kenny Woodson
Mike Fiedler
Depends On:
  Show dependency treegraph
Reported: 2017-06-21 05:14 EDT by Mike Fiedler
Modified: 2017-08-16 15 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2017-08-10 01:28:56 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Mike Fiedler 2017-06-21 05:14:00 EDT
Description of problem:

Cluster is on AWS EC2 and the AWS cloud provider is configured correctly.

The logging deployer creates a pvc with an annotation of 

volume.beta.kubernetes.io/storage-class: dynamic

After the deployer completes the pvc is not bound and no pv is created.   The events show:

4s        24m       98        logging-es-0                               PersistentVolumeClaim                                            Warning   ProvisioningFailed            persistent-volume-controller                           storageclass.storage.k8s.io "dynamic" not found

The only storageclass that exists is:

root@ip-172-31-36-76: ~ # oc get storageclass
NAME            TYPE
gp2 (default)   kubernetes.io/aws-ebs 

Removing the annotation and allowing the pvc to use the default storage class allows the pv to be created dynamically and the pvc binds to it.   Without this, the pvc just stays stuck in Pending and elasticsearch never starts.

Version-Release number of selected component (if applicable): 3.6.116 and 3.6.121 using openshift-ansible HEAD 62fcd88038910c52796f0e5b37e1e0d8019b80cf

How reproducible: Always

Steps to Reproduce:
1.  Create an inventory file for logging (see below)
2.  Run the playbooks/byo/openshift-cluster/openshift-logging.yaml playbook
3.  oc get pvc when it completes

Actual results:

pvc is not bound.  elasticsearch pod is not started

Expected results:

pvc is bound and elasticsearch starts ok

Additional info:



Comment 1 Mike Fiedler 2017-06-21 10:30:50 EDT
This is a "standard" openshift-ansible byo/config.yml install with a properly configured AWS cloud provider.
Comment 2 Scott Dodson 2017-06-21 13:13:34 EDT

How should we be making use of https://github.com/openshift/openshift-ansible/pull/4262 to use the default storage class on a given environment?

Should we actually be naming those storage classes 'default' rather than 'aws' and 'gcp' and then configure things like logging and metrics to use 'default' ?
Comment 3 Hemant Kumar 2017-06-21 14:05:20 EDT
@Scott - can we not dynamically infer storageClass name that default class name will be `gp2` on AWS and `standard` on GCE?

There was some discussion upstream and we chose not to name default classes `default` because it looks weird when you list them.

~> oc get storageclasses
default(default) xxxxx

Alternately, things like metrics and logging can skip storageclass annotation in their definition and that in turn will cause them to bind to default storageclass automatically (as Mike found out).
Comment 4 Kenny Woodson 2017-06-21 15:52:18 EDT
I have performed a small refactor on this section of the code and removed the annotation.  By removing the annotation, the pvc will use the default storage class.  If you pass the dynamic option as false then it will set the storageClassName option to "" which in turns disables the storage class for that pvc.
Comment 5 Kenny Woodson 2017-06-21 15:53:34 EDT
This time with github link: https://github.com/openshift/openshift-ansible/pull/4532
Comment 7 Mike Fiedler 2017-06-27 05:11:23 EDT
Not fixed in penshift-ansible.noarch      3.6.126-1.git.0.58d33f0.el7

Moving back to POST until available in a puddle.
Comment 8 Scott Dodson 2017-06-27 17:31:42 EDT
in openshift-ansible-
Comment 9 Mike Fiedler 2017-06-30 02:46:48 EDT
verified on
Comment 11 errata-xmlrpc 2017-08-10 01:28:56 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.