Bug 1463577 - Elasticsearch not starting - dynamic pv not created: storageclass.storage.k8s.io "dynamic" not found
Elasticsearch not starting - dynamic pv not created: storageclass.storage.k8s...
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer (Show other bugs)
3.6.0
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: Kenny Woodson
Mike Fiedler
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-21 05:14 EDT by Mike Fiedler
Modified: 2017-08-16 15 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-10 01:28:56 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Mike Fiedler 2017-06-21 05:14:00 EDT
Description of problem:

Cluster is on AWS EC2 and the AWS cloud provider is configured correctly.

The logging deployer creates a pvc with an annotation of 

volume.beta.kubernetes.io/storage-class: dynamic

After the deployer completes the pvc is not bound and no pv is created.   The events show:

4s        24m       98        logging-es-0                               PersistentVolumeClaim                                            Warning   ProvisioningFailed            persistent-volume-controller                           storageclass.storage.k8s.io "dynamic" not found

The only storageclass that exists is:

root@ip-172-31-36-76: ~ # oc get storageclass
NAME            TYPE
gp2 (default)   kubernetes.io/aws-ebs 

Removing the annotation and allowing the pvc to use the default storage class allows the pv to be created dynamically and the pvc binds to it.   Without this, the pvc just stays stuck in Pending and elasticsearch never starts.

 
Version-Release number of selected component (if applicable): 3.6.116 and 3.6.121 using openshift-ansible HEAD 62fcd88038910c52796f0e5b37e1e0d8019b80cf


How reproducible: Always


Steps to Reproduce:
1.  Create an inventory file for logging (see below)
2.  Run the playbooks/byo/openshift-cluster/openshift-logging.yaml playbook
3.  oc get pvc when it completes

Actual results:

pvc is not bound.  elasticsearch pod is not started

Expected results:

pvc is bound and elasticsearch starts ok


Additional info:

[oo_first_master]
ip-172-31-36-76

[oo_first_master:vars]
openshift_deployment_type=openshift-enterprise
openshift_release=v3.6.0

openshift_logging_install_logging=true
openshift_logging_use_ops=false
openshift_logging_master_url=https://ec2-34-223-225-62.us-west-2.compute.amazonaws.com:8443
openshift_logging_master_public_url=https://ec2-34-223-225-62.us-west-2.compute.amazonaws.com:8443
openshift_logging_kibana_hostname=kibana.0620-8i0.qe.rhcloud.com
openshift_logging_namespace=logging
openshift_logging_image_prefix=registry.ops.openshift.com/openshift3/
openshift_logging_image_version=v3.6.121
openshift_logging_es_pvc_dynamic=true
openshift_logging_es_pvc_size=50Gi
openshift_logging_fluentd_use_journal=true
#openshift_logging_fluentd_journal_read_from_head=false
openshift_logging_use_mux=true
openshift_logging_use_mux_client=true
openshift_logging_es_pv_selector=None
Comment 1 Mike Fiedler 2017-06-21 10:30:50 EDT
This is a "standard" openshift-ansible byo/config.yml install with a properly configured AWS cloud provider.
Comment 2 Scott Dodson 2017-06-21 13:13:34 EDT
Hemant,

How should we be making use of https://github.com/openshift/openshift-ansible/pull/4262 to use the default storage class on a given environment?

Should we actually be naming those storage classes 'default' rather than 'aws' and 'gcp' and then configure things like logging and metrics to use 'default' ?
Comment 3 Hemant Kumar 2017-06-21 14:05:20 EDT
@Scott - can we not dynamically infer storageClass name that default class name will be `gp2` on AWS and `standard` on GCE?

There was some discussion upstream and we chose not to name default classes `default` because it looks weird when you list them.

~> oc get storageclasses
default(default) xxxxx

Alternately, things like metrics and logging can skip storageclass annotation in their definition and that in turn will cause them to bind to default storageclass automatically (as Mike found out).
Comment 4 Kenny Woodson 2017-06-21 15:52:18 EDT
I have performed a small refactor on this section of the code and removed the annotation.  By removing the annotation, the pvc will use the default storage class.  If you pass the dynamic option as false then it will set the storageClassName option to "" which in turns disables the storage class for that pvc.
Comment 5 Kenny Woodson 2017-06-21 15:53:34 EDT
This time with github link: https://github.com/openshift/openshift-ansible/pull/4532
Comment 7 Mike Fiedler 2017-06-27 05:11:23 EDT
Not fixed in penshift-ansible.noarch      3.6.126-1.git.0.58d33f0.el7

Moving back to POST until available in a puddle.
Comment 8 Scott Dodson 2017-06-27 17:31:42 EDT
in openshift-ansible-3.6.123.1002-1.git.0.506cfa7.el7
Comment 9 Mike Fiedler 2017-06-30 02:46:48 EDT
verified on 3.6.126.1
Comment 11 errata-xmlrpc 2017-08-10 01:28:56 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716

Note You need to log in before you can comment on or make changes to this bug.