Bug 1490989 - Maximal ebs-volumes of an ec2-instance: 52/54?
Summary: Maximal ebs-volumes of an ec2-instance: 52/54?
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: ---
: 3.6.z
Assignee: Hemant Kumar
QA Contact: Chao Yang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-09-12 16:57 UTC by Hongkai Liu
Modified: 2018-04-30 04:01 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2018-04-30 04:00:18 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:1233 None None None 2018-04-30 04:01:03 UTC

Description Hongkai Liu 2017-09-12 16:57:08 UTC
Description of problem:
Use dynamic provision of ebs-pvc for pods.
For a single compute node, we have at most 52 running pods. The rest are in pending status up to pvc provision.

Version-Release number of selected component (if applicable):
oc v3.6.172.0.0

How reproducible:
Cluster: 2 masters, 1 infra, 2 compute, 1 lb.
KUBE_MAX_PD_VOLS=260 in /etc/sysconfig/atomic-openshift-master-controllers on both master nodes.

Steps to Reproduce:
1. make one of the compute node SchedulingDisabled
2. create 60 pods with pvc volumes

Actual results:
52 pods are in running status. The rest 8 are in pending status.

Expected results:
60 running pods.

Master Log:
Sep  6 16:09:15 ip-172-31-27-240 atomic-openshift-master-controllers: E0906 16:09:15.912766   23136 attacher.go:73] Error attaching volume "aws://us-west-2b/vol-064280bb36606dab9": Too many EBS volumes attached to node ip-172-31-50-156.us-west-2.compute.internal.

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:
related issues in the upstream:
https://github.com/kubernetes/kubernetes/issues/41453
https://github.com/openshift/origin/issues/13025
https://github.com/kubernetes/kubernetes/pull/41455

Comment 3 Mike Fiedler 2017-09-13 11:41:29 UTC
@bchilds - re-opening this to ask if this should be documented.   I could not find any mention of this limitation/magic number in the openshift doc - just the upstream issues (e.g. https://github.com/kubernetes/kubernetes/issues/41453) which hinted that it had been addressed.

If you don't feel it warrants documentation, feel free to close it again.   Thanks.

Comment 6 Hemant Kumar 2018-01-04 16:21:23 UTC
Yep, that is correct - the environment variable KUBE_MAX_PD_VOLS deos allow one to go beyond default 39 limit. 

I have opened documentation PR for documenting this - https://github.com/openshift/openshift-docs/pull/7002

Comment 7 Hongkai Liu 2018-01-04 17:21:30 UTC
Thanks for clarification, @Hemant.

Comment 9 Hongkai Liu 2018-01-24 16:03:39 UTC
The PR 7002 has not been merged yet.
So I guess I cannot verify the document yet.

Comment 12 Hongkai Liu 2018-04-16 15:59:35 UTC
Verified on this page:
https://docs.openshift.com/container-platform/3.6/install_config/persistent_storage/persistent_storage_aws.html

It looks good to me.
Thanks for updating the doc.

Comment 13 Hongkai Liu 2018-04-16 16:02:38 UTC
Also verified on origin doc (latest)
https://docs.openshift.org/latest/install_config/persistent_storage/persistent_storage_aws.html

Comment 24 errata-xmlrpc 2018-04-30 04:00:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:1233


Note You need to log in before you can comment on or make changes to this bug.