Bug 1490989

Summary:	Maximal ebs-volumes of an ec2-instance: 52/54?
Product:	OpenShift Container Platform	Reporter:	Hongkai Liu <hongkliu>
Component:	Storage	Assignee:	Hemant Kumar <hekumar>
Status:	CLOSED ERRATA	QA Contact:	Chao Yang <chaoyang>
Severity:	low	Docs Contact:
Priority:	unspecified
Version:	3.6.0	CC:	aos-bugs, aos-storage-staff, bchilds, chaoyang, hekumar, hongkliu, mifiedle, smunilla, wsun, wzheng
Target Milestone:	---	Keywords:	Reopened
Target Release:	3.6.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:	undefined	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-04-30 04:00:18 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Hongkai Liu 2017-09-12 16:57:08 UTC

Description of problem:
Use dynamic provision of ebs-pvc for pods.
For a single compute node, we have at most 52 running pods. The rest are in pending status up to pvc provision.

Version-Release number of selected component (if applicable):
oc v3.6.172.0.0

How reproducible:
Cluster: 2 masters, 1 infra, 2 compute, 1 lb.
KUBE_MAX_PD_VOLS=260 in /etc/sysconfig/atomic-openshift-master-controllers on both master nodes.

Steps to Reproduce:
1. make one of the compute node SchedulingDisabled
2. create 60 pods with pvc volumes

Actual results:
52 pods are in running status. The rest 8 are in pending status.

Expected results:
60 running pods.

Master Log:
Sep  6 16:09:15 ip-172-31-27-240 atomic-openshift-master-controllers: E0906 16:09:15.912766   23136 attacher.go:73] Error attaching volume "aws://us-west-2b/vol-064280bb36606dab9": Too many EBS volumes attached to node ip-172-31-50-156.us-west-2.compute.internal.

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:
related issues in the upstream:
https://github.com/kubernetes/kubernetes/issues/41453
https://github.com/openshift/origin/issues/13025
https://github.com/kubernetes/kubernetes/pull/41455

Comment 3 Mike Fiedler 2017-09-13 11:41:29 UTC

@bchilds - re-opening this to ask if this should be documented.   I could not find any mention of this limitation/magic number in the openshift doc - just the upstream issues (e.g. https://github.com/kubernetes/kubernetes/issues/41453) which hinted that it had been addressed.

If you don't feel it warrants documentation, feel free to close it again.   Thanks.

Comment 6 Hemant Kumar 2018-01-04 16:21:23 UTC

Yep, that is correct - the environment variable KUBE_MAX_PD_VOLS deos allow one to go beyond default 39 limit. 

I have opened documentation PR for documenting this - https://github.com/openshift/openshift-docs/pull/7002

Comment 7 Hongkai Liu 2018-01-04 17:21:30 UTC

Thanks for clarification, @Hemant.

Comment 9 Hongkai Liu 2018-01-24 16:03:39 UTC

The PR 7002 has not been merged yet.
So I guess I cannot verify the document yet.

Comment 12 Hongkai Liu 2018-04-16 15:59:35 UTC

Verified on this page:
https://docs.openshift.com/container-platform/3.6/install_config/persistent_storage/persistent_storage_aws.html

It looks good to me.
Thanks for updating the doc.

Comment 13 Hongkai Liu 2018-04-16 16:02:38 UTC

Also verified on origin doc (latest)
https://docs.openshift.org/latest/install_config/persistent_storage/persistent_storage_aws.html

Comment 24 errata-xmlrpc 2018-04-30 04:00:18 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:1233