Hide Forgot
Description of problem: We need to add "MaxEBSVolumeCount" as a predicate so AWS EBS limitations is taken into account during scheduling. We also need to give a number to the environment variable "KUBE_MAX_PD_VOLS" to constraint the maximum device attached on each node. Version-Release number of selected component (if applicable): dev-preview-int How reproducible: Always Steps to Reproduce: 1. Check scheduler.json on master 2. Check /etc/sysconfig/atomic-openshift-master on master Actual results: 1. # cat /etc/origin/master/scheduler.json { "kind": "Policy", "apiVersion": "v1", "predicates": [ {"name": "MatchNodeSelector"}, {"name": "PodFitsResources"}, {"name": "PodFitsPorts"}, {"name": "NoDiskConflict"}, {"name": "Region", "argument": {"serviceAffinity" : {"labels" : ["region"]}}} ],"priorities": [ {"name": "LeastRequestedPriority", "weight": 1}, {"name": "ServiceSpreadingPriority", "weight": 1}, {"name": "Zone", "weight" : 2, "argument": {"serviceAntiAffinity" : {"label": "zone"}}} ] } 2. #cat /etc/sysconfig/atomic-openshift-master OPTIONS=--loglevel=2 CONFIG_FILE=/etc/origin/master/master-config.yaml # Proxy configuration # Origin uses standard HTTP_PROXY environment variables. Be sure to set # NO_PROXY for your master #NO_PROXY=master.example.com #HTTP_PROXY=http://USER:PASSWORD@IPADDR:PORT #HTTPS_PROXY=https://USER:PASSWORD@IPADDR:PORT Expected results: 1. We should add {"name": "MaxEBSVolumeCount"} in predicates: 2. We should add "KUBE_MAX_PD_VOLS=$number" in /etc/sysconfig/atomic-openshift-master.(number < 40) Additional info:
Working on it.
This is now fixed in INT
Tested on dev-preview-int (3.2.0.6) Confirmed with Ops，we have added “MaxEBSVolumeCount” on master: [root@dev-preview-int-master-21eb2 ~]# cat /etc/origin/master/scheduler.json { "kind": "Policy", "apiVersion": "v1", "predicates": [ {"name": "MatchNodeSelector"}, {"name": "PodFitsResources"}, {"name": "PodFitsPorts"}, {"name": "NoDiskConflict"}, {"name" : "MaxEBSVolumeCount"} ],"priorities": [ {"name": "LeastRequestedPriority", "weight": 1}, {"name": "ServiceSpreadingPriority", "weight": 1} ] } But we didn't get KUBE_MAX_PD_VOLS in the cluster. Here is an ideal place where it should be. [root@dev-preview-int-master-21eb2 ~]# cat /etc/sysconfig/atomic-openshift-master OPTIONS=--loglevel=2 CONFIG_FILE=/etc/origin/master/master-config.yaml AWS_ACCESS_KEY_ID=xxxx AWS_SECRET_ACCESS_KEY=xxxx # Proxy configuration # Origin uses standard HTTP_PROXY environment variables. Be sure to set # NO_PROXY for your master #NO_PROXY=master.example.com #HTTP_PROXY=http://USER:PASSWORD@IPADDR:PORT #HTTPS_PROXY=https://USER:PASSWORD@IPADDR:PORT So I wanna confirm that KUBE_MAX_PD_VOLS=$number is configured? Except for "/etc/sysconfig/atomic-openshift-master", where or how to set the number?
If that environment variable (configuration) is not provided, the default value of 39 EBS volumes is applied. Refer: https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithmprovider/defaults/defaults.go#L113-L120 and https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithmprovider/defaults/defaults.go#L39-L51
If we use DefaultMaxEBSVolumes instead of set a value, I can verify the bug. Thanks.