Description of problem: Version-Release number of selected component (if applicable): How reproducible: Every time. Steps to Reproduce: 1. Set up an AWS M5 node 2. Attach more than 27 EBS volumes to pods Actual results: The attaches fail. Expected results: Success. Additional info: See: https://github.com/kubernetes/kubernetes/issues/59015 Fixed in Kube 1.11 by: https://github.com/kubernetes/kubernetes/pull/64154
It may be possible to fix this
Sorry my comment got submitted before I can finish typing. It is probably possible to fix this without needing volume limit feature in 3.9. It is not super clean I think but EC2 instance type is available as label in node Object and hence scheduler can potentially look at that label and deduce volume attach limit from it rather than going on with hardcoded values. I will try and open a PR for it.
upstream PR - https://github.com/kubernetes/kubernetes/pull/66397
According to PR https://github.com/openshift/ose/pull/1374/files , we have changed the limit to 25 for M5 instances, so I tested 25, and I can confirm that 25 works fine. And 26 FailedScheduling due to MaxVolumeCount. Needinfo to dev to confirm the limit change. [root@ip-172-18-4-83 ~]# df -h | grep nvme | wc -l 26 [root@ip-172-18-4-83 ~]# df -h | grep nvme | sort -V /dev/nvme0n1p2 497M 218M 280M 44% /boot /dev/nvme1n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-0657f7a1ba0d63640 /dev/nvme2n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-033a935b2ac656730 /dev/nvme3n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-012606dc3a83bd913 /dev/nvme4n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-02e4d9ee12622b075 /dev/nvme5n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-09c3506ec947f8788 /dev/nvme6n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-0e02b7ae56fb49337 /dev/nvme7n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-0a1bcd8c43460e212 /dev/nvme8n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-09709ce97880e78d2 /dev/nvme9n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-0d27c6384fbb7afb0 /dev/nvme10n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-09741317d1c1cc2bb /dev/nvme11n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-0f8d9af718eaf2b30 /dev/nvme12n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-016f535195898d248 /dev/nvme13n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-02c3d700e6a2ac862 /dev/nvme14n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-0f114c1df45f32917 /dev/nvme15n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-00e96ef30b40dd2d6 /dev/nvme16n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-06bed016701847a84 /dev/nvme17n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-0ab1eddec9497bad3 /dev/nvme18n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-0e9305521b4ae9212 /dev/nvme19n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-03375e45f7ccc0fd7 /dev/nvme20n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-0c376175244984fec /dev/nvme21n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-02322385b2b240726 /dev/nvme22n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-0c6ddcf4c2de105ba /dev/nvme23n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-082ff8b652bb0f73b /dev/nvme24n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-00ec82b61855db802 /dev/nvme25n1 976M 2.6M 907M 1% /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-024d5bffb3161fd3f [root@ip-172-18-4-83 ~]# oc describe pod ruby-ex-47-zs7w6 Name: ruby-ex-47-zs7w6 Namespace: default Node: ip-172-18-4-83.ec2.internal/172.18.4.83 Start Time: Mon, 27 Aug 2018 06:12:39 -0400 Labels: app=ruby-ex deployment=ruby-ex-47 deploymentconfig=ruby-ex Annotations: openshift.io/deployment-config.latest-version=47 openshift.io/deployment-config.name=ruby-ex openshift.io/deployment.name=ruby-ex-47 openshift.io/generated-by=OpenShiftNewApp openshift.io/scc=restricted Status: Running IP: 10.129.0.36 Controlled By: ReplicationController/ruby-ex-47 Containers: ruby-ex: Container ID: docker://878aee2b86bf4a0d0720a47eeebdc0319025d8501d2af05ce55f55ee639dc153 Image: docker-registry.default.svc:5000/default/ruby-ex@sha256:6b2ebcc4cd855747a1cf32d136f6a36dafdd4588c7c4dad46be1108e67e240ea Image ID: docker-pullable://docker-registry.default.svc:5000/default/ruby-ex@sha256:6b2ebcc4cd855747a1cf32d136f6a36dafdd4588c7c4dad46be1108e67e240ea Port: 8080/TCP State: Running Started: Mon, 27 Aug 2018 06:12:56 -0400 Ready: True Restart Count: 0 Environment: <none> Mounts: /mnt/p01 from v01 (rw) /mnt/p02 from v02 (rw) /mnt/p03 from v03 (rw) /mnt/p04 from v04 (rw) /mnt/p05 from v05 (rw) /mnt/p06 from v06 (rw) /mnt/p07 from v07 (rw) /mnt/p08 from v08 (rw) /mnt/p09 from v09 (rw) /mnt/p10 from v10 (rw) /mnt/p11 from v11 (rw) /mnt/p12 from v12 (rw) /mnt/p13 from v13 (rw) /mnt/p14 from v14 (rw) /mnt/p15 from v15 (rw) /mnt/p16 from v16 (rw) /mnt/p17 from v17 (rw) /mnt/p18 from v18 (rw) /mnt/p19 from v19 (rw) /mnt/p20 from v20 (rw) /mnt/p21 from v21 (rw) /mnt/p22 from v22 (rw) /mnt/p23 from v23 (rw) /mnt/p24 from v24 (rw) /mnt/p25 from v25 (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-64p2z (ro) Conditions: Type Status Initialized True Ready True PodScheduled True Volumes: v01: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc01 ReadOnly: false v02: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc02 ReadOnly: false v03: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc03 ReadOnly: false v04: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc04 ReadOnly: false v05: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc05 ReadOnly: false v06: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc06 ReadOnly: false v07: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc07 ReadOnly: false v08: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc08 ReadOnly: false v09: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc09 ReadOnly: false v10: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc10 ReadOnly: false v11: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc11 ReadOnly: false v12: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc12 ReadOnly: false v13: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc13 ReadOnly: false v14: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc14 ReadOnly: false v15: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc15 ReadOnly: false v16: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc16 ReadOnly: false v17: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc17 ReadOnly: false v18: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc18 ReadOnly: false v19: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc19 ReadOnly: false v20: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc20 ReadOnly: false v21: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc21 ReadOnly: false v22: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc22 ReadOnly: false v23: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc23 ReadOnly: false v24: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc24 ReadOnly: false v25: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc25 ReadOnly: false default-token-64p2z: Type: Secret (a volume populated by a Secret) SecretName: default-token-64p2z Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: <none> Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 8m default-scheduler Successfully assigned ruby-ex-47-zs7w6 to ip-172-18-4-83.ec2.internal Normal SuccessfulMountVolume 8m kubelet, ip-172-18-4-83.ec2.internal MountVolume.SetUp succeeded for volume "pvc-c345692b-a9d7-11e8-ada4-0e875169ed22" Normal SuccessfulMountVolume 8m kubelet, ip-172-18-4-83.ec2.internal MountVolume.SetUp succeeded for volume "pvc-c36ca531-a9d7-11e8-ada4-0e875169ed22" Normal SuccessfulMountVolume 8m kubelet, ip-172-18-4-83.ec2.internal MountVolume.SetUp succeeded for volume "pvc-c30370ed-a9d7-11e8-ada4-0e875169ed22" Normal SuccessfulMountVolume 8m kubelet, ip-172-18-4-83.ec2.internal MountVolume.SetUp succeeded for volume "pvc-c323331e-a9d7-11e8-ada4-0e875169ed22" Normal SuccessfulMountVolume 8m kubelet, ip-172-18-4-83.ec2.internal MountVolume.SetUp succeeded for volume "pvc-c390387c-a9d7-11e8-ada4-0e875169ed22" Normal SuccessfulMountVolume 8m kubelet, ip-172-18-4-83.ec2.internal MountVolume.SetUp succeeded for volume "pvc-c3b44c6f-a9d7-11e8-ada4-0e875169ed22" Normal SuccessfulMountVolume 8m kubelet, ip-172-18-4-83.ec2.internal MountVolume.SetUp succeeded for volume "pvc-c3d6dd26-a9d7-11e8-ada4-0e875169ed22" Normal SuccessfulMountVolume 8m kubelet, ip-172-18-4-83.ec2.internal MountVolume.SetUp succeeded for volume "pvc-c3f9c1bd-a9d7-11e8-ada4-0e875169ed22" Normal SuccessfulMountVolume 8m kubelet, ip-172-18-4-83.ec2.internal MountVolume.SetUp succeeded for volume "pvc-c4176be0-a9d7-11e8-ada4-0e875169ed22" Normal SuccessfulMountVolume 8m (x16 over 8m) kubelet, ip-172-18-4-83.ec2.internal (combined from similar events): MountVolume.SetUp succeeded for volume "default-token-64p2z" After adding the 26 volume, [root@ip-172-18-4-83 ~]# oc describe pod ruby-ex-48-5sm2b [root@ip-172-18-4-83 ~]# oc describe pod ruby-ex-48-5sm2b Name: ruby-ex-48-5sm2b Namespace: default Node: <none> Labels: app=ruby-ex deployment=ruby-ex-48 deploymentconfig=ruby-ex Annotations: openshift.io/deployment-config.latest-version=48 openshift.io/deployment-config.name=ruby-ex openshift.io/deployment.name=ruby-ex-48 openshift.io/generated-by=OpenShiftNewApp openshift.io/scc=restricted Status: Pending IP: Controlled By: ReplicationController/ruby-ex-48 Containers: ruby-ex: Image: docker-registry.default.svc:5000/default/ruby-ex@sha256:6b2ebcc4cd855747a1cf32d136f6a36dafdd4588c7c4dad46be1108e67e240ea Port: 8080/TCP Environment: <none> Mounts: /mnt/p01 from v01 (rw) /mnt/p02 from v02 (rw) /mnt/p03 from v03 (rw) /mnt/p04 from v04 (rw) /mnt/p05 from v05 (rw) /mnt/p06 from v06 (rw) /mnt/p07 from v07 (rw) /mnt/p08 from v08 (rw) /mnt/p09 from v09 (rw) /mnt/p10 from v10 (rw) /mnt/p11 from v11 (rw) /mnt/p12 from v12 (rw) /mnt/p13 from v13 (rw) /mnt/p14 from v14 (rw) /mnt/p15 from v15 (rw) /mnt/p16 from v16 (rw) /mnt/p17 from v17 (rw) /mnt/p18 from v18 (rw) /mnt/p19 from v19 (rw) /mnt/p20 from v20 (rw) /mnt/p21 from v21 (rw) /mnt/p22 from v22 (rw) /mnt/p23 from v23 (rw) /mnt/p24 from v24 (rw) /mnt/p25 from v25 (rw) /mnt/p26 from v26 (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-64p2z (ro) Conditions: Type Status PodScheduled False Volumes: v01: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc01 ReadOnly: false v02: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc02 ReadOnly: false v03: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc03 ReadOnly: false v04: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc04 ReadOnly: false v05: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc05 ReadOnly: false v06: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc06 ReadOnly: false v07: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc07 ReadOnly: false v08: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc08 ReadOnly: false v09: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc09 ReadOnly: false v10: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc10 ReadOnly: false v11: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc11 ReadOnly: false v12: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc12 ReadOnly: false v13: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc13 ReadOnly: false v14: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc14 ReadOnly: false v15: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc15 ReadOnly: false v16: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc16 ReadOnly: false v17: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc17 ReadOnly: false v18: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc18 ReadOnly: false v19: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc19 ReadOnly: false v20: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc20 ReadOnly: false v21: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc21 ReadOnly: false v22: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc22 ReadOnly: false v23: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc23 ReadOnly: false v24: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc24 ReadOnly: false v25: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc25 ReadOnly: false v26: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc26 ReadOnly: false default-token-64p2z: Type: Secret (a volume populated by a Secret) SecretName: default-token-64p2z Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: <none> Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 7s (x6 over 22s) default-scheduler 0/2 nodes are available: 2 MaxVolumeCount. [root@ip-172-18-4-83 ~]# oc get nodes --show-labels NAME STATUS ROLES AGE VERSION LABELS ip-172-18-12-16.ec2.internal Ready compute 1h v1.9.1+a0ce1bc657 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m5.large,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-east-1,failure-domain.beta.kubernetes.io/zone=us-east-1d,kubernetes.io/hostname=ip-172-18-12-16.ec2.internal,node-role.kubernetes.io/compute=true,registry=enabled,role=node,router=enabled ip-172-18-4-83.ec2.internal Ready master 1h v1.9.1+a0ce1bc657 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m5.large,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-east-1,failure-domain.beta.kubernetes.io/zone=us-east-1d,kubernetes.io/hostname=ip-172-18-4-83.ec2.internal,node-role.kubernetes.io/master=true,role=node
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2549
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days