Bug 2049832 - ContainerCreateError when trying to launch large (>500) numbers of pods across nodes
Summary: ContainerCreateError when trying to launch large (>500) numbers of pods acros...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 4.9
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
: 4.11.0
Assignee: Peter Hunt
QA Contact: Sunil Choudhary
URL:
Whiteboard:
Depends On:
Blocks: 2050310
TreeView+ depends on / blocked
 
Reported: 2022-02-02 18:57 UTC by Erik M Jacobs
Modified: 2022-08-10 10:47 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-08-10 10:46:42 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github cri-o cri-o pull 5590 0 None open server: fix race with kubelet 2022-02-02 19:35:18 UTC
Red Hat Product Errata RHSA-2022:5069 0 None None None 2022-08-10 10:47:06 UTC

Description Erik M Jacobs 2022-02-02 18:57:39 UTC 
Description of problem:
I am working on using the kubeflow mpi operator to run large MPI jobs within OCP. I have noticed that when trying to run a very large workload (512 pods with 1cpu) I get a large number of ContainerCreateErrors.

Version-Release number of selected component (if applicable):
4.9.17

How reproducible:
Fairly consistently

Steps to Reproduce:
1. Create 40+ node cluster
2. Install kubeflow mpi operator 0.3
3. Create an MPI job with 512 pods

Actual results:
Many ContainerCreateError that occasionally don't resolve.

Expected results:
All pods should launch quickly and without issue.


Additional info:
Some background on this work:
https://github.com/OpenShiftDemos/kubeflow-mpi-openfoam
Comment 1 Peter Hunt 2022-02-02 19:35:18 UTC 
fixed in attached PR
Comment 18 Sunil Choudhary 2022-03-16 10:03:02 UTC 
Marking it verified based on comment 6, 7 & 8.
Comment 20 errata-xmlrpc 2022-08-10 10:46:42 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069
Note You need to log in before you can comment on or make changes to this bug.