Bug 2049832

Summary:	ContainerCreateError when trying to launch large (>500) numbers of pods across nodes
Product:	OpenShift Container Platform	Reporter:	Erik M Jacobs <ejacobs>
Component:	Node	Assignee:	Peter Hunt <pehunt>
Node sub component:	CRI-O	QA Contact:	Sunil Choudhary <schoudha>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	medium
Priority:	high	CC:	aos-bugs, carangog, kkarampo, qili
Version:	4.9
Target Milestone:	---
Target Release:	4.11.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-08-10 10:46:42 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2050310

Description Erik M Jacobs 2022-02-02 18:57:39 UTC

Description of problem:
I am working on using the kubeflow mpi operator to run large MPI jobs within OCP. I have noticed that when trying to run a very large workload (512 pods with 1cpu) I get a large number of ContainerCreateErrors.

Version-Release number of selected component (if applicable):
4.9.17

How reproducible:
Fairly consistently

Steps to Reproduce:
1. Create 40+ node cluster
2. Install kubeflow mpi operator 0.3
3. Create an MPI job with 512 pods

Actual results:
Many ContainerCreateError that occasionally don't resolve.

Expected results:
All pods should launch quickly and without issue.


Additional info:
Some background on this work:
https://github.com/OpenShiftDemos/kubeflow-mpi-openfoam

Comment 1 Peter Hunt 2022-02-02 19:35:18 UTC

fixed in attached PR

Comment 18 Sunil Choudhary 2022-03-16 10:03:02 UTC

Marking it verified based on comment 6, 7 & 8.

Comment 20 errata-xmlrpc 2022-08-10 10:46:42 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069