Bug 2064424

Summary:	Short Liveness and Readiness Probes can cause indirect image migration registry pods to crash loop
Product:	Migration Toolkit for Containers	Reporter:	Jason Montleon <jmontleo>
Component:	Controller	Assignee:	Jason Montleon <jmontleo>
Status:	CLOSED ERRATA	QA Contact:	mohamed <midays>
Severity:	urgent	Docs Contact:	Richard Hoch <rhoch>
Priority:	urgent
Version:	1.6.3	CC:	ernelson, midays, rjohnson
Target Milestone:	---
Target Release:	1.6.4
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2064739 (view as bug list)		Environment:
Last Closed:	2022-04-19 08:08:11 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2064739

Description Jason Montleon 2022-03-15 19:17:42 UTC

Description of problem:
The liveness and readiness probe responses for registry pods take more than 3 seconds once a reasonable amount of data is transferred to the s3 bucket.

Version-Release number of selected component (if applicable):
MTC 1.6.3

How reproducible:
Reasonably easy to reproduce with sufficient image data in a migration

Steps to Reproduce:
1. Install OCS or ODF
2. Configure MTC to use a MCG bucket
3. Attempt to migrate a namespace with ~7-10 GB of images using the MCG bucket indirect image migration

Actual results:
Registry pods will start restarting once 5-6GB of images have been transferred to the bucket, eventually entering a Crash Loop and failing the migration

Expected results:
The Migration Completes successfully

Additional info:
Increasing the probes to 300 or 600 seconds seems to stop the problem from happening. The goal of these checks is to prevent a migration from stalling indefinitely, so a 5 or 10 minute wait is probably OK.

Comment 9 errata-xmlrpc 2022-04-19 08:08:11 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Migration Toolkit for Containers (MTC) 1.6.4 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1393