Bug 2064739

Summary:	Short Liveness and Readiness Probes can cause indirect image migration registry pods to crash loop
Product:	Migration Toolkit for Containers	Reporter:	Erik Nelson <ernelson>
Component:	Operator	Assignee:	Jason Montleon <jmontleo>
Status:	CLOSED ERRATA	QA Contact:	Xin jiang <xjiang>
Severity:	urgent	Docs Contact:	Steve Goodman <sgoodman>
Priority:	urgent
Version:	1.6.3	CC:	ernelson, jmontleo, rjohnson, sgoodman, ssingla, xjiang
Target Milestone:	---
Target Release:	1.7.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	2064424	Environment:
Last Closed:	2022-03-24 06:32:54 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2064424
Bug Blocks:

Description Erik Nelson 2022-03-16 13:04:32 UTC

+++ This bug was initially created as a clone of Bug #2064424 +++

Description of problem:
The liveness and readiness probe responses for registry pods take more than 3 seconds once a reasonable amount of data is transferred to the s3 bucket.

Version-Release number of selected component (if applicable):
MTC 1.6.3

How reproducible:
Reasonably easy to reproduce with sufficient image data in a migration

Steps to Reproduce:
1. Install OCS or ODF
2. Configure MTC to use a MCG bucket
3. Attempt to migrate a namespace with ~7-10 GB of images using the MCG bucket indirect image migration

Actual results:
Registry pods will start restarting once 5-6GB of images have been transferred to the bucket, eventually entering a Crash Loop and failing the migration

Expected results:
The Migration Completes successfully

Additional info:
Increasing the probes to 300 or 600 seconds seems to stop the problem from happening. The goal of these checks is to prevent a migration from stalling indefinitely, so a 5 or 10 minute wait is probably OK.

Comment 7 errata-xmlrpc 2022-03-24 06:32:54 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Migration Toolkit for Containers (MTC) 1.7.0 release advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1043