Bug 2064424 - Short Liveness and Readiness Probes can cause indirect image migration registry pods to crash loop
Summary: Short Liveness and Readiness Probes can cause indirect image migration regist...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Migration Toolkit for Containers
Classification: Red Hat
Component: Controller
Version: 1.6.3
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 1.6.4
Assignee: Jason Montleon
QA Contact: mohamed
Richard Hoch
URL:
Whiteboard:
Depends On:
Blocks: 2064739
TreeView+ depends on / blocked
 
Reported: 2022-03-15 19:17 UTC by Jason Montleon
Modified: 2022-04-19 08:08 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2064739 (view as bug list)
Environment:
Last Closed: 2022-04-19 08:08:11 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github konveyor mig-controller pull 1269 0 None open Bug 2064739: Increase Liveness and Readiness probe timeouts 2022-03-16 13:49:10 UTC
Github konveyor mig-controller pull 1271 0 None open Bug 2064424: Increase Liveness and Readiness probe timeouts 2022-03-16 13:49:10 UTC
Red Hat Product Errata RHBA-2022:1393 0 None None None 2022-04-19 08:08:17 UTC

Description Jason Montleon 2022-03-15 19:17:42 UTC
Description of problem:
The liveness and readiness probe responses for registry pods take more than 3 seconds once a reasonable amount of data is transferred to the s3 bucket.

Version-Release number of selected component (if applicable):
MTC 1.6.3

How reproducible:
Reasonably easy to reproduce with sufficient image data in a migration

Steps to Reproduce:
1. Install OCS or ODF
2. Configure MTC to use a MCG bucket
3. Attempt to migrate a namespace with ~7-10 GB of images using the MCG bucket indirect image migration

Actual results:
Registry pods will start restarting once 5-6GB of images have been transferred to the bucket, eventually entering a Crash Loop and failing the migration

Expected results:
The Migration Completes successfully

Additional info:
Increasing the probes to 300 or 600 seconds seems to stop the problem from happening. The goal of these checks is to prevent a migration from stalling indefinitely, so a 5 or 10 minute wait is probably OK.

Comment 9 errata-xmlrpc 2022-04-19 08:08:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Migration Toolkit for Containers (MTC) 1.6.4 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1393


Note You need to log in before you can comment on or make changes to this bug.