Bug 2064739 - Short Liveness and Readiness Probes can cause indirect image migration registry pods to crash loop
Summary: Short Liveness and Readiness Probes can cause indirect image migration regist...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Migration Toolkit for Containers
Classification: Red Hat
Component: Operator
Version: 1.6.3
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 1.7.0
Assignee: Jason Montleon
QA Contact: Xin jiang
Steve Goodman
URL:
Whiteboard:
Depends On: 2064424
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-03-16 13:04 UTC by Erik Nelson
Modified: 2022-03-24 06:32 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 2064424
Environment:
Last Closed: 2022-03-24 06:32:54 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github konveyor mig-controller pull 1269 0 None open Bug 2064739: Increase Liveness and Readiness probe timeouts 2022-03-16 13:39:09 UTC
Github konveyor mig-controller pull 1270 0 None open Bug 2064739: Increase Liveness and Readiness probe timeouts 2022-03-16 13:47:17 UTC
Red Hat Product Errata RHBA-2022:1043 0 None None None 2022-03-24 06:32:57 UTC

Description Erik Nelson 2022-03-16 13:04:32 UTC
+++ This bug was initially created as a clone of Bug #2064424 +++

Description of problem:
The liveness and readiness probe responses for registry pods take more than 3 seconds once a reasonable amount of data is transferred to the s3 bucket.

Version-Release number of selected component (if applicable):
MTC 1.6.3

How reproducible:
Reasonably easy to reproduce with sufficient image data in a migration

Steps to Reproduce:
1. Install OCS or ODF
2. Configure MTC to use a MCG bucket
3. Attempt to migrate a namespace with ~7-10 GB of images using the MCG bucket indirect image migration

Actual results:
Registry pods will start restarting once 5-6GB of images have been transferred to the bucket, eventually entering a Crash Loop and failing the migration

Expected results:
The Migration Completes successfully

Additional info:
Increasing the probes to 300 or 600 seconds seems to stop the problem from happening. The goal of these checks is to prevent a migration from stalling indefinitely, so a 5 or 10 minute wait is probably OK.

Comment 7 errata-xmlrpc 2022-03-24 06:32:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Migration Toolkit for Containers (MTC) 1.7.0 release advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1043


Note You need to log in before you can comment on or make changes to this bug.