Bug 1919023 - Rollback can get stuck waiting for rsync daemon to finish terminating
Summary: Rollback can get stuck waiting for rsync daemon to finish terminating
Keywords:
Status: CLOSED DUPLICATE of bug 1920559
Alias: None
Product: Migration Toolkit for Containers
Classification: Red Hat
Component: Controller
Version: 1.4.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 1.5.0
Assignee: Derek Whatley
QA Contact: Xin jiang
Avital Pinnick
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-01-21 22:48 UTC by Derek Whatley
Modified: 2021-06-16 19:37 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-06-16 19:37:17 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Derek Whatley 2021-01-21 22:48:59 UTC
Description of problem:
If DVM controller rsync daemon pods are not cleaned up properly, rollback will not delete them due to missing correlation labels, but it _will_ wait for the Pods to be terminated (which will never happen).

Version-Release number of selected component (if applicable):
1.4.0

How reproducible:
Saw this once.


Steps to Reproduce:
1. Kill DVM in middle of migration (in my case, was triggered by an out of date operator not providing the "RSYNC_PRIVILEGED" env var
2. Migration will fail with DVM errors
3. Subsequent rollback should stall


Expected results:
DVM pods have correlation labels attached and will get removed by rollback

Comment 1 Xin jiang 2021-01-25 02:14:09 UTC
I believe the Target Release should 1.4.z.

Comment 3 Sergio 2021-03-10 13:54:13 UTC
Just in case you need another way to reproduce the issue. I hit this issue and I could reproduce it forcing an OutOfcpu failure like this:


Steps to Reproduce:
1. Configure limits/requests for the rsync pod in the source cluster high enough to trigger an OutOfcpu error.

$ oc edit migrationcontroller -n openshift-migration
.....
  spec:
.....
    source_rsync_pod_cpu_limits: 10
    source_rsync_pod_cpu_requests: 9


2. Migrate any application using a PVC. Use Direct Volume Migration

3. The migration will report a warning saying that it could not migrate the PVCs

4. Rollback the migration

Comment 4 Derek Whatley 2021-06-16 19:37:17 UTC

*** This bug has been marked as a duplicate of bug 1920559 ***

Comment 5 Derek Whatley 2021-06-16 19:37:49 UTC
Fix attached to duplicate https://bugzilla.redhat.com/show_bug.cgi?id=1920559


Note You need to log in before you can comment on or make changes to this bug.