Bug 1919023

Summary:	Rollback can get stuck waiting for rsync daemon to finish terminating
Product:	Migration Toolkit for Containers	Reporter:	Derek Whatley <dwhatley>
Component:	Controller	Assignee:	Derek Whatley <dwhatley>
Status:	CLOSED DUPLICATE	QA Contact:	Xin jiang <xjiang>
Severity:	medium	Docs Contact:	Avital Pinnick <apinnick>
Priority:	medium
Version:	1.4.0	CC:	chezhang, ernelson, sregidor, whu
Target Milestone:	---
Target Release:	1.5.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-06-16 19:37:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Derek Whatley 2021-01-21 22:48:59 UTC

Description of problem:
If DVM controller rsync daemon pods are not cleaned up properly, rollback will not delete them due to missing correlation labels, but it _will_ wait for the Pods to be terminated (which will never happen).

Version-Release number of selected component (if applicable):
1.4.0

How reproducible:
Saw this once.


Steps to Reproduce:
1. Kill DVM in middle of migration (in my case, was triggered by an out of date operator not providing the "RSYNC_PRIVILEGED" env var
2. Migration will fail with DVM errors
3. Subsequent rollback should stall


Expected results:
DVM pods have correlation labels attached and will get removed by rollback

Comment 1 Xin jiang 2021-01-25 02:14:09 UTC

I believe the Target Release should 1.4.z.

Comment 3 Sergio 2021-03-10 13:54:13 UTC

Just in case you need another way to reproduce the issue. I hit this issue and I could reproduce it forcing an OutOfcpu failure like this:


Steps to Reproduce:
1. Configure limits/requests for the rsync pod in the source cluster high enough to trigger an OutOfcpu error.

$ oc edit migrationcontroller -n openshift-migration
.....
  spec:
.....
    source_rsync_pod_cpu_limits: 10
    source_rsync_pod_cpu_requests: 9


2. Migrate any application using a PVC. Use Direct Volume Migration

3. The migration will report a warning saying that it could not migrate the PVCs

4. Rollback the migration

Comment 4 Derek Whatley 2021-06-16 19:37:17 UTC


*** This bug has been marked as a duplicate of bug 1920559 ***

Comment 5 Derek Whatley 2021-06-16 19:37:49 UTC

Fix attached to duplicate https://bugzilla.redhat.com/show_bug.cgi?id=1920559