Bug 2160662 - Velero pod crashing leading to migrations being stuck during Backup Phase
Summary: Velero pod crashing leading to migrations being stuck during Backup Phase
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Migration Toolkit for Containers
Classification: Red Hat
Component: Velero
Version: 1.7.7
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 1.7.7
Assignee: Scott Seago
QA Contact: mohamed
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-01-13 08:43 UTC by ssingla
Modified: 2023-02-09 02:18 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-02-09 02:18:07 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift velero pull 236 0 None Merged updating json-iterator to 1.1.12 for golang 1.19 compatibility 2023-01-16 18:27:20 UTC

Description ssingla 2023-01-13 08:43:23 UTC
Description of problem:
When any migration is triggered from 3.11 to 4.x cluster, the migrations stuck at the Backup phase. It is being observed that the velero pod on 3.11 cluster is getting restarted and having errors.

Version-Release number of selected component (if applicable):
3.x -> 4.x

How reproducible:
Always


Steps to Reproduce:
1. Deploy django or any application on the source cluster
2. Create migplan from MTC UI and perform a migration.


Actual results:
Velero pod restarts and Migration stuck in UI


Expected results:
Migration should be successfully completed.

Additional info:

Controller logs:

openshift-migration migration-controller-77cdcf4674-56gj5 mtc {"level":"info","ts":1673532636.4577825,"logger":"migration","msg":"Initial Velero Backup is incomplete. Waiting.","migMigration":"ocp-24872-mssql-mig-1673526503","phase":"InitialBackupCreated","backup":"openshift-migration/ocp-24872-mssql-mig-1673526503-initial-mfc2b","backupPhase":"InProgress","backupProgress":"34/34","backupWarnings":0,"backupErrors":0}


Velero pod logs:

time="2023-01-12T14:14:01Z" level=error msg="Error getting backup metadata from backup store" backup=ocp-33393-bz1845569-mig-1663065481-initial-cssqf backupLocation=automatic-t6r8s controller=backup-sync error="rpc error: code = Unknown desc = storage: object doesn't exist" error.file="/remote-source/velero/app/pkg/persistence/object_store.go:289" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(*objectBackupStore).GetBackupMetadata" logSource="/remote-source/velero/app/pkg/controller/backup_sync_controller.go:216"

Comment 5 mohamed 2023-01-22 13:07:00 UTC
Verified using MTC 1.7.7 prestage

metadata_nvr: openshift-migration-operator-metadata-container-v1.7.7-7

Comment 6 mohamed 2023-01-23 09:13:52 UTC
Migrations are now failing when migrating from OCP 4.x -> OCP 4.x 

showing the same logs and error cause as explained above.

Comment 8 mohamed 2023-01-23 17:23:20 UTC
Moved Back to verified as the fix is included in the latest build

Comment 9 mohamed 2023-01-30 08:36:12 UTC
The fix is included in the latest build

Comment 13 errata-xmlrpc 2023-02-09 02:18:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Migration Toolkit for Containers (MTC) 1.7.7 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:0693


Note You need to log in before you can comment on or make changes to this bug.