Bug 2160662
| Summary: | Velero pod crashing leading to migrations being stuck during Backup Phase | ||
|---|---|---|---|
| Product: | Migration Toolkit for Containers | Reporter: | ssingla |
| Component: | Velero | Assignee: | Scott Seago <sseago> |
| Status: | CLOSED ERRATA | QA Contact: | mohamed <midays> |
| Severity: | urgent | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 1.7.7 | CC: | midays, rjohnson |
| Target Milestone: | --- | Keywords: | Regression |
| Target Release: | 1.7.7 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-02-09 02:18:07 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Verified using MTC 1.7.7 prestage metadata_nvr: openshift-migration-operator-metadata-container-v1.7.7-7 Migrations are now failing when migrating from OCP 4.x -> OCP 4.x showing the same logs and error cause as explained above. Moved Back to verified as the fix is included in the latest build The fix is included in the latest build Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Migration Toolkit for Containers (MTC) 1.7.7 security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:0693 |
Description of problem: When any migration is triggered from 3.11 to 4.x cluster, the migrations stuck at the Backup phase. It is being observed that the velero pod on 3.11 cluster is getting restarted and having errors. Version-Release number of selected component (if applicable): 3.x -> 4.x How reproducible: Always Steps to Reproduce: 1. Deploy django or any application on the source cluster 2. Create migplan from MTC UI and perform a migration. Actual results: Velero pod restarts and Migration stuck in UI Expected results: Migration should be successfully completed. Additional info: Controller logs: openshift-migration migration-controller-77cdcf4674-56gj5 mtc {"level":"info","ts":1673532636.4577825,"logger":"migration","msg":"Initial Velero Backup is incomplete. Waiting.","migMigration":"ocp-24872-mssql-mig-1673526503","phase":"InitialBackupCreated","backup":"openshift-migration/ocp-24872-mssql-mig-1673526503-initial-mfc2b","backupPhase":"InProgress","backupProgress":"34/34","backupWarnings":0,"backupErrors":0} Velero pod logs: time="2023-01-12T14:14:01Z" level=error msg="Error getting backup metadata from backup store" backup=ocp-33393-bz1845569-mig-1663065481-initial-cssqf backupLocation=automatic-t6r8s controller=backup-sync error="rpc error: code = Unknown desc = storage: object doesn't exist" error.file="/remote-source/velero/app/pkg/persistence/object_store.go:289" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(*objectBackupStore).GetBackupMetadata" logSource="/remote-source/velero/app/pkg/controller/backup_sync_controller.go:216"