Bug 2001786

Summary: Migration fails during "Stage Backup" step because volume path on host not found
Product: Migration Toolkit for Containers Reporter: Prasad Joshi <prajoshi>
Component: ControllerAssignee: Pranav Gaikwad <pgaikwad>
Status: CLOSED ERRATA QA Contact: Xin jiang <xjiang>
Severity: urgent Docs Contact: Avital Pinnick <apinnick>
Priority: urgent    
Version: 1.6.0CC: ernelson, prajoshi, rjohnson, sregidor, ssingla, xjiang
Target Milestone: ---   
Target Release: 1.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-09-29 14:35:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Prasad Joshi 2021-09-07 07:53:30 UTC
Description of problem: When we execute migration it is failing at the StageBackup step. PodVolumeBackup is failed due to 'error getting volume path on host: expected one matching path, got 0'


Version-Release number of selected component (if applicable):
SOURCE CLUSTER: Azure OCP 4.6 (MTC 1.6.0)
TARGET CLUSTER: Azure OCP 4.9 (MTC 1.6.0)
REPLICATION REPOSITORY: Azure Blob Storage


How reproducible:
Always


Steps to Reproduce:
1. In source cluster create django application

$ oc new-project ocp-40299-cancelrollback
$ oc new-app django-psql-persistent

2. Go to the MTC UI and create a migration plan for this namespace.
3. Execute stage migration  
4. Execute Cutover migration



Actual results: Migration is completed with warning. Backup is partiallyFailed and PodVolumeBackup failed due to 'error getting volume path on host: expected one matching path, got 0'


Expected results: StageBackup step should be completed without any failure.


Additional info: 

oc get backup migration-614c4-stage-z2shw -n openshift-migration -o yaml

status:
  completionTimestamp: "2021-09-06T10:45:38Z"
  errors: 1
  expiration: "2021-10-06T10:45:23Z"
  formatVersion: 1.1.0
  phase: PartiallyFailed
  progress:
    itemsBackedUp: 6
    totalItems: 6
  startTimestamp: "2021-09-06T10:45:23Z"
  version: 1

oc get podvolumebackup migration-614c4-stage-z2shw-qstbx -n openshift-migration -o yaml

status:
  completionTimestamp: "2021-09-06T10:45:37Z"
  message: 'error getting volume path on host: expected one matching path, got 0'
  phase: Failed
  progress: {}
  startTimestamp: "2021-09-06T10:45:37Z"


oc get migmigration migration-614c4 -n openshift-migration -o yaml

status:
  conditions:
  - category: Warn
    durable: true
    lastTransitionTime: "2021-09-06T10:45:39Z"
    message: 'Stage Backup openshift-migration/migration-614c4-stage-z2shw: partially failed on source cluster'
    status: "True"
    type: VeleroStageBackupPartiallyFailed
  - category: Advisory
    durable: true
    lastTransitionTime: "2021-09-06T10:46:36Z"
    message: The migration has completed with warnings, please look at `Warn` conditions.
    reason: Completed
    status: "True"
    type: SucceededWithWarnings
  itinerary: Final
  observedDigest: b6379d5e70f2e9f71791cd922a3b0e6e305a78498919ce72207017d20e61278b
  phase: Completed
  pipeline:
  - completed: "2021-09-06T10:44:42Z"
    message: Completed
    name: Prepare
    started: "2021-09-06T10:44:15Z"
  - completed: "2021-09-06T10:45:05Z"
    message: Completed
    name: Backup
    progress:
    - 'Backup openshift-migration/migration-614c4-initial-ddsvp: 44 out of estimated total of 44 objects backed up (20s)'
    started: "2021-09-06T10:44:42Z"
  - completed: "2021-09-06T10:45:46Z"
    message: Completed
    name: StageBackup
    progress:
    - 'Backup openshift-migration/migration-614c4-stage-z2shw: partially failed. 6 out of estimated total of 6 objects backed up (15s)'
    - 'PodVolumeBackup openshift-migration/migration-614c4-stage-z2shw-qstbx: Failed. 0 bytes out of 0 bytes backed up (0s)'
    started: "2021-09-06T10:45:05Z"
  - completed: "2021-09-06T10:46:24Z"
    message: Completed
    name: StageRestore
    progress:
    - 'Restore openshift-migration/migration-614c4-stage-6h5dn: 4 out of estimated total of 4 objects restored (2s)'
    - stage-postgresql-fd4km is missing. Migration might fail
    started: "2021-09-06T10:45:46Z"

Comment 1 Prasad Joshi 2021-09-07 07:59:27 UTC
It failed in IDVM, IDIM

Comment 2 Xin jiang 2021-09-10 03:35:05 UTC
Indirect migration often failed due to this bug.

Comment 6 Xin jiang 2021-09-15 02:58:53 UTC
verified with mtc 1.6.0

registry.redhat.io/rhmtc/openshift-migration-controller-rhel8@sha256:3b5efa9c8197fe0313a2ab7eb184d135ba9749c9a4f0d15a6abb11c0d18b9194

Comment 8 errata-xmlrpc 2021-09-29 14:35:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Migration Toolkit for Containers (MTC) 1.6.0 security & bugfix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3694