Bug 1848041

Summary: Migration sometimes fails in EnsureLabelsDeleted phase
Product: OpenShift Container Platform Reporter: John Matthews <jmatthew>
Component: Migration ToolingAssignee: Scott Seago <sseago>
Status: CLOSED ERRATA QA Contact: Xin jiang <xjiang>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.5CC: chezhang, dwhatley, ernelson, jortel, mberube, rjohnson, rpattath, sregidor, sseago, whu, xjiang
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1847993 Environment:
Last Closed: 2020-09-30 18:42:32 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1847993    

Description John Matthews 2020-06-17 14:48:17 UTC
+++ This bug was initially created as a clone of Bug #1847993 +++

Description of problem:
Sometimes the migration fails while EnsureLabelsDeleted phase because of a problem updating the replicationcontroller resource information.

If the migration is run again, the migration will end successfully.


Version-Release number of selected component (if applicable):
CAM 1.2.2 
SOURCE CLUSTER: OCP 3.11 AWS
TARGET CLUSTER: OCP 4.4 AWS
NOOBAA BUCKET

How reproducible:
Intermittent

Steps to Reproduce:
1. The problem happened while migrating a mysql DeploymentConfig. I attach the jinja2 template that we use to deploy this application, but it should happen with any DC when a race condition happens.

Actual results:
The migration fails in EnsureLabelsDeleted stage, and the MigMigration resource shows this failure

status:
  conditions:
  - category: Advisory
    durable: true
    lastTransitionTime: "2020-06-17T12:54:49Z"
    message: 'The migration has failed.  See: Errors.'
    reason: EnsureLabelsDeleted
    status: "True"
    type: Failed
  errors:
  - 'Operation cannot be fulfilled on replicationcontrollers "mysql-1": the object
    has been modified; please apply your changes to the latest version and try again'
  itenerary: Failed


If the migration is run again, the migration will end successfully.


Expected results:
The migration should end without problems.

Additional info:

Full MigMigration resource:

apiVersion: migration.openshift.io/v1alpha1
kind: MigMigration
metadata:
  annotations:
    openshift.io/touch: baced36f-b099-11ea-8cbd-0a580a820268
  creationTimestamp: "2020-06-17T12:50:26Z"
  generation: 30
  labels:
    controller-tools.k8s.io: "1.0"
  name: ocp-28967-migplan-naming-mig-1592398188
  namespace: openshift-migration
  ownerReferences:
  - apiVersion: migration.openshift.io/v1alpha1
    kind: MigPlan
    name: ocp-28967-migplan-naming.migplan.1592398188
    uid: 2aeb54f6-8389-4fd3-9fa8-468135f57f5b
  resourceVersion: "117389"
  selfLink: /apis/migration.openshift.io/v1alpha1/namespaces/openshift-migration/migmigrations/ocp-28967-migplan-naming-mig-1592398188
  uid: 1204a4ca-e660-4f5d-90ff-95418a197aea
spec:
  migPlanRef:
    name: ocp-28967-migplan-naming.migplan.1592398188
    namespace: openshift-migration
  stage: false
status:
  conditions:
  - category: Advisory
    durable: true
    lastTransitionTime: "2020-06-17T12:54:49Z"
    message: 'The migration has failed.  See: Errors.'
    reason: EnsureLabelsDeleted
    status: "True"
    type: Failed
  errors:
  - 'Operation cannot be fulfilled on replicationcontrollers "mysql-1": the object
    has been modified; please apply your changes to the latest version and try again'
  itenerary: Failed
  observedDigest: d67b950bb9516f04ae2a6bdf60b0c8e70aea4ef3e52bf63f7278cd1826fa075a
  phase: Completed
  startTimestamp: "2020-06-17T12:50:26Z"

--- Additional comment from Sergio on 2020-06-17 14:06:02 UTC ---



--- Additional comment from Sergio on 2020-06-17 14:06:33 UTC ---



--- Additional comment from Sergio on 2020-06-17 14:06:58 UTC ---



--- Additional comment from Sergio on 2020-06-17 14:07:19 UTC ---

Comment 4 Sergio 2020-09-17 10:19:31 UTC
Verified MTC 1.3 stage

The issue was not reproduced in any of the testcases executed for this version.

Comment 8 errata-xmlrpc 2020-09-30 18:42:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Migration Toolkit for Containers (MTC) Tool image release advisory 1.3.0), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4148