Bug 1920455
| Summary: | Migration status is "Succeeded" but the application did not migrate to GCP target cluster | ||
|---|---|---|---|
| Product: | Migration Toolkit for Containers | Reporter: | whu |
| Component: | General | Assignee: | Dylan Murray <dymurray> |
| Status: | CLOSED ERRATA | QA Contact: | Xin jiang <xjiang> |
| Severity: | medium | Docs Contact: | Avital Pinnick <apinnick> |
| Priority: | high | ||
| Version: | 1.4.0 | CC: | chezhang, ernelson, rjohnson, sregidor, whu, xjiang |
| Target Milestone: | --- | ||
| Target Release: | 1.5.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-07-28 04:08:00 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
whu
2021-01-26 11:04:54 UTC
In indirect migration mode, this issue exists too.
Version-Release number of selected component (if applicable):
MTC 1.4.0
registry.stage.redhat.io/rhmtc/openshift-migration-rhel7-operator@sha256:79f524931e7188bfbfddf1e3d23f491b627d691ef7849a42432c7aec2d5f8a54
source cluster: GCP OCP 3.11
target cluster: GCP OCP 4.3
$ oc get migplan mig-plan-00000-ocp-00000-nginx -o yaml
apiVersion: migration.openshift.io/v1alpha1
kind: MigPlan
metadata:
......
name: mig-plan-00000-ocp-00000-nginx
namespace: openshift-migration
......
spec:
destMigClusterRef:
name: host
namespace: openshift-migration
indirectImageMigration: true
indirectVolumeMigration: true
migStorageRef:
name: automatic
namespace: openshift-migration
namespaces:
- ocp-00000-nginx
persistentVolumes:
- capacity: 1Gi
name: pvc-ef257b85-604c-11eb-880f-0ededc9b8185
pvc:
accessModes:
- ReadWriteOnce
hasReference: true
name: nginx-logs
namespace: ocp-00000-nginx
selection:
action: copy
copyMethod: filesystem
storageClass: gp2
storageClass: gp2
supported:
actions:
- copy
- move
copyMethods:
- filesystem
- snapshot
- capacity: 1Gi
name: pvc-ef2ba5e8-604c-11eb-880f-0ededc9b8185
pvc:
accessModes:
- ReadWriteOnce
hasReference: true
name: nginx-html
namespace: ocp-00000-nginx
selection:
action: copy
copyMethod: filesystem
storageClass: gp2
storageClass: gp2
supported:
actions:
- copy
- move
copyMethods:
- filesystem
- snapshot
srcMigClusterRef:
name: source-cluster
namespace: openshift-migration
status:
conditions:
......
- category: Required
lastTransitionTime: "2021-01-27T03:23:54Z"
message: The migration plan is ready.
status: "True"
type: Ready
$ oc get migmigration mig-migration-00000-ocp-00000-nginx -o yaml
apiVersion: migration.openshift.io/v1alpha1
kind: MigMigration
metadata:
.....
name: mig-migration-00000-ocp-00000-nginx
namespace: openshift-migration
ownerReferences:
- apiVersion: migration.openshift.io/v1alpha1
kind: MigPlan
name: mig-plan-00000-ocp-00000-nginx
.....
spec:
migPlanRef:
name: mig-plan-00000-ocp-00000-nginx
namespace: openshift-migration
quiescePods: true
stage: false
status:
conditions:
- category: Advisory
durable: true
lastTransitionTime: "2021-01-27T03:30:58Z"
message: The migration has completed successfully.
reason: Completed
status: "True"
type: Succeeded
itinerary: Final
observedDigest: 9ecf816fba1c0c4560be4973ce222c7b625305241d652b30dfe3f718c64fc5e4
phase: Completed
pipeline:
- completed: "2021-01-27T03:28:27Z"
message: Completed
name: Prepare
started: "2021-01-27T03:27:52Z"
- completed: "2021-01-27T03:28:44Z"
message: Completed
name: Backup
progress:
- 'Backup openshift-migration/mig-migration-00000-ocp-00000-nginx-t5zjc: 39 out
of estimated total of 39 objects backed up (6s)'
started: "2021-01-27T03:28:27Z"
- completed: "2021-01-27T03:29:52Z"
message: Completed
name: StageBackup
progress:
- 'Backup openshift-migration/mig-migration-00000-ocp-00000-nginx-n746j: 7 out
of estimated total of 7 objects backed up (11s)'
- 'PodVolumeBackup openshift-migration/mig-migration-00000-ocp-00000-nginx-n746j-99t77:
429 bytes out of 429 bytes backed up (4s)'
- 'PodVolumeBackup openshift-migration/mig-migration-00000-ocp-00000-nginx-n746j-m6xtg:
21 bytes out of 21 bytes backed up (3s)'
started: "2021-01-27T03:28:44Z"
- completed: "2021-01-27T03:30:36Z"
message: Completed
name: StageRestore
progress:
- 'Restore openshift-migration/mig-migration-00000-ocp-00000-nginx-84fwn: Completed'
- stage-nginx-deployment-557dd97bf8-pmj4l-mfjhj is missing. Migration might fail
started: "2021-01-27T03:29:52Z"
- message: Skipped
name: DirectImage
skipped: true
- message: Skipped
name: DirectVolume
skipped: true
- completed: "2021-01-27T03:30:55Z"
message: Completed
name: Restore
progress:
- 'Restore openshift-migration/mig-migration-00000-ocp-00000-nginx-n4w7t: Completed'
started: "2021-01-27T03:30:36Z"
- completed: "2021-01-27T03:30:58Z"
message: Completed
name: Cleanup
started: "2021-01-27T03:30:55Z"
startTimestamp: "2021-01-27T03:27:52Z"
After migration in target cluster
$ oc get all -n ocp-00000-nginx
No resources found in ocp-00000-nginx namespace.
$ oc get pvc -n ocp-00000-nginx
No resources found in ocp-00000-nginx namespace.
Filter error from log in target cluster
$ grep -i error ocp-00000-nginx-idvm-target.log
velero-85575cc6b5-bvk4m velero time="2021-01-27T03:30:47Z" level=warning msg="unable to restore additional item" additionalResource=persistentvolumeclaims additionalResourceName=nginx-logs additionalResourceNamespace=ocp-00000-nginx error="stat /tmp/764925421/resources/persistentvolumeclaims/namespaces/ocp-00000-nginx/nginx-logs.json: no such file or directory" logSource="pkg/restore/restore.go:1034" restore=openshift-migration/mig-migration-00000-ocp-00000-nginx-n4w7t
velero-85575cc6b5-bvk4m velero time="2021-01-27T03:30:47Z" level=warning msg="unable to restore additional item" additionalResource=persistentvolumeclaims additionalResourceName=nginx-html additionalResourceNamespace=ocp-00000-nginx error="stat /tmp/764925421/resources/persistentvolumeclaims/namespaces/ocp-00000-nginx/nginx-html.json: no such file or directory" logSource="pkg/restore/restore.go:1034" restore=openshift-migration/mig-migration-00000-ocp-00000-nginx-n4w7t
Also get "The migration has completed successfully." message from log
migration-controller-84799ff68c-mcl27 mtc {"level":"info","ts":1611718258.3558404,"logger":"migration|m4n64","msg":"CR","migration":"openshift-migration/mig-migration-00000-ocp-00000-nginx","conditions":{"conditions":[{"type":"Succeeded","status":"True","reason":"Completed","category":"Advisory","message":"The migration has completed successfully.","lastTransitionTime":"2021-01-27T03:30:58Z","durable":true}]}}
I can reproduce the GCP issue in a fresh MTC clusters GCP 3.11->4.3. The current behavior I observed: Deploy an application with PVC 2. Create a migplan with DIM+DVM 3, migration the application result: migration process completed successfully only imagestream was migrated to target cluster, others were NOT migrated to target cluster errors: 1. time="2021-01-28T10:24:59Z" level=error msg="Error getting a backup store" backupstoragelocation=automatic-495v4 controller=backupstoragelocation error="rpc error: code = Unknown desc = google: error getting credentials using GOOGLE_APPLICATION_CREDENTIALS environment variable: unexpected end of JSON input" error.file="/go/src/github.com/konveyor/openshift-migration-velero-plugin-for-gcp/velero-plugin-for-gcp/object_store.go:87" error.function="main.(*ObjectStore).Init" logSource="pkg/controller/backupstoragelocation_controller.go:87" 2. time="2021-01-28T10:29:59Z" level=warning msg="unable to restore additional item" additionalResource=persistentvolumeclaims additionalResourceName=mysql additionalResourceNamespace=ocp-cakephpaaaa error="stat /tmp/411562558/resources/persistentvolumeclaims/namespaces/ocp-cakephpaaaa/mysql.json: no such file or directory" logSource="pkg/restore/restore.go:1034" restore=openshift-migration/776b9090-6153-11eb-a0b1-458bf8b2b8de-dm6vh time="2021-01-28T10:29:59Z" level=info msg="Executing item action for pods" logSource="pkg/restore/restore.go:1002" restore=openshift-migration/776b9090-6153-11eb-a0b1-458bf8b2b8de-dm6vh We tried indirect migration on GCP 3.11->GCP 4.7, it worked fine. So I'm a little confused based on what I've read here and in slack. It sounds like sometimes indirect migration is successful, and sometimes direct migration is unsuccessful. The error itself is GCP specific, that the BSL data references a secret that does not have a valid google credentials string. See more information here: https://github.com/vmware-tanzu/velero/issues/1530. Please if you can reproduce this I would be curious about what data exists in a few resources: `oc get backupstoragelocation -n openshift-migration` `oc get migstorage -n openshift-migration` and `oc get secret -n openshift-migration cloud-credentials -o yaml` Please run these on both the source and destination clusters at the moment of failure that would be very helpful. Thank you. Pusing to 1.4.1 based on Dylan's investigation Friday -- this is an upstream bug that appears to be some kind of race condition with how the replication repo data gets updated and how it's ultimately propagated to the related secret. Dylan, you are right. this error is GCP specific. The data.cloud is empty string.
$ oc get backupstoragelocation -n openshift-migration -o yaml
apiVersion: v1
items:
- apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
creationTimestamp: "2021-02-01T06:07:48Z"
generateName: automatic-
generation: 1
labels:
app.kubernetes.io/part-of: openshift-migration
migplan: 8e31b300-20e5-4c02-a276-c94fb5b74b23
migrationcontroller: edd9fa5c-ec77-4a05-88d4-cf3afdcfb02e
name: automatic-bsr5m
namespace: openshift-migration
resourceVersion: "80131"
selfLink: /apis/velero.io/v1/namespaces/openshift-migration/backupstoragelocations/automatic-bsr5m
uid: 5aa7d1e8-5729-428e-82e3-3dc91e210dbb
spec:
objectStorage:
bucket: camreplication
prefix: velero
provider: gcp
status:
lastSyncedTime: "2021-02-01T06:35:37Z"
lastValidationTime: "2021-02-01T06:35:38Z"
phase: Available
kind: List
metadata:
resourceVersion: ""
selfLink: ""
$ oc get migstorage -n openshift-migration -o yaml
apiVersion: v1
items:
- apiVersion: migration.openshift.io/v1alpha1
kind: MigStorage
metadata:
annotations:
openshift.io/touch: dff6d7a9-6447-11eb-bfd3-0a580a810207
creationTimestamp: "2021-02-01T04:42:21Z"
generation: 2
labels:
controller-tools.k8s.io: "1.0"
name: automatic
namespace: openshift-migration
resourceVersion: "47506"
selfLink: /apis/migration.openshift.io/v1alpha1/namespaces/openshift-migration/migstorages/automatic
uid: 32f9f044-f650-4aab-9042-704ff9f930bf
spec:
backupStorageConfig:
credsSecretRef:
name: automatic-creds
namespace: openshift-migration
gcpBucket: camreplication
backupStorageProvider: gcp
volumeSnapshotConfig:
credsSecretRef:
name: automatic-creds
namespace: openshift-migration
volumeSnapshotProvider: gcp
status:
conditions:
- category: Required
lastTransitionTime: "2021-02-01T04:42:22Z"
message: The storage is ready.
status: "True"
type: Ready
observedDigest: 6f0b70974a47a9343ea8f67e3949b1f5f6eef1be762fc90d3b41f54db1785766
kind: List
metadata:
resourceVersion: ""
selfLink: ""
$ oc get secret -n openshift-migration cloud-credentials -o yaml
apiVersion: v1
data:
cloud: ""
kind: Secret
metadata:
creationTimestamp: "2021-02-01T04:40:02Z"
name: cloud-credentials
namespace: openshift-migration
ownerReferences:
- apiVersion: migration.openshift.io/v1alpha1
kind: MigrationController
name: migration-controller
uid: edd9fa5c-ec77-4a05-88d4-cf3afdcfb02e
resourceVersion: "46440"
selfLink: /api/v1/namespaces/openshift-migration/secrets/cloud-credentials
uid: 56e894a3-235b-4d81-be33-15639f8aa481
type: Opaque
We believe this issue to have been fixed upstream, and as of our Velero rebase for 1.5.0 this should have been picked up: https://github.com/vmware-tanzu/velero/issues/1530 Moving to MODIFIED for Rayford to place onto the errata and alert QE this is ready for verification with upcoming build. Verified using MTC 1.5.0
openshift-migration-rhel7-operator@sha256:00e77706ca22bcb557d13c16822180fc877e6ea1639a72fda8eb9f5488b039a2
- name: MIG_CONTROLLER_REPO
value: openshift-migration-controller-rhel8@sha256
- name: MIG_CONTROLLER_TAG
value: 7f657df15e9514df4ef42da3431f558a19b8d3233a2ef1222cd8e27793c93816
Verified with: GCP 4.3 -> GCP 4.7 with GCP replication repository.
We could migrate applications in GCP using direct and indirect migrations.
Moved to VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Migration Toolkit for Containers (MTC) image release advisory 1.5.0), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2021:2929 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days |