Bug 1845678
| Summary: | [RFE] Add retry logic to image migration | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Dylan Murray <dymurray> | |
| Component: | Migration Tooling | Assignee: | Dylan Murray <dymurray> | |
| Status: | CLOSED ERRATA | QA Contact: | Xin jiang <xjiang> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 4.3.0 | CC: | chezhang, jmatthew, jmontleo, mberube, rjohnson, sregidor, whu | |
| Target Milestone: | --- | |||
| Target Release: | 4.4.z | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | No Doc Update | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1845695 (view as bug list) | Environment: | ||
| Last Closed: | 2020-06-17 00:04:13 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1845695 | |||
| Bug Blocks: | ||||
|
Description
Dylan Murray
2020-06-09 19:55:58 UTC
Verified CAM 1.2.2 stage
1. We deployed a django application using a imagestream.
2. During the migration we run on the background a script removing the registry pod in an endless loop, so that it were impossible to connect to the registry pod.
3. When the error was detected by velero and it was shown in its log, we stopped deleting the pod.
4. The retry logic worked as expected and the migration was done properly.
This is the velero log showing the retry logic, 3 tries until success:
time="2020-06-10T11:50:49Z" level=info msg="[is-restore] copying to: docker://image-registry.openshift-image-registry.svc:5000/ocp-24730-django
/django-psql-persistent:latest" cmd=/plugins/velero-plugins logSource="/go/src/github.com/konveyor/openshift-migration-plugin/velero-plugins/mi
gimagestream/restore.go:93" pluginName=velero-plugins restore=openshift-migration/0e90d2a0-ab10-11ea-b28f-a30be5677f8a-gs88k
time="2020-06-10T11:50:49Z" level=info msg="copying image: docker://172.30.220.75:5000/ocp-24730-django/django-psql-persistent@sha256:6458203d3
763a43b217cf2a90e33a85a8eaad6108060b808df59bd3a8c3d25bb; will attempt up to 5 times..." cmd=/plugins/velero-plugins logSource="/go/src/github.c
om/konveyor/openshift-migration-plugin/velero-plugins/migimagestream/shared.go:41" pluginName=velero-plugins restore=openshift-migration/0e90d2
a0-ab10-11ea-b28f-a30be5677f8a-gs88k
time="2020-06-10T11:51:22Z" level=info msg="attempt #0 failed, waiting 5s and then retrying" cmd=/plugins/velero-plugins logSource="/go/src/github.com/konveyor/openshift-migration-plugin/velero-plugins/migimagestream/shared.go:53" pluginName=velero-plugins restore=openshift-migration/0e90d2a0-ab10-11ea-b28f-a30be5677f8a-gs88k
time="2020-06-10T11:51:31Z" level=info msg="attempt #1 failed, waiting 10s and then retrying" cmd=/plugins/velero-plugins logSource="/go/src/github.com/konveyor/openshift-migration-plugin/velero-plugins/migimagestream/shared.go:53" pluginName=velero-plugins restore=openshift-migration/0e90d2a0-ab10-11ea-b28f-a30be5677f8a-gs88k
time="2020-06-10T11:52:41Z" level=info msg="attempt #2 failed, waiting 15s and then retrying" cmd=/plugins/velero-plugins logSource="/go/src/github.com/konveyor/openshift-migration-plugin/velero-plugins/migimagestream/shared.go:53" pluginName=velero-plugins restore=openshift-migration/0e90d2a0-ab10-11ea-b28f-a30be5677f8a-gs88k
time="2020-06-10T11:52:59Z" level=info msg="[is-restore] manifest of copied image: {\n \"schemaVersion\": 2,\n \"med..............
............
We have seen a problem when the connection cannot be ever established. After executing all the retries, if they are all unsuccessful, the migration does not fail. We move the BZ to assigned so that it can be fixed. Thank you!! Verified CAM 1.2.2 stage. 1. We deployed a django application using a imagestream. # oc new-project django # oc new-app --template django-psql-persistent 2.removing the registry pod on source cluster, so that it were impossible to connect to the registry pod. # oc scale --replicas=0 dc docker-registry -n default 3. When the error was detected by velero and I noticed the retry logic worked in velero log 4. After executing all the retries, the migration failed Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:2571 |