Bug 1738564 - MigStorage object not updated when AWS bucket is deleted from AWS
Summary: MigStorage object not updated when AWS bucket is deleted from AWS
Alias: None
Product: Migration Toolkit for Containers
Classification: Red Hat
Component: General
Version: 1.3.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 1.5.0
Assignee: John Matthews
QA Contact: Xin jiang
Avital Pinnick
Depends On:
Blocks: 1760994
TreeView+ depends on / blocked
Reported: 2019-08-07 13:17 UTC by Sergio
Modified: 2020-10-28 15:50 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1760994 (view as bug list)
Last Closed:
Target Upstream Version:

Attachments (Terms of Use)

Description Sergio 2019-08-07 13:17:41 UTC
Description of problem:

When an AWS storage is configured in App Migration, and later on it is deleted in AWS, velero pod shows in the log that the AWS bucket does not exist, but the App Migration tool does not update the status of the MigStorage object.

Version-Release number of selected component (if applicable):

OCP4 version:

$ oc version
Client Version: version.Info{Major:"4", Minor:"1+", GitVersion:"v4.1.4", GitCommit:"c9e4f28ff", GitTreeState:"clean", BuildDate:"2019-06-26T23:04:27Z", GoVersion:"go1.12.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+838b4fa", GitCommit:"838b4fa", GitTreeState:"clean", BuildDate:"2019-05-19T23:51:04Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

Controller image version used:

image: quay.io/ocpmigrate/mig-controller:latest
imageID: quay.io/ocpmigrate/mig-controller@sha256:89067b5fd1ee3800a121b32d089ddb71e0790184b2becfd3f7e5cc3fc91b3b52

Velero image version used:

image: quay.io/ocpmigrate/velero:fusor-dev
imageID: quay.io/ocpmigrate/velero@sha256:b707ae4f22ba1828ca6f9992b190134eaef145364cb57146d84616ccefdafbb7
image: quay.io/ocpmigrate/migration-plugin:latest
imageID: quay.io/ocpmigrate/migration-plugin@sha256:d34af290b3c6d808ad360a1f2d41d91e06bff5aa912f9a5a78fed3ea2f0f8f71

How reproducible:

Steps to Reproduce:
1. Add an AWS storage to the App Migration and check that it is connected and that velero pod is not complaining.

$ oc get MigStorage -o yaml -n mig
      message: The storage is ready.
      status: "True"
      type: Ready
$ oc logs $(oc get pods -n mig -l component=velero -o NAME) -n mig

There are no errors regarding AWS bucket.

2. Delete the storaget AWS bucket from AWS

Actual results:

1. Check that after a few seconds velero is failing 

$ oc logs $(oc get pods -n mig -l component=velero -o NAME) -n mig

time="2019-08-07T12:28:21Z" level=error msg="Error listing backups in backup store" backupLocation=XXXXXXXXXXXXX=backup-sync error="rpc error: code = Unknown desc = NoSuchBucket: The specified bucket does not exist\n\tstatus code: 404, request id: XXXXXXXX, host id: XXXXXXXXXXXXXXXXXX error.file="/go/src/github.com/heptio/velero/pkg/cloudprovider/aws/object_store.go:275" error.function="github.com/heptio/velero/pkg/cloudprovider/aws.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:174

2. But the MigStorage object is not updated

$ oc get MigStorage -o yaml
      message: The storage is ready.
      status: "True"
      type: Ready


Expected results:

1. Velero is failing
2. The MigSgorage object should be updated, showing the error like:

The Backup storage cloudprovider test failed [NoSuchBucket: The specified
       bucket does not exist\n\tstatus code: 404, request id: XXXXXXXXXX, host

Additional info:

Comment 1 John Matthews 2019-08-27 16:14:16 UTC
For 4.2 we do not intend to perform a check on external resources after we've established a ready condition, unless the details of the CR has changed.
i.e., if the CR is updated, we'll re-verify.

We will consider this for 4.3 release to add a periodic check on external resources.

Tracked in github issue:

Comment 2 Eric Rich 2019-09-05 21:51:41 UTC
Should we create a docs bug for documenting this as a known issue as part of 4.2?

Comment 3 John Matthews 2019-10-11 21:26:36 UTC
Will add to 4.2 docs as a known issue:

Note You need to log in before you can comment on or make changes to this bug.