Bug 1738564 - MigStorage object not updated when AWS bucket is deleted from AWS
Summary: MigStorage object not updated when AWS bucket is deleted from AWS
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Migration Toolkit for Containers
Classification: Red Hat
Component: General
Version: 1.3.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 1.4.4
Assignee: John Matthews
QA Contact: Xin jiang
Avital Pinnick
URL:
Whiteboard:
Depends On:
Blocks: 1760994
TreeView+ depends on / blocked
 
Reported: 2019-08-07 13:17 UTC by Sergio
Modified: 2021-05-11 15:34 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1760994 (view as bug list)
Environment:
Last Closed: 2021-05-11 15:34:45 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Sergio 2019-08-07 13:17:41 UTC
Description of problem:

When an AWS storage is configured in App Migration, and later on it is deleted in AWS, velero pod shows in the log that the AWS bucket does not exist, but the App Migration tool does not update the status of the MigStorage object.


Version-Release number of selected component (if applicable):


OCP4 version:

$ oc version
Client Version: version.Info{Major:"4", Minor:"1+", GitVersion:"v4.1.4", GitCommit:"c9e4f28ff", GitTreeState:"clean", BuildDate:"2019-06-26T23:04:27Z", GoVersion:"go1.12.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+838b4fa", GitCommit:"838b4fa", GitTreeState:"clean", BuildDate:"2019-05-19T23:51:04Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}


Controller image version used:

image: quay.io/ocpmigrate/mig-controller:latest
imageID: quay.io/ocpmigrate/mig-controller@sha256:89067b5fd1ee3800a121b32d089ddb71e0790184b2becfd3f7e5cc3fc91b3b52


Velero image version used:

image: quay.io/ocpmigrate/velero:fusor-dev
imageID: quay.io/ocpmigrate/velero@sha256:b707ae4f22ba1828ca6f9992b190134eaef145364cb57146d84616ccefdafbb7
image: quay.io/ocpmigrate/migration-plugin:latest
imageID: quay.io/ocpmigrate/migration-plugin@sha256:d34af290b3c6d808ad360a1f2d41d91e06bff5aa912f9a5a78fed3ea2f0f8f71



How reproducible:


Steps to Reproduce:
1. Add an AWS storage to the App Migration and check that it is connected and that velero pod is not complaining.

$ oc get MigStorage -o yaml -n mig
....
      message: The storage is ready.
      status: "True"
      type: Ready
....
$ oc logs $(oc get pods -n mig -l component=velero -o NAME) -n mig

There are no errors regarding AWS bucket.



2. Delete the storaget AWS bucket from AWS




Actual results:

1. Check that after a few seconds velero is failing 

$ oc logs $(oc get pods -n mig -l component=velero -o NAME) -n mig

time="2019-08-07T12:28:21Z" level=error msg="Error listing backups in backup store" backupLocation=XXXXXXXXXXXXX=backup-sync error="rpc error: code = Unknown desc = NoSuchBucket: The specified bucket does not exist\n\tstatus code: 404, request id: XXXXXXXX, host id: XXXXXXXXXXXXXXXXXX error.file="/go/src/github.com/heptio/velero/pkg/cloudprovider/aws/object_store.go:275" error.function="github.com/heptio/velero/pkg/cloudprovider/aws.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:174

2. But the MigStorage object is not updated

$ oc get MigStorage -o yaml
.....
      message: The storage is ready.
      status: "True"
      type: Ready

.....



Expected results:

1. Velero is failing
2. The MigSgorage object should be updated, showing the error like:

The Backup storage cloudprovider test failed [NoSuchBucket: The specified
       bucket does not exist\n\tstatus code: 404, request id: XXXXXXXXXX, host
       id: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX]."



Additional info:

Comment 1 John Matthews 2019-08-27 16:14:16 UTC
For 4.2 we do not intend to perform a check on external resources after we've established a ready condition, unless the details of the CR has changed.
i.e., if the CR is updated, we'll re-verify.

We will consider this for 4.3 release to add a periodic check on external resources.

Tracked in github issue:
https://github.com/fusor/mig-controller/issues/292

Comment 2 Eric Rich 2019-09-05 21:51:41 UTC
Should we create a docs bug for documenting this as a known issue as part of 4.2?

Comment 3 John Matthews 2019-10-11 21:26:36 UTC
Will add to 4.2 docs as a known issue:
https://bugzilla.redhat.com/show_bug.cgi?id=1760994

Comment 4 Erik Nelson 2021-04-07 20:48:03 UTC
Sergio, could you confirm that this remains a BZ as of MTC 1.4.2?

Comment 5 Sergio 2021-05-07 14:19:14 UTC
In 1.4.4 it is not happening any more. After deleting the bucket in AWS the migstorage is updated properly.

I think that we can close this issue.

Comment 6 Avital Pinnick 2021-05-09 08:32:00 UTC
(In reply to Sergio from comment #5)
> In 1.4.4 it is not happening any more. After deleting the bucket in AWS the
> migstorage is updated properly.
> 
> I think that we can close this issue.

The bug is included in current docs, Known Issues. I will create a PR to remove it.


Note You need to log in before you can comment on or make changes to this bug.