Bug 1760994

+++ This bug was initially created as a clone of Bug #1738564 +++

Description of problem:

When an AWS storage is configured in App Migration, and later on it is deleted in AWS, velero pod shows in the log that the AWS bucket does not exist, but the App Migration tool does not update the status of the MigStorage object.


Version-Release number of selected component (if applicable):


OCP4 version:

$ oc version
Client Version: version.Info{Major:"4", Minor:"1+", GitVersion:"v4.1.4", GitCommit:"c9e4f28ff", GitTreeState:"clean", BuildDate:"2019-06-26T23:04:27Z", GoVersion:"go1.12.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+838b4fa", GitCommit:"838b4fa", GitTreeState:"clean", BuildDate:"2019-05-19T23:51:04Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}


Controller image version used:

image: quay.io/ocpmigrate/mig-controller:latest
imageID: quay.io/ocpmigrate/mig-controller@sha256:89067b5fd1ee3800a121b32d089ddb71e0790184b2becfd3f7e5cc3fc91b3b52


Velero image version used:

image: quay.io/ocpmigrate/velero:fusor-dev
imageID: quay.io/ocpmigrate/velero@sha256:b707ae4f22ba1828ca6f9992b190134eaef145364cb57146d84616ccefdafbb7
image: quay.io/ocpmigrate/migration-plugin:latest
imageID: quay.io/ocpmigrate/migration-plugin@sha256:d34af290b3c6d808ad360a1f2d41d91e06bff5aa912f9a5a78fed3ea2f0f8f71



How reproducible:


Steps to Reproduce:
1. Add an AWS storage to the App Migration and check that it is connected and that velero pod is not complaining.

$ oc get MigStorage -o yaml -n mig
....
      message: The storage is ready.
      status: "True"
      type: Ready
....
$ oc logs $(oc get pods -n mig -l component=velero -o NAME) -n mig

There are no errors regarding AWS bucket.



2. Delete the storaget AWS bucket from AWS




Actual results:

1. Check that after a few seconds velero is failing 

$ oc logs $(oc get pods -n mig -l component=velero -o NAME) -n mig

time="2019-08-07T12:28:21Z" level=error msg="Error listing backups in backup store" backupLocation=XXXXXXXXXXXXX=backup-sync error="rpc error: code = Unknown desc = NoSuchBucket: The specified bucket does not exist\n\tstatus code: 404, request id: XXXXXXXX, host id: XXXXXXXXXXXXXXXXXX error.file="/go/src/github.com/heptio/velero/pkg/cloudprovider/aws/object_store.go:275" error.function="github.com/heptio/velero/pkg/cloudprovider/aws.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:174

2. But the MigStorage object is not updated

$ oc get MigStorage -o yaml
.....
      message: The storage is ready.
      status: "True"
      type: Ready

.....



Expected results:

1. Velero is failing
2. The MigSgorage object should be updated, showing the error like:

The Backup storage cloudprovider test failed [NoSuchBucket: The specified
       bucket does not exist\n\tstatus code: 404, request id: XXXXXXXXXX, host
       id: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX]."



Additional info:

--- Additional comment from John Matthews on 2019-08-27 16:14:16 UTC ---

For 4.2 we do not intend to perform a check on external resources after we've established a ready condition, unless the details of the CR has changed.
i.e., if the CR is updated, we'll re-verify.

We will consider this for 4.3 release to add a periodic check on external resources.

Tracked in github issue:
https://github.com/fusor/mig-controller/issues/292

--- Additional comment from Eric Rich on 2019-09-05 21:51:41 UTC ---

Should we create a docs bug for documenting this as a known issue as part of 4.2?