Bug 1738564 - MigStorage object not updated when AWS bucket is deleted from AWS
Summary: MigStorage object not updated when AWS bucket is deleted from AWS
Keywords:
Status: ASSIGNED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Migration Tooling
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.5.0
Assignee: Sam Lucidi
QA Contact: Sergio
URL:
Whiteboard:
Depends On:
Blocks: 1760994
TreeView+ depends on / blocked
 
Reported: 2019-08-07 13:17 UTC by Sergio
Modified: 2020-03-28 16:03 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1760994 (view as bug list)
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)

Description Sergio 2019-08-07 13:17:41 UTC
Description of problem:

When an AWS storage is configured in App Migration, and later on it is deleted in AWS, velero pod shows in the log that the AWS bucket does not exist, but the App Migration tool does not update the status of the MigStorage object.


Version-Release number of selected component (if applicable):


OCP4 version:

$ oc version
Client Version: version.Info{Major:"4", Minor:"1+", GitVersion:"v4.1.4", GitCommit:"c9e4f28ff", GitTreeState:"clean", BuildDate:"2019-06-26T23:04:27Z", GoVersion:"go1.12.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+838b4fa", GitCommit:"838b4fa", GitTreeState:"clean", BuildDate:"2019-05-19T23:51:04Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}


Controller image version used:

image: quay.io/ocpmigrate/mig-controller:latest
imageID: quay.io/ocpmigrate/mig-controller@sha256:89067b5fd1ee3800a121b32d089ddb71e0790184b2becfd3f7e5cc3fc91b3b52


Velero image version used:

image: quay.io/ocpmigrate/velero:fusor-dev
imageID: quay.io/ocpmigrate/velero@sha256:b707ae4f22ba1828ca6f9992b190134eaef145364cb57146d84616ccefdafbb7
image: quay.io/ocpmigrate/migration-plugin:latest
imageID: quay.io/ocpmigrate/migration-plugin@sha256:d34af290b3c6d808ad360a1f2d41d91e06bff5aa912f9a5a78fed3ea2f0f8f71



How reproducible:


Steps to Reproduce:
1. Add an AWS storage to the App Migration and check that it is connected and that velero pod is not complaining.

$ oc get MigStorage -o yaml -n mig
....
      message: The storage is ready.
      status: "True"
      type: Ready
....
$ oc logs $(oc get pods -n mig -l component=velero -o NAME) -n mig

There are no errors regarding AWS bucket.



2. Delete the storaget AWS bucket from AWS




Actual results:

1. Check that after a few seconds velero is failing 

$ oc logs $(oc get pods -n mig -l component=velero -o NAME) -n mig

time="2019-08-07T12:28:21Z" level=error msg="Error listing backups in backup store" backupLocation=XXXXXXXXXXXXX=backup-sync error="rpc error: code = Unknown desc = NoSuchBucket: The specified bucket does not exist\n\tstatus code: 404, request id: XXXXXXXX, host id: XXXXXXXXXXXXXXXXXX error.file="/go/src/github.com/heptio/velero/pkg/cloudprovider/aws/object_store.go:275" error.function="github.com/heptio/velero/pkg/cloudprovider/aws.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:174

2. But the MigStorage object is not updated

$ oc get MigStorage -o yaml
.....
      message: The storage is ready.
      status: "True"
      type: Ready

.....



Expected results:

1. Velero is failing
2. The MigSgorage object should be updated, showing the error like:

The Backup storage cloudprovider test failed [NoSuchBucket: The specified
       bucket does not exist\n\tstatus code: 404, request id: XXXXXXXXXX, host
       id: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX]."



Additional info:

Comment 1 John Matthews 2019-08-27 16:14:16 UTC
For 4.2 we do not intend to perform a check on external resources after we've established a ready condition, unless the details of the CR has changed.
i.e., if the CR is updated, we'll re-verify.

We will consider this for 4.3 release to add a periodic check on external resources.

Tracked in github issue:
https://github.com/fusor/mig-controller/issues/292

Comment 2 Eric Rich 2019-09-05 21:51:41 UTC
Should we create a docs bug for documenting this as a known issue as part of 4.2?

Comment 3 John Matthews 2019-10-11 21:26:36 UTC
Will add to 4.2 docs as a known issue:
https://bugzilla.redhat.com/show_bug.cgi?id=1760994


Note You need to log in before you can comment on or make changes to this bug.