Fedora Account System
Red Hat Associate
Red Hat Customer
Created attachment 1777267 [details] debugger image showing the sdk not panicking when it should Description of problem: The velero-plugin-for-aws uses Version 2(V2) of S3 API to ListObjects and prefixes for finding out the backups that exist in the S3 bucket. It might be possible the S3 vendor used backup the data does not support V2 API. This problem surfaced in a customer environment specifically at this location https://github.com/konveyor/velero-plugin-for-aws/blob/369d0ef1f7c322974ef759b0d66ebbcf5877a857/velero-plugin-for-aws/object_store.go#L324 in the codebase. The following symptoms were observed to root cause the problem: 1. The s3cmd cli[1] returned a total of ~788 prefixes with the following command [alpatel@alpatel ~]$ s3cmd -c ~/config.cfg ls s3://migration/velero/backups/ | wc -l 788 2. When we turned on the debug flag for velero, the prefix match was only around 555 $ oc logs -f velero-7d4784f984-pm4l4 | grep backupCount time="2021-04-29T00:29:03Z" level=debug msg="Got backups from backup store" backupCount=555 backupLocation=velero-4-lqmkd controller=backup-sync logSource="pkg/controller/backup_sync_controll er.go:179" 3. It was clear that the velero plugin was not seeing all the backups. 4. After looking closely at the debug output of s3cmd, we found that it uses `marker` mechanism for pagination, which is different from S3 V2 List* implementation used by the velero plugin. DEBUG: format_uri(): /migration/?delimiter=%2F&marker=velero%2Fbackups%2F<last-bucket-name-in-this-page>fh%2F&prefix=velero%2Fbackups%2F DEBUG: Sending request method_string='GET', uri='/migration/?delimiter=%2F&marker=velero%2Fbackups%2F<last-bucket-name-in-this-page>%2F&prefix=velero%2Fbackups%2F', headers={' x-amz-date': 'Thu, 29 Apr 2021 17:15:02 +0000', 'Authorization': 'AWS XXXXXXX'}, body=(0 bytes) DEBUG: ConnMan.put(): connection put back to pool (https://URI_HIDDEN#2) the marker field in the above URL, indicates V1 usage in s3cmd. 5. Since the plugin did not return valid results on V2, we used a dummy Go program to just query S3 api directly, it would found that the API misbehaved in the following way: - it returned 555 items in the list even though the total prefixes were 788, this is technically not a misbehavior, the API guarantees states that it can return less than maxKeys(1000 here), with `isTruncated: true` and it did. - However when `isTruncated: True` the V2 api expects the server to respond with NextContinuousToken. This is used by the V2 Pagination logic to go to the next page. This is the misbehavior on the server-side, triggering a bug on the s3 sdk. The bug on the s3 sdk side is that even if `isTruncated: true` && `NextContinuationToken`is blank, it is not panicking, instead its going through hiding the server bug. - Because the server did not respond with this token, the pagination library was making a wrong call that this is the last page instead of second last. - The remaining prefixes were ignored apart from the first 555 Additionally, it was observed that the pagination worked in an alphabetically sorted manner, i.e. if the backup prefix were to be `velero/backups/0000abc` then it would be returned in the first page, making the total count to 556, but if it were to be `velero/backups/zzzzabc` then it would remain 555, but the s3cmd library would return 789. So there was no magic number that was identified before which it is safe to expect the results on the first page. Version-Release number of selected component (if applicable): all versions of MTC(I think) How reproducible: When using a specific s3 vendor to configure object store. Steps to Reproduce: 1. Use the s3 vendor used by the customer 2. Create 200 backups 3. Configure use a custom image velero-plugin-for-aws that sets the maxKeys: 100 which will force pagination and trigger this bug Actual results: You will find 100 backups Expected results: It needs to return 200 backups Additional info: I have attached a screenshot that shows the NextContinuousTokens is nil in the debugger and isTruncted: True when using the V2 API 1. https://s3tools.org/
In order to prove out if the above is accurate we used this https://github.com/konveyor/velero-plugin-for-aws/pull/11 image to verify if migrations work of the same s3 bucket it was failing earlier. I can verify that the backupCount that velero saw with this older method in the pull request, worked and the migration was successful. Note: That PR is not intended to be merged, rather for education purposes only. We might need to a more generic solution than switching the function calls. Also note that noobaa client sdk has a configuration flag https://github.com/minio/minio-go/blob/bcf34777544b0209530bb4ae53c93e16b867335e/api-list.go#L631 for choice of List* api version, we could potentially expose something like that as a configuration flag to velcro that could work for all kinds of S3 implementations. It would be even better if this configuration value is at the BSL level. That would allow for maximum modularity in usage.
https://github.com/konveyor/velero-plugin-for-aws/pull/12
The code change didn't impact the existing function. we have done the regression testing and do not have swift storage to verify. Close it.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Migration Toolkit for Containers (MTC) image release advisory 1.5.0), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2021:2929