Bug 1955221

Created attachment 1777267 [details]
debugger image showing the sdk not panicking when it should

Description of problem:

The velero-plugin-for-aws uses Version 2(V2) of S3 API to ListObjects and prefixes for finding out the backups that exist in the S3 bucket. It might be possible the S3 vendor used backup the data does not support V2 API. This problem surfaced in a customer environment specifically at this location https://github.com/konveyor/velero-plugin-for-aws/blob/369d0ef1f7c322974ef759b0d66ebbcf5877a857/velero-plugin-for-aws/object_store.go#L324 in the codebase. The following symptoms were observed to root cause the problem:

1. The s3cmd cli[1] returned a total of ~788 prefixes with the following command

[alpatel@alpatel ~]$ s3cmd -c ~/config.cfg ls s3://migration/velero/backups/ | wc -l
788

2. When we turned on the debug flag for velero, the prefix match was only around 555

$ oc logs -f velero-7d4784f984-pm4l4 | grep backupCount
time="2021-04-29T00:29:03Z" level=debug msg="Got backups from backup store" backupCount=555 backupLocation=velero-4-lqmkd controller=backup-sync logSource="pkg/controller/backup_sync_controll
er.go:179"

3. It was clear that the velero plugin was not seeing all the backups.

4. After looking closely at the debug output of s3cmd, we found that it uses `marker` mechanism for pagination, which is different from S3 V2 List* implementation used by the velero plugin.


DEBUG: format_uri(): /migration/?delimiter=%2F&marker=velero%2Fbackups%2F<last-bucket-name-in-this-page>fh%2F&prefix=velero%2Fbackups%2F
DEBUG: Sending request method_string='GET', uri='/migration/?delimiter=%2F&marker=velero%2Fbackups%2F<last-bucket-name-in-this-page>%2F&prefix=velero%2Fbackups%2F', headers={'
x-amz-date': 'Thu, 29 Apr 2021 17:15:02 +0000', 'Authorization': 'AWS XXXXXXX'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (https://URI_HIDDEN#2)

the marker field in the above URL, indicates V1 usage in s3cmd.

5. Since the plugin did not return valid results on V2, we used a dummy Go program to just query S3 api directly, it would found that the API misbehaved in the following way:

- it returned 555 items in the list even though the total prefixes were 788, this is technically not a misbehavior, the API guarantees states that it can return less than maxKeys(1000 here), with `isTruncated: true` and it did. 
- However when `isTruncated: True` the V2 api expects the server to respond with NextContinuousToken. This is used by the V2 Pagination logic to go to the next page. This is the misbehavior on the server-side, triggering a bug on the s3 sdk. The bug on the s3 sdk side is that even if `isTruncated: true` && `NextContinuationToken`is blank, it is not panicking, instead its going through hiding the server bug.
- Because the server did not respond with this token, the pagination library was making a wrong call that this is the last page instead of second last. 
- The remaining prefixes were ignored apart from the first 555

Additionally, it was observed that the pagination worked in an alphabetically sorted manner, i.e. if the backup prefix were to be `velero/backups/0000abc` then it would be returned in the first page, making the total count to 556, but if it were to be `velero/backups/zzzzabc` then it would remain 555, but the s3cmd library would return 789. So there was no magic number that was identified before which it is safe to expect the results on the first page.
  

Version-Release number of selected component (if applicable):
all versions of MTC(I think)

How reproducible:
When using a specific s3 vendor to configure object store.

Steps to Reproduce:
1. Use the s3 vendor used by the customer
2. Create 200 backups
3. Configure use a custom image velero-plugin-for-aws that sets the maxKeys: 100 which will force pagination and trigger this bug

Actual results:
You will find 100 backups


Expected results:
It needs to return 200 backups


Additional info:
I have attached a screenshot that shows the NextContinuousTokens is nil in the debugger and isTruncted: True when using the V2 API


1. https://s3tools.org/