Bug 1647125

Summary: Calls to AWS provider go over AWS 200 item filter limit
Product: Red Hat CloudForms Management Engine Reporter: Robb Manes <rmanes>
Component: ProvidersAssignee: Bronagh Sorota <bsorota>
Status: CLOSED DUPLICATE QA Contact: Matouš Mojžíš <mmojzis>
Severity: high Docs Contact:
Priority: high    
Version: 5.9.3CC: dmetzger, gblomqui, jfrey, jhardy, mmojzis, obarenbo, simaishi
Target Milestone: GA   
Target Release: 5.9.6   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: ec2
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-11-19 18:42:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: AWS Target Upstream Version:
Embargoed:

Description Robb Manes 2018-11-06 17:37:34 UTC 
Description of problem:
If attempting to get images from an AWS provider, and a call is made to a method that does filtering (as an example, `referenced_images`), AWS will return an error expressing it does not allow over 200 filtered items.

[----] E, [2018-10-31T04:36:12.585801 #24179:e33118] ERROR -- : MIQ(ManageIQ::Providers::Amazon::CloudManager::Refresher#refresh) EMS: [AWS-Product-Dev], id: [1000000000003] Refresh failed
[----] E, [2018-10-31T04:36:12.586062 #24179:e33118] ERROR -- : [Aws::EC2::Errors::FilterLimitExceeded]: The maximum number of filter values specified on a single call is 200  Method:[block in method_missing]
[----] E, [2018-10-31T04:36:12.586166 #24179:e33118] ERROR -- : /opt/rh/cfme-gemset/gems/aws-sdk-core-2.9.44/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.9.44/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.9.44/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.9.44/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.9.44/lib/aws-sdk-core/plugins/response_paging.rb:26:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.9.44/lib/seahorse/client/plugins/response_target.rb:21:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.9.44/lib/seahorse/client/request.rb:70:in `send_request'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.9.44/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'
/opt/rh/cfme-gemset/bundler/gems/cfme-providers-amazon-0c9d5e149707/app/models/manageiq/providers/amazon/inventory/collector/cloud_manager.rb:51:in `referenced_images'
/opt/rh/cfme-gemset/bundler/gems/cfme-providers-amazon-0c9d5e149707/app/models/manageiq/providers/amazon/inventory/parser/cloud_manager.rb:39:in `referenced_images'
/opt/rh/cfme-gemset/bundler/gems/cfme-providers-amazon-0c9d5e149707/app/models/manageiq/providers/amazon/inventory/parser/cloud_manager.rb:18:in `parse'
/var/www/miq/vmdb/app/models/manager_refresh/inventory.rb:38:in `block in parse'
/var/www/miq/vmdb/app/models/manager_refresh/inventory.rb:35:in `each'
/var/www/miq/vmdb/app/models/manager_refresh/inventory.rb:35:in `parse'

This is fixed upstream via this commit:
https://github.com/ManageIQ/manageiq-providers-amazon/commit/bee067db964cc488b301d6fa01b666f34c66a232

Version-Release number of selected component (if applicable):
End customer is using 5.9.3 however, this appears to not have been fixed in latest-z at this time.

How reproducible:
Every time.

Steps to Reproduce:
1. Connect to an AWS provider
2. Attempt to perform some query that would require >200 items in a filter
3. Observe error

Actual results:
AWS calls fail when large filtered results are in use

Expected results:
AWS calls should be broken up into multiple requests
Comment 2 Satoe Imaishi 2018-11-19 18:42:57 UTC 

*** This bug has been marked as a duplicate of bug 1612086 ***