Bug 1507599

Summary:

Service Unavailable Error while using ec2api service

Product:

Red Hat OpenStack

Reporter:

Archit Modi <amodi>

Component:

puppet-tripleo

Assignee:

Sven Anderson <svanders>

Status:

CLOSED ERRATA

QA Contact:

Archit Modi <amodi>

Severity:

high

Docs Contact:

Priority:

high

Version:

12.0 (Pike)

CC:

amodi, aschultz, berrange, dasmith, eglynn, jjoyce, jschluet, kchamart, lyarwood, sbauza, sferdjao, sgordon, slinaber, srevivo, svanders, tvignaud, vromanso

Target Milestone:

Keywords:

Triaged

Target Release:

12.0 (Pike)

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

puppet-tripleo-7.4.3-4.el7ost

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2017-12-13 22:18:18 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
Steps to configure ec2api service	none

Description Archit Modi 2017-10-30 16:51:32 UTC

Description of problem: Service Unavailable Error while using ec2api service


Version-Release number of selected component (if applicable):


How reproducible: Always


Steps to Reproduce:
1.
2.
3.

Actual results:
(overcloud) [stack@undercloud-0 ~]$ aws --debug --endpoint-url http://10.0.0.107:8788 ec2 describe-instances
2017-10-25 16:29:47,578 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.11.177 Python/2.7.5 Linux/3.10.0-693.5.2.el7.x86_64 botocore/1.7.35
2017-10-25 16:29:47,578 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['--debug', '--endpoint-url', 'http://10.0.0.107:8788', 'ec2', 'describe-instances']
2017-10-25 16:29:47,578 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x3133938>
2017-10-25 16:29:47,578 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-10-25 16:29:47,578 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x2bc5938>
2017-10-25 16:29:47,578 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-10-25 16:29:47,578 - MainThread - botocore.session - DEBUG - Loading variable credentials_file from defaults.
2017-10-25 16:29:47,578 - MainThread - botocore.session - DEBUG - Loading variable config_file from defaults.
2017-10-25 16:29:47,579 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-10-25 16:29:47,579 - MainThread - botocore.session - DEBUG - Loading variable metadata_service_timeout from defaults.
2017-10-25 16:29:47,579 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-10-25 16:29:47,579 - MainThread - botocore.session - DEBUG - Loading variable metadata_service_num_attempts from defaults.
2017-10-25 16:29:47,579 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-10-25 16:29:47,579 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-10-25 16:29:47,579 - MainThread - botocore.session - DEBUG - Loading variable api_versions from defaults.
2017-10-25 16:29:47,584 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/lib/python2.7/site-packages/botocore-1.7.35-py2.7.egg/botocore/data/ec2/2016-11-15/service-2.json
2017-10-25 16:29:47,674 - MainThread - botocore.hooks - DEBUG - Event service-data-loaded.ec2: calling handler <function register_retries_for_service at 0x2895e60>
2017-10-25 16:29:47,675 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: ec2
2017-10-25 16:29:47,682 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ec2: calling handler <functools.partial object at 0x3141e68>
2017-10-25 16:29:47,682 - MainThread - awscli.customizations.removals - DEBUG - Removing operation: import-instance
2017-10-25 16:29:47,682 - MainThread - awscli.customizations.removals - DEBUG - Removing operation: import-volume
2017-10-25 16:29:47,682 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ec2: calling handler <function add_waiters at 0x313ba28>
2017-10-25 16:29:47,684 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/lib/python2.7/site-packages/botocore-1.7.35-py2.7.egg/botocore/data/ec2/2016-11-15/waiters-2.json
2017-10-25 16:29:47,688 - MainThread - awscli.clidriver - DEBUG - OrderedDict([(u'filters', <awscli.arguments.ListArgument object at 0x3c7dd50>), (u'instance-ids', <awscli.arguments.ListArgument object at 0x3c7dfd0>), (u'dry-run', <awscli.arguments.BooleanArgument object at 0x3c7dd90>), (u'no-dry-run', <awscli.arguments.BooleanArgument object at 0x3d7e050>), (u'max-results', <awscli.arguments.CLIArgument object at 0x3d7e090>), (u'next-token', <awscli.arguments.CLIArgument object at 0x3c7dcd0>)])
2017-10-25 16:29:47,688 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.describe-instances: calling handler <function add_streaming_output_arg at 0x3133b90>
2017-10-25 16:29:47,688 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.describe-instances: calling handler <function _rename_arg at 0x31779b0>
2017-10-25 16:29:47,688 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.describe-instances: calling handler <function _rename_arg at 0x3181578>
2017-10-25 16:29:47,688 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.describe-instances: calling handler <functools.partial object at 0x3141f18>
2017-10-25 16:29:47,688 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.describe-instances: calling handler <function add_cli_input_json at 0x2bd2758>
2017-10-25 16:29:47,688 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.describe-instances: calling handler <function unify_paging_params at 0x30a2500>
2017-10-25 16:29:47,690 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/lib/python2.7/site-packages/botocore-1.7.35-py2.7.egg/botocore/data/ec2/2016-11-15/paginators-1.json
2017-10-25 16:29:47,691 - MainThread - awscli.customizations.paginate - DEBUG - Modifying paging parameters for operation: DescribeInstances
2017-10-25 16:29:47,691 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.describe-instances: calling handler <function add_generate_skeleton at 0x30518c0>
2017-10-25 16:29:47,691 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ec2.describe-instances: calling handler <bound method CliInputJSONArgument.override_required_args of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x3c5d050>>
2017-10-25 16:29:47,691 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ec2.describe-instances: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x3d7e690>>
2017-10-25 16:29:47,692 - MainThread - botocore.hooks - DEBUG - Event operation-args-parsed.ec2.describe-instances: calling handler <functools.partial object at 0x3c65890>
2017-10-25 16:29:47,692 - MainThread - botocore.hooks - DEBUG - Event operation-args-parsed.ec2.describe-instances: calling handler <functools.partial object at 0x3c65998>
2017-10-25 16:29:47,692 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-instances.filters: calling handler <function uri_param at 0x2b401b8>
2017-10-25 16:29:47,692 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-instances.instance-ids: calling handler <function uri_param at 0x2b401b8>
2017-10-25 16:29:47,692 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-instances.dry-run: calling handler <function uri_param at 0x2b401b8>
2017-10-25 16:29:47,692 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-instances.max-results: calling handler <function uri_param at 0x2b401b8>
2017-10-25 16:29:47,692 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-instances.next-token: calling handler <function uri_param at 0x2b401b8>
2017-10-25 16:29:47,693 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-instances.cli-input-json: calling handler <function uri_param at 0x2b401b8>
2017-10-25 16:29:47,693 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-instances.starting-token: calling handler <function uri_param at 0x2b401b8>
2017-10-25 16:29:47,693 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-instances.page-size: calling handler <function uri_param at 0x2b401b8>
2017-10-25 16:29:47,693 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-instances.max-items: calling handler <function uri_param at 0x2b401b8>
2017-10-25 16:29:47,693 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-instances.generate-cli-skeleton: calling handler <function uri_param at 0x2b401b8>
2017-10-25 16:29:47,693 - MainThread - botocore.hooks - DEBUG - Event calling-command.ec2.describe-instances: calling handler <bound method CliInputJSONArgument.add_to_call_parameters of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x3c5d050>>
2017-10-25 16:29:47,693 - MainThread - botocore.hooks - DEBUG - Event calling-command.ec2.describe-instances: calling handler <bound method GenerateCliSkeletonArgument.generate_json_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x3d7e690>>
2017-10-25 16:29:47,693 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-10-25 16:29:47,693 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-10-25 16:29:47,693 - MainThread - botocore.session - DEBUG - Loading variable region from config file with value 'regionOne'.
2017-10-25 16:29:47,693 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-10-25 16:29:47,693 - MainThread - botocore.session - DEBUG - Loading variable ca_bundle from defaults.
2017-10-25 16:29:47,693 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-10-25 16:29:47,694 - MainThread - botocore.session - DEBUG - Loading variable api_versions from defaults.
2017-10-25 16:29:47,694 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2017-10-25 16:29:47,694 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2017-10-25 16:29:47,694 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2017-10-25 16:29:47,694 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2017-10-25 16:29:47,694 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/lib/python2.7/site-packages/botocore-1.7.35-py2.7.egg/botocore/data/endpoints.json
2017-10-25 16:29:47,709 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-10-25 16:29:47,718 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.ec2: calling handler <function add_generate_presigned_url at 0x2834a28>
2017-10-25 16:29:47,718 - MainThread - botocore.client - DEBUG - Assuming an endpoint for ec2, regionOne: http://10.0.0.107:8788
2017-10-25 16:29:47,718 - MainThread - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None
2017-10-25 16:29:47,722 - MainThread - botocore.endpoint - DEBUG - Setting ec2 timeout as (60, 60)
2017-10-25 16:29:47,723 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: ec2
2017-10-25 16:29:47,724 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-10-25 16:29:47,724 - MainThread - botocore.session - DEBUG - Loading variable output from defaults.
2017-10-25 16:29:47,724 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.ec2.DescribeInstances: calling handler <bound method ParameterAlias.alias_parameter_in_call of <botocore.handlers.ParameterAlias object at 0x288ce50>>
2017-10-25 16:29:47,724 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.ec2.DescribeInstances: calling handler <function generate_idempotent_uuid at 0x28958c0>
2017-10-25 16:29:47,725 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=DescribeInstances) (verify_ssl=True) with params: {'body': {'Action': u'DescribeInstances', 'Version': u'2016-11-15'}, 'url': u'http://10.0.0.107:8788/', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'aws-cli/1.11.177 Python/2.7.5 Linux/3.10.0-693.5.2.el7.x86_64 botocore/1.7.35'}, 'context': {'auth_type': None, 'client_region': 'regionOne', 'has_streaming_input': False, 'client_config': <botocore.config.Config object at 0x3fe1650>}, 'query_string': '', 'url_path': '/', 'method': u'POST'}
2017-10-25 16:29:47,725 - MainThread - botocore.hooks - DEBUG - Event request-created.ec2.DescribeInstances: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x3fe1550>>
2017-10-25 16:29:47,725 - MainThread - botocore.hooks - DEBUG - Event choose-signer.ec2.DescribeInstances: calling handler <function set_operation_specific_signer at 0x28957d0>
2017-10-25 16:29:47,725 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2017-10-25 16:29:47,725 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:10.0.0.107:8788
x-amz-date:20171025T202947Z

content-type;host;x-amz-date
6171eb09865e32b0602af0f7957e26573a51f53caaedff02ff88883cb0275885
2017-10-25 16:29:47,725 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20171025T202947Z
20171025/regionOne/ec2/aws4_request
7db16a0a141a0c0417aa4d70b82a03ba04ccce1c9aeb224891365e3786e73668
2017-10-25 16:29:47,726 - MainThread - botocore.auth - DEBUG - Signature:
545b26a834a54301baf9543bab533401bdd6c9c6fbd702f8a617e3df0abc61e0
2017-10-25 16:29:47,731 - MainThread - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [POST]>
2017-10-25 16:29:47,731 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTP connection (1): 10.0.0.107
2017-10-25 16:29:47,733 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "POST / HTTP/1.1" 503 None
2017-10-25 16:29:47,733 - MainThread - botocore.parsers - DEBUG - Response headers: {'connection': 'close', 'content-type': 'text/html', 'cache-control': 'no-cache'}
2017-10-25 16:29:47,734 - MainThread - botocore.parsers - DEBUG - Response body:
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

2017-10-25 16:29:47,734 - MainThread - botocore.parsers - DEBUG - Received a non protocol specific error response from the service, unable to populate error code and message.
2017-10-25 16:29:47,734 - MainThread - botocore.hooks - DEBUG - Event needs-retry.ec2.DescribeInstances: calling handler <botocore.retryhandler.RetryHandler object at 0x3bd5f90>
2017-10-25 16:29:47,734 - MainThread - botocore.retryhandler - DEBUG - retry needed: retryable HTTP status code received: 503
2017-10-25 16:29:47,734 - MainThread - botocore.retryhandler - DEBUG - Retry needed, action of: 0.739124779422
2017-10-25 16:29:47,734 - MainThread - botocore.endpoint - DEBUG - Response received to retry, sleeping for 0.739124779422 seconds
2017-10-25 16:29:48,474 - MainThread - botocore.hooks - DEBUG - Event request-created.ec2.DescribeInstances: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x3fe1550>>
2017-10-25 16:29:48,474 - MainThread - botocore.hooks - DEBUG - Event choose-signer.ec2.DescribeInstances: calling handler <function set_operation_specific_signer at 0x28957d0>
2017-10-25 16:29:48,475 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2017-10-25 16:29:48,475 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:10.0.0.107:8788
x-amz-date:20171025T202948Z

content-type;host;x-amz-date
6171eb09865e32b0602af0f7957e26573a51f53caaedff02ff88883cb0275885
2017-10-25 16:29:48,475 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20171025T202948Z
20171025/regionOne/ec2/aws4_request
a4c588ddfd3ba5a06352b1c9384b5f8d9b7f719ce4497644bdaee894b76c8bf0
2017-10-25 16:29:48,475 - MainThread - botocore.auth - DEBUG - Signature:
ef31b173b1df01fd1b4eaf7cec6514588e49318d0a268bc6ab593f1497c543a5
2017-10-25 16:29:48,475 - MainThread - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [POST]>
2017-10-25 16:29:48,475 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Resetting dropped connection: 10.0.0.107
2017-10-25 16:29:48,476 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "POST / HTTP/1.1" 503 None
2017-10-25 16:29:48,477 - MainThread - botocore.parsers - DEBUG - Response headers: {'connection': 'close', 'content-type': 'text/html', 'cache-control': 'no-cache'}
2017-10-25 16:29:48,477 - MainThread - botocore.parsers - DEBUG - Response body:
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

2017-10-25 16:29:48,477 - MainThread - botocore.parsers - DEBUG - Received a non protocol specific error response from the service, unable to populate error code and message.
2017-10-25 16:29:48,477 - MainThread - botocore.hooks - DEBUG - Event needs-retry.ec2.DescribeInstances: calling handler <botocore.retryhandler.RetryHandler object at 0x3bd5f90>
2017-10-25 16:29:48,477 - MainThread - botocore.retryhandler - DEBUG - retry needed: retryable HTTP status code received: 503
2017-10-25 16:29:48,477 - MainThread - botocore.retryhandler - DEBUG - Retry needed, action of: 0.757518377161
2017-10-25 16:29:48,477 - MainThread - botocore.endpoint - DEBUG - Response received to retry, sleeping for 0.757518377161 seconds
2017-10-25 16:29:49,235 - MainThread - botocore.hooks - DEBUG - Event request-created.ec2.DescribeInstances: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x3fe1550>>
2017-10-25 16:29:49,235 - MainThread - botocore.hooks - DEBUG - Event choose-signer.ec2.DescribeInstances: calling handler <function set_operation_specific_signer at 0x28957d0>
2017-10-25 16:29:49,236 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2017-10-25 16:29:49,236 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:10.0.0.107:8788
x-amz-date:20171025T202949Z

content-type;host;x-amz-date
6171eb09865e32b0602af0f7957e26573a51f53caaedff02ff88883cb0275885
2017-10-25 16:29:49,236 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20171025T202949Z
20171025/regionOne/ec2/aws4_request
b52ab528386c3a3ec3ef189a05badef3565e153d1d7894dcb28be29d0d5f09e2
2017-10-25 16:29:49,236 - MainThread - botocore.auth - DEBUG - Signature:
ba8c02f7f677c6a80906d4380da925b1c97d77d094978bfac7123c65bcbd6c26
2017-10-25 16:29:49,236 - MainThread - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [POST]>
2017-10-25 16:29:49,236 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Resetting dropped connection: 10.0.0.107
2017-10-25 16:29:49,237 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "POST / HTTP/1.1" 503 None
2017-10-25 16:29:49,238 - MainThread - botocore.parsers - DEBUG - Response headers: {'connection': 'close', 'content-type': 'text/html', 'cache-control': 'no-cache'}
2017-10-25 16:29:49,238 - MainThread - botocore.parsers - DEBUG - Response body:
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

2017-10-25 16:29:49,238 - MainThread - botocore.parsers - DEBUG - Received a non protocol specific error response from the service, unable to populate error code and message.
2017-10-25 16:29:49,238 - MainThread - botocore.hooks - DEBUG - Event needs-retry.ec2.DescribeInstances: calling handler <botocore.retryhandler.RetryHandler object at 0x3bd5f90>
2017-10-25 16:29:49,238 - MainThread - botocore.retryhandler - DEBUG - retry needed: retryable HTTP status code received: 503
2017-10-25 16:29:49,238 - MainThread - botocore.retryhandler - DEBUG - Retry needed, action of: 3.46478836676
2017-10-25 16:29:49,238 - MainThread - botocore.endpoint - DEBUG - Response received to retry, sleeping for 3.46478836676 seconds
2017-10-25 16:29:52,707 - MainThread - botocore.hooks - DEBUG - Event request-created.ec2.DescribeInstances: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x3fe1550>>
2017-10-25 16:29:52,707 - MainThread - botocore.hooks - DEBUG - Event choose-signer.ec2.DescribeInstances: calling handler <function set_operation_specific_signer at 0x28957d0>
2017-10-25 16:29:52,708 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2017-10-25 16:29:52,708 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:10.0.0.107:8788
x-amz-date:20171025T202952Z

content-type;host;x-amz-date
6171eb09865e32b0602af0f7957e26573a51f53caaedff02ff88883cb0275885
2017-10-25 16:29:52,708 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20171025T202952Z
20171025/regionOne/ec2/aws4_request
29f7b0a6facd178c025b7bf556dc9709edeeb8004057624d32903592d3d548fc
2017-10-25 16:29:52,708 - MainThread - botocore.auth - DEBUG - Signature:
fec29e33afae1a56d652e88caadb4cab689a1486c5f5c20a0f9ee5cc01405208
2017-10-25 16:29:52,708 - MainThread - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [POST]>
2017-10-25 16:29:52,708 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Resetting dropped connection: 10.0.0.107
2017-10-25 16:29:52,709 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "POST / HTTP/1.1" 503 None
2017-10-25 16:29:52,710 - MainThread - botocore.parsers - DEBUG - Response headers: {'connection': 'close', 'content-type': 'text/html', 'cache-control': 'no-cache'}
2017-10-25 16:29:52,710 - MainThread - botocore.parsers - DEBUG - Response body:
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

2017-10-25 16:29:52,710 - MainThread - botocore.parsers - DEBUG - Received a non protocol specific error response from the service, unable to populate error code and message.
2017-10-25 16:29:52,710 - MainThread - botocore.hooks - DEBUG - Event needs-retry.ec2.DescribeInstances: calling handler <botocore.retryhandler.RetryHandler object at 0x3bd5f90>
2017-10-25 16:29:52,710 - MainThread - botocore.retryhandler - DEBUG - retry needed: retryable HTTP status code received: 503
2017-10-25 16:29:52,710 - MainThread - botocore.retryhandler - DEBUG - Retry needed, action of: 5.24492734334
2017-10-25 16:29:52,710 - MainThread - botocore.endpoint - DEBUG - Response received to retry, sleeping for 5.24492734334 seconds
2017-10-25 16:29:57,958 - MainThread - botocore.hooks - DEBUG - Event request-created.ec2.DescribeInstances: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x3fe1550>>
2017-10-25 16:29:57,959 - MainThread - botocore.hooks - DEBUG - Event choose-signer.ec2.DescribeInstances: calling handler <function set_operation_specific_signer at 0x28957d0>
2017-10-25 16:29:57,959 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2017-10-25 16:29:57,959 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:10.0.0.107:8788
x-amz-date:20171025T202957Z

content-type;host;x-amz-date
6171eb09865e32b0602af0f7957e26573a51f53caaedff02ff88883cb0275885
2017-10-25 16:29:57,959 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20171025T202957Z
20171025/regionOne/ec2/aws4_request
2536597d488159afbae635b796635ee3f05b2a76b0817993cf3467dc1d7f4920
2017-10-25 16:29:57,959 - MainThread - botocore.auth - DEBUG - Signature:
309e73472bf0198e1dffc7b1141f170bbb19209dc17f9a6ba0ba51c73675ab9f
2017-10-25 16:29:57,960 - MainThread - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [POST]>
2017-10-25 16:29:57,960 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Resetting dropped connection: 10.0.0.107
2017-10-25 16:29:57,961 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "POST / HTTP/1.1" 503 None
2017-10-25 16:29:57,961 - MainThread - botocore.parsers - DEBUG - Response headers: {'connection': 'close', 'content-type': 'text/html', 'cache-control': 'no-cache'}
2017-10-25 16:29:57,961 - MainThread - botocore.parsers - DEBUG - Response body:
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

2017-10-25 16:29:57,962 - MainThread - botocore.parsers - DEBUG - Received a non protocol specific error response from the service, unable to populate error code and message.
2017-10-25 16:29:57,962 - MainThread - botocore.hooks - DEBUG - Event needs-retry.ec2.DescribeInstances: calling handler <botocore.retryhandler.RetryHandler object at 0x3bd5f90>
2017-10-25 16:29:57,962 - MainThread - botocore.retryhandler - DEBUG - retry needed: retryable HTTP status code received: 503
2017-10-25 16:29:57,962 - MainThread - botocore.retryhandler - DEBUG - Reached the maximum number of retry attempts: 5
2017-10-25 16:29:57,962 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2017-10-25 16:29:57,968 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/awscli-1.11.177-py2.7.egg/awscli/clidriver.py", line 200, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/usr/lib/python2.7/site-packages/awscli-1.11.177-py2.7.egg/awscli/clidriver.py", line 338, in __call__
    return command_table[parsed_args.operation](remaining, parsed_globals)
  File "/usr/lib/python2.7/site-packages/awscli-1.11.177-py2.7.egg/awscli/clidriver.py", line 508, in __call__
    call_parameters, parsed_globals)
  File "/usr/lib/python2.7/site-packages/awscli-1.11.177-py2.7.egg/awscli/clidriver.py", line 628, in invoke
    self._display_response(operation_name, response, parsed_globals)
  File "/usr/lib/python2.7/site-packages/awscli-1.11.177-py2.7.egg/awscli/clidriver.py", line 648, in _display_response
    formatter(command_name, response)
  File "/usr/lib/python2.7/site-packages/awscli-1.11.177-py2.7.egg/awscli/formatter.py", line 69, in __call__
    response_data = response.build_full_result()
  File "/usr/lib/python2.7/site-packages/botocore-1.7.35-py2.7.egg/botocore/paginate.py", line 449, in build_full_result
    for response in self:
  File "/usr/lib/python2.7/site-packages/botocore-1.7.35-py2.7.egg/botocore/paginate.py", line 255, in __iter__
    response = self._make_request(current_kwargs)
  File "/usr/lib/python2.7/site-packages/botocore-1.7.35-py2.7.egg/botocore/paginate.py", line 332, in _make_request
    return self._method(**current_kwargs)
  File "/usr/lib/python2.7/site-packages/botocore-1.7.35-py2.7.egg/botocore/client.py", line 312, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python2.7/site-packages/botocore-1.7.35-py2.7.egg/botocore/client.py", line 605, in _make_api_call
    raise error_class(parsed_response, operation_name)
ClientError: An error occurred (503) when calling the DescribeInstances operation (reached max retries: 4): Service Unavailable
2017-10-25 16:29:57,969 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

An error occurred (503) when calling the DescribeInstances operation (reached max retries: 4): Service Unavailable
(overcloud) [stack@undercloud-0 ~]$

Expected results:


Additional info:

Comment 2 Archit Modi 2017-10-30 16:59:11 UTC

Created attachment 1345544 [details]
Steps to configure ec2api service

Comment 3 Sven Anderson 2017-11-03 14:36:56 UTC

Most probably this is not an EC2 API issue, but haproxy is running on the port instead but not configured properly. Can you check?

Comment 4 Sven Anderson 2017-11-03 16:54:15 UTC

The problem is, that haproxy does a healthcheck that expects a 200 OK response. EC2-API however returns 400, if not authenticated. I submitted a fix upstream.

Comment 11 errata-xmlrpc 2017-12-13 22:18:18 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462