Bug 1295823 - "Errno::ETIMEDOUT: Connection timed out" when deploying RHELOSP
"Errno::ETIMEDOUT: Connection timed out" when deploying RHELOSP
Status: CLOSED NOTABUG
Product: Red Hat Quickstart Cloud Installer
Classification: Red Hat
Component: Installation - RHELOSP (Show other bugs)
1.0
Unspecified Unspecified
unspecified Severity high
: TP2
: 1.0
Assigned To: Jason Montleon
Dave Johnson
Dan Macpherson
: Triaged
Depends On:
Blocks: rhci-sprint-12
  Show dependency treegraph
 
Reported: 2016-01-05 09:53 EST by Antonin Pagac
Modified: 2016-04-29 12:20 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-01-15 12:43:16 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Antonin Pagac 2016-01-05 09:53:55 EST
Description of problem:
RHELOSP deployment failed at 86.5% with:

Errno::ETIMEDOUT: Connection timed out - connect(2)

It has previously successfully discovered undercloud, register two nodes and synced content, all without error.

Version-Release number of selected component (if applicable):
RHCI-6.0-RHEL-7-20151223.t.0-RHCI-x86_64-dvd1.iso
RHCIOOO-7-RHEL-7-20151208.t.0-RHCIOOO-x86_64-boot.iso

How reproducible:
Happened to me once

Steps to Reproduce:
1. Start a RHELOSP deployment
2. Wait for content to sync and then for installation to reach 86.5%
3. Error appears.

Actual results:
Connection timed out

Expected results:
No errors; installation continues to successful finish

Additional info:
Backtrace from UI:
/opt/rh/ruby193/root/usr/share/gems/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:70:in `initialize'
/opt/rh/ruby193/root/usr/share/gems/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:70:in `open'
/opt/rh/ruby193/root/usr/share/gems/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:70:in `block in initialize'
/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:55:in `timeout'
/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:100:in `timeout'
/opt/rh/ruby193/root/usr/share/gems/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:67:in `initialize'
/opt/rh/ruby193/root/usr/share/gems/gems/net-ssh-2.9.2/lib/net/ssh.rb:207:in `new'
/opt/rh/ruby193/root/usr/share/gems/gems/net-ssh-2.9.2/lib/net/ssh.rb:207:in `start'
/opt/rh/ruby193/root/usr/share/gems/gems/fusor_server-0.0.1/app/lib/actions/fusor/deployment/openstack/controller_cleanup.rb:67:in `block in fix_controller_services'
/opt/rh/ruby193/root/usr/share/gems/gems/fusor_server-0.0.1/app/lib/actions/fusor/deployment/openstack/controller_cleanup.rb:65:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/fusor_server-0.0.1/app/lib/actions/fusor/deployment/openstack/controller_cleanup.rb:65:in `fix_controller_services'
/opt/rh/ruby193/root/usr/share/gems/gems/fusor_server-0.0.1/app/lib/actions/fusor/deployment/openstack/controller_cleanup.rb:35:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:487:in `block (3 levels) in execute_run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/progress.rb:16:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.73/app/lib/actions/middleware/keep_locale.rb:23:in `block in run'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.73/app/lib/actions/middleware/keep_locale.rb:34:in `with_locale'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.73/app/lib/actions/middleware/keep_locale.rb:23:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:486:in `block (2 levels) in execute_run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:485:in `catch'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:485:in `block in execute_run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `block in with_error_handling'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `catch'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `with_error_handling'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:480:in `execute_run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:262:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/executors/parallel/worker.rb:20:in `block in on_message'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:859:in `block in assigns'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `tap'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `assigns'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:138:in `match_value'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:116:in `block in match'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `match'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/executors/parallel/worker.rb:17:in `on_message'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:82:in `on_envelope'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:72:in `receive'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `block (2 levels) in run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `loop'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `block in run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `catch'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:13:in `block in initialize'
/opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
Comment 1 Antonin Pagac 2016-01-06 08:28:11 EST
Reproduced.

Nodes on Director are provisioned. Task Label: Actions::Fusor::Deployment::OpenStack::Deploy fails on timeout. Output from the UI:

{"task"=>true, "poll_attempts"=>{"total"=>88, "failed"=>0}}

Director shows 6 of 7 steps completed in the overview, after I click on that it strangely shows everything checked except "Register Nodes". I can see all roles assigned and one of my nodes assigned to compute and the other to controller.
Comment 2 Antonin Pagac 2016-01-06 09:56:54 EST
From nova-compute.log:

2016-01-06 08:43:26.638 20224 DEBUG keystoneclient.session [req-af6806ad-864d-4e46-ac72-ef61d1a9a9aa - - - - -] REQ: curl -g -i -X GET http://192.0.2.1:9696/v2.0/floatingips.json?fixed_ip_address=192.0.2.8&port_id=a4ca7b58-68ab-4fa3-9830-46c89dfb691d -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}cfee068ae1b44bd7e6e6a7166b2f7c726952dc06" _http_log_request /usr/lib/python2.7/site-packages/keystoneclient/session.py:195
2016-01-06 08:43:26.642 20224 DEBUG keystoneclient.session [req-af6806ad-864d-4e46-ac72-ef61d1a9a9aa - - - - -] RESP: _http_log_response /usr/lib/python2.7/site-packages/keystoneclient/session.py:224
2016-01-06 08:43:26.642 20224 DEBUG neutronclient.v2_0.client [req-af6806ad-864d-4e46-ac72-ef61d1a9a9aa - - - - -] Error message: 404 Not Found

The resource could not be found.

    _handle_fault_response /usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py:176

2016-01-06 08:42:26.559 20224 DEBUG keystoneclient.session [req-af6806ad-864d-4e46-ac72-ef61d1a9a9aa - - - - -] REQ: curl -g -i -X GET http://192.0.2.1:9696/v2.0/floatingips.json?fixed_ip_address=192.0.2.7&port_id=2516a6eb-bb97-4c32-bbbd-672c76f38893 -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}cfee068ae1b44bd7e6e6a7166b2f7c726952dc06" _http_log_request /usr/lib/python2.7/site-packages/keystoneclient/session.py:195
2016-01-06 08:42:26.563 20224 DEBUG keystoneclient.session [req-af6806ad-864d-4e46-ac72-ef61d1a9a9aa - - - - -] RESP: _http_log_response /usr/lib/python2.7/site-packages/keystoneclient/session.py:224
2016-01-06 08:42:26.563 20224 DEBUG neutronclient.v2_0.client [req-af6806ad-864d-4e46-ac72-ef61d1a9a9aa - - - - -] Error message: 404 Not Found

The resource could not be found.

    _handle_fault_response /usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py:176


Found also a traceback in nova-api.log:
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack Traceback (most recent call last):
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/__init__.py", line 125, in __call__
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     return req.get_response(self.application)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     application, catch_exc_info=False)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     return resp(environ, start_response)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 634, in __call__
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     return self._call_app(env, start_response)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 554, in _call_app
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     return self._app(env, _fake_start_response)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     return resp(environ, start_response)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     return resp(environ, start_response)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     response = self.app(environ, start_response)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     return resp(environ, start_response)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 756, in __call__
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     content_type, body, accept)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 821, in _process_stack
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 911, in dispatch
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     return method(req=request, **action_args)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/floating_ips.py", line 103, in index
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     floating_ips = self.network_api.get_floating_ips_by_project(context)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1252, in get_floating_ips_by_project
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     fips = client.list_floatingips(tenant_id=project_id)['floatingips']
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in with_params
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     ret = self.function(instance, *args, **kwargs)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 691, in list_floatingips
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     **_params)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 307, in list
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     for r in self._pagination(collection, path, **params):
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 320, in _pagination
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     res = self.get(path, params=params)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 293, in get
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     headers=headers, params=params)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 270, in retry_request
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     headers=headers, params=params)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 211, in do_request
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     self._handle_fault_response(status_code, replybody)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     exception_handler_v20(status_code, des_error_body)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 83, in exception_handler_v20
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack     message=message)
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack NeutronClientException: 404 Not Found
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack
2016-01-06 08:53:57.025 20272 TRACE nova.api.openstack The resource could not be found.


From what I gather, the IPs are like so:

192.0.2.1: Director
192.0.2.7: Overcloud
192.0.2.8: probably the node

I can login to the Overcloud but it seems to be empty, there is nothing added after the installation.
Comment 5 Antonin Pagac 2016-01-08 08:48:17 EST
While trying to reproduce this, I hit similar timeout issue:

ERROR: deployment failed with status: CREATE_FAILED and reason: Create timed out


Action:

Actions::Fusor::Deployment::OpenStack::Deploy

Input:

{"deployment_id"=>1, "locale"=>"en"}

Output:

{"task"=>false, "poll_attempts"=>{"total"=>220, "failed"=>3}}

Exception:

RuntimeError: ERROR: deployment failed with status: CREATE_FAILED and reason: Create timed out

Backtrace:

/opt/rh/ruby193/root/usr/share/gems/gems/fusor_server-0.0.1/app/lib/actions/fusor/deployment/openstack/deploy.rb:92:in `poll_external_task'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/polling.rb:98:in `poll_external_task_with_rescue'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/polling.rb:21:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:487:in `block (3 levels) in execute_run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/progress.rb:16:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.73/app/lib/actions/middleware/keep_locale.rb:23:in `block in run'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.73/app/lib/actions/middleware/keep_locale.rb:34:in `with_locale'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.73/app/lib/actions/middleware/keep_locale.rb:23:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:486:in `block (2 levels) in execute_run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:485:in `catch'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:485:in `block in execute_run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `block in with_error_handling'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `catch'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `with_error_handling'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:480:in `execute_run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:262:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/executors/parallel/worker.rb:20:in `block in on_message'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:859:in `block in assigns'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `tap'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `assigns'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:138:in `match_value'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:116:in `block in match'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `match'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/executors/parallel/worker.rb:17:in `on_message'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:82:in `on_envelope'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:72:in `receive'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `block (2 levels) in run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `loop'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `block in run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `catch'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:13:in `block in initialize'
/opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'

From nova-api.log:

2016-01-08 08:34:25.276 20566 DEBUG neutronclient.v2_0.client [req-db0211af-a545-4356-8d72-661e013ff095 e4ccb487cd7f4450ae21f34732137bd1 3898b9158ca043eca697372f33b15fc2 - - -] Error message: 404 Not Found

The resource could not be found.

    _handle_fault_response /usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py:176
2016-01-08 08:34:25.276 20566 ERROR nova.api.openstack [req-db0211af-a545-4356-8d72-661e013ff095 e4ccb487cd7f4450ae21f34732137bd1 3898b9158ca043eca697372f33b15fc2 - - -] Caught error: 404 Not Found

The resource could not be found.
Comment 6 Jason Montleon 2016-01-14 17:01:17 EST
Looking at the logs earlier on the controller it looks like you assigned a non-existent interface for public access.
Comment 7 Jason Montleon 2016-01-15 12:43:16 EST
Confirmed this was an issue with the interface assigned for the controller nodes to use for the external network.

Note You need to log in before you can comment on or make changes to this bug.