| Summary: | [RHEL-OSP] RuntimeError: CFME Launch failed | ||
|---|---|---|---|
| Product: | Red Hat Quickstart Cloud Installer | Reporter: | Thom Carlin <tcarlin> |
| Component: | Installation - CloudForms | Assignee: | Jason Montleon <jmontleo> |
| Status: | CLOSED ERRATA | QA Contact: | Thom Carlin <tcarlin> |
| Severity: | medium | Docs Contact: | Dan Macpherson <dmacpher> |
| Priority: | unspecified | ||
| Version: | 1.0 | CC: | bthurber, jmatthew, tpapaioa |
| Target Milestone: | ga | Keywords: | Triaged |
| Target Release: | 1.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-09-13 16:27:41 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Bug Depends On: | 1250736 | ||
| Bug Blocks: | |||
/var/log/nova/nova-compute.log on compute node
2016-03-31 07:36:52.655 17081 WARNING nova.compute.manager [req-f60ed881-cd0d-4fd0-9eb3-a4adc61d1130 - - - - -] Bandwidth usage not supported by hypervisor.
2016-03-31 07:38:41.487 17081 ERROR nova.compute.manager [req-2a589d30-e90c-4226-8142-263e76dc73cd b09d45574a7f438caf01b747dc498462 46b2a351f52a4ba691eb93af82e5ba82 - - -] [instance: 11f28531-be53-44f
e-89e0-7c57ad86e3aa] Instance failed to spawn
2016-03-31 07:38:41.487 17081 TRACE nova.compute.manager [instance: 11f28531-be53-44fe-89e0-7c57ad86e3aa] Traceback (most recent call last):
2016-03-31 07:38:41.487 17081 TRACE nova.compute.manager [instance: 11f28531-be53-44fe-89e0-7c57ad86e3aa] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2474, in _build_resour
ces
2016-03-31 07:38:41.487 17081 TRACE nova.compute.manager [instance: 11f28531-be53-44fe-89e0-7c57ad86e3aa] yield resources
2016-03-31 07:38:41.487 17081 TRACE nova.compute.manager [instance: 11f28531-be53-44fe-89e0-7c57ad86e3aa] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2346, in _build_and_ru
n_instance
2016-03-31 07:38:41.487 17081 TRACE nova.compute.manager [instance: 11f28531-be53-44fe-89e0-7c57ad86e3aa] block_device_info=block_device_info)
2016-03-31 07:38:41.487 17081 TRACE nova.compute.manager [instance: 11f28531-be53-44fe-89e0-7c57ad86e3aa] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2460, in spawn
2016-03-31 07:38:41.487 17081 TRACE nova.compute.manager [instance: 11f28531-be53-44fe-89e0-7c57ad86e3aa] write_to_disk=True)
2016-03-31 07:38:41.487 17081 TRACE nova.compute.manager [instance: 11f28531-be53-44fe-89e0-7c57ad86e3aa] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4322, in _get_gues
t_xml
2016-03-31 07:38:41.487 17081 TRACE nova.compute.manager [instance: 11f28531-be53-44fe-89e0-7c57ad86e3aa] context)
2016-03-31 07:38:41.487 17081 TRACE nova.compute.manager [instance: 11f28531-be53-44fe-89e0-7c57ad86e3aa] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4193, in _get_gues
t_config
2016-03-31 07:38:41.487 17081 TRACE nova.compute.manager [instance: 11f28531-be53-44fe-89e0-7c57ad86e3aa] flavor, virt_type)
2016-03-31 07:38:41.487 17081 TRACE nova.compute.manager [instance: 11f28531-be53-44fe-89e0-7c57ad86e3aa] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 374, in get_config
2016-03-31 07:38:41.487 17081 TRACE nova.compute.manager [instance: 11f28531-be53-44fe-89e0-7c57ad86e3aa] _("Unexpected vif_type=%s") % vif_type)
2016-03-31 07:38:41.487 17081 TRACE nova.compute.manager [instance: 11f28531-be53-44fe-89e0-7c57ad86e3aa] NovaException: Unexpected vif_type=binding_failed
2016-03-31 07:38:41.487 17081 TRACE nova.compute.manager [instance: 11f28531-be53-44fe-89e0-7c57ad86e3aa]
/var/log/neutron/openvswitch-agent.log
2016-03-31 07:33:57.363 17027 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2016-03-31 07:33:57.363 17027 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 281, in _
report_state
2016-03-31 07:33:57.363 17027 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self.use_call)
2016-03-31 07:33:57.363 17027 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 80, in report_state
2016-03-31 07:33:57.363 17027 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent return method(context, 'report_state', **kwargs)
2016-03-31 07:33:57.363 17027 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
2016-03-31 07:33:57.363 17027 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent retry=self.retry)
2016-03-31 07:33:57.363 17027 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2016-03-31 07:33:57.363 17027 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent timeout=timeout, retry=retry)
2016-03-31 07:33:57.363 17027 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
2016-03-31 07:33:57.363 17027 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent retry=retry)
2016-03-31 07:33:57.363 17027 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 339, in _send
2016-03-31 07:33:57.363 17027 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent result = self._waiter.wait(msg_id, timeout)
2016-03-31 07:33:57.363 17027 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 243, in wait
2016-03-31 07:33:57.363 17027 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent message = self.waiters.get(msg_id, timeout=timeout)
2016-03-31 07:33:57.363 17027 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 149, in get
2016-03-31 07:33:57.363 17027 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent 'to message ID %s' % msg_id)
2016-03-31 07:33:57.363 17027 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID cc82b6695c0e474f9a856fe58120ad25
Similar errors on Controller node
Also in /var/log/neutron/server.log
2016-03-31 08:36:03.079 2534 WARNING neutron.plugins.ml2.drivers.type_tunnel [req-17b909e2-f70f-4eff-82e0-e81ad9e7c070 ] Endpoint with ip 192.0.2.14 already exists
2016-03-31 08:37:30.215 2531 ERROR neutron.plugins.ml2.managers [req-c09d9133-7e18-495d-80b8-efae61ee607d ] Failed to bind port 313d550f-eecf-4f7d-944e-08306be9f444 on host overcloud-compute-0.localdo
main
2016-03-31 08:37:30.215 2531 ERROR neutron.plugins.ml2.managers [req-c09d9133-7e18-495d-80b8-efae61ee607d ] Failed to bind port 313d550f-eecf-4f7d-944e-08306be9f444 on host overcloud-compute-0.localdo
main
2016-03-31 09:37:44.004 2533 WARNING keystonemiddleware.auth_token [req-623f39c6-4364-4d61-b272-4ac1a96ecd9a ] Authorization failed for token
2016-03-31 09:38:45.018 2530 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2016-03-31 09:38:45.018 2530 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "Could not find token: 0167665db8bf4c52bc749aa7dfaf4e40", "code": 404, "title": "Not Fou
nd"}}
Was the host created in OSP? Was the corresponding record created in Satellite? The nova instance failed to spawn as detailed in https://bugzilla.redhat.com/show_bug.cgi?id=1322824#c1 The environment has been destroyed. I'm in the process of re-deploying now and will update/contact. Error reoccurred. This time:
The Host was created in OpenStack but the corresponding record was not created in Satellite 6.
/var/log/foreman/deployments/<deployment_name-number>/var/log/foreman
2016-04-03 17:35:02 [I] Waiting for openstack-cfme-cfme.<domain> to become ready
2016-04-03 17:45:04 [W] Failed to get IP for openstack-cfme-cfme.<domain>: The specified wait_for timeout (600 seconds) was exceeded/opt/rh/ruby193/root/usr/share/gems/gems/fog-cor
e-1.24.0/lib/fog/core/wait_for.rb:11:in `wait_for'
/opt/rh/ruby193/root/usr/share/gems/gems/fog-core-1.24.0/lib/fog/core/model.rb:65:in `wait_for'
nova list --all-tenants
STATUS = ERROR
Power State = NOSTATE
Networks = <deployment_name>-net=192.168.254.3, 192.168.253.2
nova show <instance_uuid>
{"message": "No valid host was found. There are not enough hosts available.", "code": 500, "details": " File \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 671, in build_instances |
| | request_spec, filter_properties) |
| | File \"/usr/lib/python2.7/site-packages/nova/scheduler/utils.py\", line 337, in wrapped |
| | return func(*args, **kwargs) |
| | File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\", line 52, in select_destinations |
| | context, request_spec, filter_properties) |
| | File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\", line 37, in __run_method |
| | return getattr(self.instance, __name)(*args, **kwargs) |
| | File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py\", line 34, in select_destinations |
| | context, request_spec, filter_properties) |
| | File \"/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py\", line 120, in select_destinations |
| | request_spec=request_spec, filter_properties=filter_properties) |
| | File \"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py\", line 156, in call | {"message": "No valid host was found. There are not enough hosts available.", "code": 500, "details": " File \"/usr/lib/python2.7/site- {"message": "No valid host was found. There are not enough hosts available.", "code": 500, "details": " File \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 671, in build_instances |
| | request_spec, filter_properties) |
| | File \"/usr/lib/python2.7/site-packages/nova/scheduler/utils.py\", line 337, in wrapped |
| | return func(*args, **kwargs) |
| | File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\", line 52, in select_destinations |
| | context, request_spec, filter_properties) |
| | File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\", line 37, in __run_method |
| | return getattr(self.instance, __name)(*args, **kwargs) |
| | File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py\", line 34, in select_destinations |
| | context, request_spec, filter_properties) |
| | File \"/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py\", line 120, in select_destinations |
| | request_spec=request_spec, filter_properties=filter_properties) |
| | File \"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py\", line 156, in call |
| | retry=self.retry) |
| | File \"/usr/lib/python2.7/site-packages/oslo_messaging/transport.py\", line 90, in _send |
| | timeout=timeout, retry=retry) |
| | File \"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 350, in send |
| | retry=retry) |
| | File \"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 341, in _send |
| | raise result |
| | ", "created": "2016-04-03T22:36:17Z"} packages/nova/conductor/manager.py\", line 671, in build_instances |
| | request_spec, filter_properties) |
| | File \"/usr/lib/python2.7/site-packages/nova/scheduler/utils.py\", line 337, in wrapped |
| | return func(*args, **kwargs) |
| | File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\", line 52, in select_destinations |
| | context, request_spec, filter_properties) |
| | File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\", line 37, in __run_method |
| | return getattr(self.instance, __name)(*args, **kwargs) |
| | File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py\", line 34, in select_destinations |
| | context, request_spec, filter_properties) |
| | File \"/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py\", line 120, in select_destinations |
| | request_spec=request_spec, filter_properties=filter_properties) |
| | File \"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py\", line 156, in call |
| | retry=self.retry) |
| | File \"/usr/lib/python2.7/site-packages/oslo_messaging/transport.py\", line 90, in _send |
| | timeout=timeout, retry=retry) |
| | File \"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 350, in send |
| | retry=retry) |
| | File \"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 341, in _send |
| | raise result |
| | ", "created": "2016-04-03T22:36:17Z"}
| | retry=self.retry) |
| | File \"/usr/lib/python2.7/site-packages/oslo_messaging/transport.py\", line 90, in _send |
| | timeout=timeout, retry=retry) |
| | File \"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 350, in send |
| | retry=retry) |
| | File \"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 341, in _send |
| | raise result |
| | ", "created": "2016-04-03T22:36:17Z"}
nova hypervisor-stats
+----------------------+-------+
| Property | Value |
+----------------------+-------+
| count | 1 |
| current_workload | 0 |
| running_vms | 1 |
Frequency is becoming frequent. /var/log/foreman/production.log excerpt: 2016-04-04 21:33:55 [I] Client disconnected. 2016-04-04 21:40:03 [W] Failed to get IP for openstack-cfme-cfme.<domain>: The specified wait_for timeout (600 seconds) was exceeded/opt/rh/ruby193/root/usr/share/gems/gems/fog-core-1.24.0/lib/fog/core/wait_for.rb:11:in `wait_for' /opt/rh/ruby193/root/usr/share/gems/gems/fog-core-1.24.0/lib/fog/core/model.rb:65:in `wait_for' /usr/share/foreman/app/models/concerns/orchestration/compute.rb:136:in `setComputeIP' /usr/share/foreman/app/models/concerns/orchestration.rb:137:in `execute' /usr/share/foreman/app/models/concerns/orchestration.rb:85:in `block in process' /usr/share/foreman/app/models/concerns/orchestration.rb:77:in `each' /usr/share/foreman/app/models/concerns/orchestration.rb:77:in `process' /usr/share/foreman/app/models/concerns/orchestration.rb:18:in `on_save' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:671:in `_run__1441965389761913084__save__2656673132011711271__callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_save_callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/callbacks.rb:264:in `create_or_update' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/persistence.rb:84:in `save' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:50:in `save' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/attribute_methods/dirty.rb:22:in `save' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:241:in `block (2 levels) in save' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:295:in `block in with_transaction_returning_status' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:208:in `transaction' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:293:in `with_transaction_returning_status' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:241:in `block in save' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:252:in `rollback_active_record_state!' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:240:in `save' /usr/share/foreman/app/models/concerns/foreman/sti.rb:29:in `save_with_type' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/persistence.rb:45:in `create' /usr/share/foreman/app/models/host.rb:16:in `method_missing' /opt/rh/ruby193/root/usr/share/gems/gems/fusor_server-0.0.1/app/lib/actions/fusor/deployment/open_stack/cfme_launch.rb:103:in `create_host' /opt/rh/ruby193/root/usr/share/gems/gems/fusor_server-0.0.1/app/lib/actions/fusor/deployment/open_stack/cfme_launch.rb:35:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:487:in `block (3 levels) in execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/progress.rb:30:in `with_progress_calculation' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/progress.rb:16:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.83/app/lib/actions/middleware/keep_locale.rb:23:in `block in run' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.83/app/lib/actions/middleware/keep_locale.rb:34:in `with_locale' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.83/app/lib/actions/middleware/keep_locale.rb:23:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/world.rb:30:in `execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:486:in `block (2 levels) in execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:485:in `catch' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:485:in `block in execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `block in with_error_handling' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `catch' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `with_error_handling' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:480:in `execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:262:in `execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/executors/parallel/worker.rb:20:in `block in on_message' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:859:in `block in assigns' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `tap' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `assigns' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:138:in `match_value' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:116:in `block in match' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `each' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `match' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/executors/parallel/worker.rb:17:in `on_message' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:82:in `on_envelope' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:72:in `receive' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `block (2 levels) in run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `loop' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `block in run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `catch' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:13:in `block in initialize' /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context' 2016-04-04 21:40:03 [W] Rolling back due to a problem: [Acquire IP address for openstack-cfme-cfme.<domain> 3 failed [#<Host::Managed id: nil, name: "openstack-cfme-cfme.<domain>", ip: nil, last_compile: nil, last_freshcheck: nil, last_report: nil, updated_at: nil, source_file_id: nil, created_at: nil, mac: nil, root_pass: "$1$<snippy>", serial: nil, puppet_status: 0, domain_id: 1, architecture_id: 1, operatingsystem_id: 3, environment_id: 3, subnet_id: nil, ptable_id: nil, medium_id: nil, build: true, comment: nil, disk: nil, installed_at: nil, model_id: nil, hostgroup_id: nil, owner_id: 3, owner_type: "User", enabled: true, puppet_ca_proxy_id: nil, managed: true, use_image: nil, image_file: nil, uuid: nil, compute_resource_id: 1, puppet_proxy_id: nil, certname: nil, image_id: 1, organization_id: 1, location_id: 2, type: "Host::Managed", otp: nil, realm_id: nil, compute_profile_id: 4, provision_method: "image", content_source_id: nil, primary_interface: nil, grub_pass: "$1$<snippy>", discovery_rule_id: nil, content_view_id: nil, lifecycle_environment_id: nil>, :setComputeIP]] 2016-04-04 21:40:03 [I] Removing Compute instance for openstack-cfme-cfme.<domain> 2016-04-04 21:40:03 [E] CFME Launch failed (RuntimeError) /opt/rh/ruby193/root/usr/share/gems/gems/fusor_server-0.0.1/app/lib/actions/fusor/deployment/open_stack/cfme_launch.rb:47:in `cfme_launch_failed' /opt/rh/ruby193/root/usr/share/gems/gems/fusor_server-0.0.1/app/lib/actions/fusor/deployment/open_stack/cfme_launch.rb:109:in `create_host' /opt/rh/ruby193/root/usr/share/gems/gems/fusor_server-0.0.1/app/lib/actions/fusor/deployment/open_stack/cfme_launch.rb:35:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:487:in `block (3 levels) in execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/progress.rb:30:in `with_progress_calculation' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/progress.rb:16:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.83/app/lib/actions/middleware/keep_locale.rb:23:in `block in run' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.83/app/lib/actions/middleware/keep_locale.rb:34:in `with_locale' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.83/app/lib/actions/middleware/keep_locale.rb:23:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/world.rb:30:in `execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:486:in `block (2 levels) in execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:485:in `catch' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:485:in `block in execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `block in with_error_handling' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `catch' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `with_error_handling' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:480:in `execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:262:in `execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/executors/parallel/worker.rb:20:in `block in on_message' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:859:in `block in assigns' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `tap' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `assigns' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:138:in `match_value' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:116:in `block in match' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `each' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `match' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/executors/parallel/worker.rb:17:in `on_message' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:82:in `on_envelope' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:72:in `receive' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `block (2 levels) in run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `loop' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `block in run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `catch' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:13:in `block in initialize' /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context' 2016-04-04 21:40:03 [E] A sub task failed (RuntimeError) /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/with_sub_plans.rb:158:in `check_for_errors!' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/with_sub_plans.rb:80:in `try_to_finish' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/with_sub_plans.rb:19:in `block in run' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:859:in `block in assigns' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `tap' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `assigns' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:138:in `match_value' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:116:in `block in match' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `each' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `match' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/with_sub_plans.rb:9:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:487:in `block (3 levels) in execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.15.7/app/lib/actions/middleware/keep_current_user.rb:23:in `block in run' /opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.15.7/app/lib/actions/middleware/keep_current_user.rb:34:in `restore_curent_user' /opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.15.7/app/lib/actions/middleware/keep_current_user.rb:23:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/progress.rb:30:in `with_progress_calculation' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/progress.rb:16:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.83/app/lib/actions/middleware/keep_locale.rb:23:in `block in run' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.83/app/lib/actions/middleware/keep_locale.rb:34:in `with_locale' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.83/app/lib/actions/middleware/keep_locale.rb:23:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/world.rb:30:in `execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:486:in `block (2 levels) in execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:485:in `catch' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:485:in `block in execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `block in with_error_handling' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `catch' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `with_error_handling' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:480:in `execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:262:in `execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/executors/parallel/worker.rb:20:in `block in on_message' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:859:in `block in assigns' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `tap' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `assigns' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:138:in `match_value' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:116:in `block in match' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `each' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `match' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/executors/parallel/worker.rb:17:in `on_message' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:82:in `on_envelope' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:72:in `receive' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `block (2 levels) in run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `loop' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `block in run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `catch' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:13:in `block in initialize' /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context' 2016-04-04 21:40:03 [W] message could not be sent Dynflow::Listeners::Serialization::Protocol::Done[request_id: 2] because Broken pipe 2016-04-04 21:41:20 [I] Processing by ForemanTasks::TasksController#show as HTML Also occurs in TP3 RC3. mcornea found the clock/hwclock was off by 1 hour on overcloud controller node. Adding https://bugzilla.redhat.com/show_bug.cgi?id=1250736 Please re-test to see if this still occurs with the egon and OSP8 changes. With QCI-1.2-RHEL-7-20160723.t.0 and QCIOOO-8.0-RHEL-7-20160719.t.0, have been able to do multiple deployments flawlessly. Moving to VERIFIED -- please reopen if this reoccurs. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2016:1862 |
Description of problem: On redeployment, CFME launch failed Version-Release number of selected component (if applicable): TP3 RC2 How reproducible: Uncertain Steps to Reproduce: 1. Install/deploy RHEL-OSP 2. Undeploy Overcloud using Director web UI 3. Create new (2nd) deployment RHEL-OSP + CFME Actual results: RHEL-OSP deployed CFME failed, error message Expected results: RHEL-OSP deployed CFME deployed Both RHEL-OSP and CFME operational Additional info: Stack traceback: Actions::Fusor::Deployment::OpenStack::CfmeLaunch Input: {"deployment_id"=>2, "locale"=>"en"} Output: {} Exception: RuntimeError: CFME Launch failed Backtrace: /opt/rh/ruby193/root/usr/share/gems/gems/fusor_server-0.0.1/app/lib/actions/fusor/deployment/open_stack/cfme_launch.rb:47:in `cfme_launch_failed' /opt/rh/ruby193/root/usr/share/gems/gems/fusor_server-0.0.1/app/lib/actions/fusor/deployment/open_stack/cfme_launch.rb:109:in `create_host' /opt/rh/ruby193/root/usr/share/gems/gems/fusor_server-0.0.1/app/lib/actions/fusor/deployment/open_stack/cfme_launch.rb:35:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:487:in `block (3 levels) in execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/progress.rb:30:in `with_progress_calculation' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action/progress.rb:16:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware.rb:16:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.83/app/lib/actions/middleware/keep_locale.rb:23:in `block in run' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.83/app/lib/actions/middleware/keep_locale.rb:34:in `with_locale' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.83/app/lib/actions/middleware/keep_locale.rb:23:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/stack.rb:22:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/middleware/world.rb:30:in `execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:486:in `block (2 levels) in execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:485:in `catch' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:485:in `block in execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `block in with_error_handling' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `catch' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:402:in `with_error_handling' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:480:in `execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/action.rb:262:in `execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/executors/parallel/worker.rb:20:in `block in on_message' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:859:in `block in assigns' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `tap' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `assigns' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:138:in `match_value' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:116:in `block in match' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `each' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `match' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/executors/parallel/worker.rb:17:in `on_message' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:82:in `on_envelope' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:72:in `receive' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `block (2 levels) in run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `loop' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `block in run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `catch' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:99:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.9/lib/dynflow/micro_actor.rb:13:in `block in initialize' /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'