Bug 1608334

Summary: No amphora image and octavia-ssh-key keypair are created during the deployment
Product: Red Hat OpenStack Reporter: Alexander Stafeyev <astafeye>
Component: openstack-tripleo-commonAssignee: Carlos Goncalves <cgoncalves>
Status: CLOSED ERRATA QA Contact: Alexander Stafeyev <astafeye>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 14.0 (Rocky)CC: amuller, apevec, astafeye, bcafarel, gcheresh, itbrown, lhh, mburns, mweetman, slinaber, srevivo, tvignaud
Target Milestone: betaKeywords: AutomationBlocker, Regression, Triaged
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-common-9.4.1-0.20181012010875.67bab16.el7ost openstack-tripleo-heat-templates-9.0.1-0.20181013060883.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1615128 (view as bug list) Environment:
Last Closed: 2019-01-11 11:50:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1640024    
Bug Blocks:    

Description Alexander Stafeyev 2018-07-25 10:31:55 UTC
I tried to create a LB but it failed . after looking in logs I saw that there is no image for octavia to use. 

2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server [-] Exception during message handling: ComputeBuildException: Failed to build compute instance due to: No Glance images are tagged with amphora-image tag.
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/controller/queue/endpoint.py", line 44, in create_load_balancer
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server     self.worker.create_load_balancer(load_balancer_id)
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 214, in wrapped_f
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server     return self.call(f, *args, **kw)
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 295, in call
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server     start_time=start_time)
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 252, in iter
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server     return fut.result()
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server     return self.__get_result()
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 298, in call
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server     result = fn(*args, **kwargs)
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/controller/worker/controller_worker.py", line 319, in create_load_balancer
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server     create_lb_tf.run()
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server     for _state in self.run_iter(timeout=timeout):
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server     failure.Failure.reraise_if_any(er_failures)
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 336, in reraise_if_any
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server     failures[0].reraise()
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 343, in reraise
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server     six.reraise(*self._exc_info)
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server     result = task.execute(**arguments)
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/controller/worker/tasks/compute_tasks.py", line 144, in execute
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server     server_group_id=server_group_id, ports=ports)
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/controller/worker/tasks/compute_tasks.py", line 98, in execute
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server     server_group_id=server_group_id)
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/compute/drivers/nova_driver.py", line 160, in build
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server     raise exceptions.ComputeBuildException(fault=e)
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server ComputeBuildException: Failed to build compute instance due to: No Glance images are tagged with amphora-image tag.
2018-07-25 10:20:42.397 22 ERROR oslo_messaging.rpc.server

Comment 2 Carlos Goncalves 2018-07-25 10:43:44 UTC
In the unlikely event that the RPM package containing the amphora image cannot be installed, the installation should fail right away, so if it did not fail I wonder what happened.

https://github.com/openstack/tripleo-common/blob/master/playbooks/roles/octavia-undercloud/tasks/main.yml#L13-L14

Could you please share the log from the Mistral octavia workflow?

Comment 3 Alexander Stafeyev 2018-07-25 11:36:54 UTC
(In reply to Carlos Goncalves from comment #2)
> In the unlikely event that the RPM package containing the amphora image
> cannot be installed, the installation should fail right away, so if it did
> not fail I wonder what happened.
> 
> https://github.com/openstack/tripleo-common/blob/master/playbooks/roles/
> octavia-undercloud/tasks/main.yml#L13-L14
> 
> Could you please share the log from the Mistral octavia workflow?

Which logs exactly do u need ? 

p.s.
I will add sosreport soon

Comment 5 Alexander Stafeyev 2018-07-25 11:43:26 UTC
We have the same issue with keypair - /var/log/containers/octavia/worker.log:2018-07-25 11:35:31.046 22 ERROR oslo_messaging.rpc.server ComputeBuildException: Failed to build compute instance due to: Invalid key_name provided.  


No keypair found

Comment 6 Carlos Goncalves 2018-08-13 14:14:23 UTC
From /var/lib/mistral/4842bfd0-8a7c-4ff2-b008-e52c6ea9c405/octavia-ansible/octavia-ansible.log


2018-08-13 08:52:17,312 p=14240 u=mistral |   [WARNING]: Could not match supplied host pattern, ignoring: undercloud

2018-08-13 08:52:17,314 p=14240 u=mistral |  PLAY [undercloud[0]] ***********************************************************
2018-08-13 08:52:17,314 p=14240 u=mistral |  skipping: no hosts matched
2018-08-13 08:52:17,319 p=14240 u=mistral |  PLAY [octavia_nodes[0]] ********************************************************
2018-08-13 08:52:17,338 p=14240 u=mistral |  TASK [Gathering Facts] *********************************************************
2018-08-13 08:52:17,338 p=14240 u=mistral |  Monday 13 August 2018  08:52:17 -0400 (0:00:00.105)       0:00:00.105 *********
2018-08-13 08:52:24,123 p=14240 u=mistral |  ok: [controller-2]
[...]

Host "undercloud[0]" could not be found when running the playbook tripleo-common/playbooks/octavia-files.yaml [1]. That must be due to the migration of Octavia to external_deploy_task [2] that happened in Rocky cycle.

So, the trouble maker was the renaming of 'undercloud' to 'Undercloud' in tripleo-heat-templates/tree/docker/services/octavia/octavia-deployment-config.yaml. 

[root@undercloud-0 mistral]# cat /var/lib/mistral/4842bfd0-8a7c-4ff2-b008-e52c6ea9c405/octavia-ansible/inventory.yaml
octavia_nodes:
  hosts:
    controller-2:
      ansible_user: tripleo-admin
      ansible_host: 192.168.24.19
      ansible_become: true
    controller-1:
      ansible_user: tripleo-admin
      ansible_host: 192.168.24.14
      ansible_become: true
    controller-0:
      ansible_user: tripleo-admin
      ansible_host: 192.168.24.16
      ansible_become: true
    
Undercloud:
  hosts:
    undercloud-0:
      ansible_host: localhost
      ansible_become: false
      ansible_connection: local


We need to update octavia-files.yaml to reflect that change. 

[1] https://github.com/openstack/tripleo-common/blob/336cd3c6364c8c9be88c840253db0d0b54336c75/playbooks/octavia-files.yaml#L2
[2] https://review.openstack.org/#/c/559374/

Comment 11 Nir Magnezi 2018-10-07 10:27:04 UTC
We are waiting for a W+1 for Carlos's patch: https://review.openstack.org/#/c/591413/

Comment 25 errata-xmlrpc 2019-01-11 11:50:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045

Comment 26 Red Hat Bugzilla 2023-09-14 04:32:04 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days