Bug 1904971 - Deployment with octavia fails when using cinder as a backend for glance
Summary: Deployment with octavia fails when using cinder as a backend for glance
Keywords:
Status: CLOSED EOL
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Linux
urgent
high
Target Milestone: ---
: ---
Assignee: Cyril Roelandt
QA Contact: Brent Eagles
URL:
Whiteboard:
Depends On: 1712421 1955186
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-12-07 09:51 UTC by rohit londhe
Modified: 2024-06-13 23:39 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-07-27 14:43:32 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 759276 0 None MERGED Switch Octavia external tasks to 'post deploy' 2021-02-19 04:08:27 UTC
Red Hat Issue Tracker OSP-3464 0 None None None 2022-08-23 16:13:15 UTC

Description rohit londhe 2020-12-07 09:51:00 UTC
Description of problem:
When we set the GlanceBackend parameter to cinder our overcloud fails to install due to octavia image not being able to upload to glance image store.

~~~~~~

2020-11-20 12:38:02Z [overcloud.AllNodesDeploySteps.WorkflowTasks_Step5_Execution]: CREATE_IN_PROGRESS  state changed
2020-11-20 12:39:35Z [overcloud.AllNodesDeploySteps.WorkflowTasks_Step5_Execution]: CREATE_FAILED  resources.WorkflowTasks_Step5_Execution: Failure caused by error in tasks: octavia_post_workflow
  octavia_post_workflow [task_ex_id=35476e8e-2568-4ae3-93c4-dda0dad4bdfc] -> Failure caused by error in tasks: upload_amphora
  upload_amphora [task_ex_id=c
2020-11-20 12:39:35Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED  Resource CREATE failed: resources.WorkflowTasks_Step5_Execution: Failure caused by error in tasks: octavia_post_workflow
  octavia_post_workflow [task_ex_id=35476e8e-2568-4ae3-93c4-dda0dad4bdfc] -> Failure caused by error in tasks: upload_amphora
  uplo
2020-11-20 12:39:36Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED  resources.WorkflowTasks_Step5_Execution: resources.AllNodesDeploySteps.Failure caused by error in tasks: octavia_post_workflow
  octavia_post_workflow [task_ex_id=35476e8e-2568-4ae3-93c4-dda0dad4bdfc] -> Failure caused by error in tasks: upload_amphora
2020-11-20 12:39:36Z [overcloud]: CREATE_FAILED  Resource CREATE failed: resources.WorkflowTasks_Step5_Execution: resources.AllNodesDeploySteps.Failure caused by error in tasks: octavia_post_workflow
  octavia_post_workflow [task_ex_id=35476e8e-2568-4ae3-93c4-dda0dad4bdfc] -> Failure caused by error in
 Stack overcloud CREATE_FAILED
overcloud.AllNodesDeploySteps.WorkflowTasks_Step5_Execution:
  resource_type: OS::TripleO::WorkflowSteps
  physical_resource_id: a0e2dfb8-47b6-4794-8e15-b6156d9c5ae7
  status: CREATE_FAILED
  status_reason: |
    ...

     Unexpected error while running command.
    Command: ansible-playbook -vvvvv /usr/share/tripleo-common/playbooks/octavia-files.yaml --user stack --extra-vars {"amp_image_tag": "amphora-image", "amp_ssh_key_path": "", "amp_to_raw": false,

~~~~~

When I check the glance configuration on the controller nodes it is configured as per the templates

~~~~~

[glance_store]
#
# From glance.store
#
# List of enabled Glance stores.
#
#stores = file,http
stores=http,cinder

#
# The default scheme to use for storing images.
#
#default_store = file
default_store=cinder

~~~~~

I think the problem is that cinder-volume has not been set up at this stage in the overcloud deployment (it is not running when I execute command pcs status) so it cannot upload the image to the backend storage array.


Version-Release number of selected component (if applicable):
RHOSP13 Z12

openstack-octavia-api-2.1.2-4.el7ost.noarch                 Mon Aug  3 10:24:30 2020
openstack-octavia-common-2.1.2-4.el7ost.noarch              Mon Aug  3 10:22:53 2020
openstack-octavia-health-manager-2.1.2-4.el7ost.noarch      Mon Aug  3 10:24:30 2020
openstack-octavia-housekeeping-2.1.2-4.el7ost.noarch        Mon Aug  3 10:24:30 2020
openstack-octavia-worker-2.1.2-4.el7ost.noarch              Mon Aug  3 10:24:30 2020
puppet-octavia-12.4.0-16.el7ost.noarch                      Mon Aug  3 10:29:49 2020
python2-octaviaclient-1.4.1-1.el7ost.noarch                 Mon Aug  3 10:21:28 2020
python-octavia-2.1.2-4.el7ost.noarch                        Mon Aug  3 10:22:51 2020


How reproducible:
100%

Steps to Reproduce:

attachments contain the templates used for overcloud deployment

Actual results:
overcloud+octavia installation failing for below command:

"openstack image create --disk-format qcow2 --container-format bare --tag amphora-image --file /usr/share/rhosp-        director-images/octavia-amphora.qcow2 --property hw_architecture=x86_64 --private octavia-amphora-13.0-20200901.1.x86_64" 

Expected results:
overcloud+octavia installation should get successfully installed when GlanceBackend is set to cinder
Additional info:

We are trying to deploy RHOSP13 Z12 - Please note when we configure the templates for GlanceBackend file, the overcloud successfully installs.
There is an observation that cinder-volume was not up at the time Octavia image tried to upload to glance so I think it is an ordering thing. Can you check the code to see the order in the install? If Octavia is before cinder-volume setup then it will never work with both features as cinder volume will not be up in time for the Octavia image to be uploaded.

Comment 3 Cyril Roelandt 2020-12-10 03:34:40 UTC
This looks a lot like https://bugzilla.redhat.com/show_bug.cgi?id=1712421 . I added a comment there, let's see if we can find a solution.

Comment 7 Cyril Roelandt 2021-01-11 20:23:43 UTC
Hello,


Could you confirm this is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1712421 as I mentioned in #3?



Regards,
Cyril

Comment 8 rohit londhe 2021-01-20 03:44:00 UTC
(In reply to Cyril Roelandt from comment #7)
> Hello,
> 
> 
> Could you confirm this is a duplicate of
> https://bugzilla.redhat.com/show_bug.cgi?id=1712421 as I mentioned in #3?
> 
> 
> 
> Regards,
> Cyril

I confirm that this is the same problem. We can mark this as a duplicate of the above

Comment 13 Cyril Roelandt 2021-01-20 20:29:27 UTC
Yes, I think we should verify this bz. I added a link to the patch currently under review upstream (mentioned by Brent at https://bugzilla.redhat.com/show_bug.cgi?id=1712421#c43).


Note You need to log in before you can comment on or make changes to this bug.