Bug 1593341

Summary: Allow option for amphora image to be automatically uploaded in raw format
Product: Red Hat OpenStack Reporter: Andrew Austin <aaustin>
Component: openstack-tripleo-commonAssignee: Carlos Goncalves <cgoncalves>
Status: CLOSED ERRATA QA Contact: Alexander Stafeyev <astafeye>
Severity: medium Docs Contact:
Priority: urgent    
Version: 13.0 (Queens)CC: aaustin, astafeye, augol, bschmaus, cgoncalves, gfidente, hrybacki, itbrown, jtaleric, mariel, mburns, slinaber, tvignaud
Target Milestone: z1Keywords: Regression, Triaged, ZStream
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-common-9.4.1-0.20190119050432.261de49 openstack-tripleo-heat-templates-9.0.1-0.20181013060912.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1598146 (view as bug list) Environment:
Last Closed: 2019-03-18 13:03:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1598146    

Description Andrew Austin 2018-06-20 15:21:31 UTC
Description of problem:
When deploying an overcloud with Octavia, the amphora image is uploaded to Glance as QCOW2. When using the RBD backend for Glance and Nova, this causes the image to be converted to RAW on each deployment. With slower storage, this leads to load balancer creation to fail with a compute timeout exception from Octavia.

It would be nice to have an option to cause the post-deploy workflow to convert this image to raw and upload it as such automatically rather than having the user delete our automatically-uploaded image and replace it with a new one every time.

Version-Release number of selected component (if applicable):
RHOSP13 RC

How reproducible:
Deploy an overcloud using Ceph storage and OSP13 with Octavia. The amphora image will be uploaded to Glance in QCOW2 format and creating a load balancer will either fail or complete after a long period of time due to image conversion.

Steps to Reproduce:
1. Deploy an overcloud using Ceph storage and Octavia. Use realistically slow storage if using a nested virtualization testbed.
2. Observe that an amphora image is uploaded automatically to Glance in QCOW2 format
3. Attempt to create a load balancer

Actual results:
The load balancer creation fails with a compute timeout exception

Expected results:
The load balancer creation succeeds.

Additional info:

Comment 1 Carlos Goncalves 2018-06-22 18:04:45 UTC
Thank you for spotting this! In addition to a new option, we should weight the pros/cons of uploading in raw format the default.

Comment 4 Giulio Fidente 2018-06-28 15:31:50 UTC
The user *can* upload an image in QCOW2 format but to boot from rbd Nova will have to convert it in RAW format.

If you want to do an automatic conversion of the image to RAW format when rbd backend is enabled in Nova, set it based on the value of the "NovaEnableRbdBackend"

Comment 5 Carlos Goncalves 2018-06-29 16:49:06 UTC
Removing RFE tag. This is an issue that was observed by a field member, acked by engineering teams (Storage and Network DFGs) and that should be fixed.

If RBD backend is enabled, Director will convert the image to RAW and upload it to Glance as so. If not, no format conversion will be performed.

Comment 7 Joe Talerico 2018-12-18 17:25:06 UTC
Trying with latest OSP14 puddle and this seems to still be a issue.

Comment 8 Joe Talerico 2018-12-18 23:48:59 UTC
Sharing the results : https://gist.github.com/jtaleric/747b66798f7bb155698b91add1eeb790

From the gist above :
3d607db4-1264-4f08-87a3-6d549c3cd4e5 is with the image as raw on Ceph
90c092cf-d9d8-49c9-b325-548358c19a8e is with the image as qcow2 on Ceph

From the gist above you can see Octavia will fail with any concurrency with the image misconfigured (qcow2 instead of raw).

Comment 19 Carlos Goncalves 2019-02-25 15:58:07 UTC
*** Bug 1680520 has been marked as a duplicate of this bug. ***

Comment 31 Alexander Stafeyev 2019-03-04 12:54:14 UTC
+--------------------------------------+----------------------------------------+--------+
| ID                                   | Name                                   | Status |
+--------------------------------------+----------------------------------------+--------+
| 2216d407-4193-4c27-b1cc-e0fc23748811 | octavia-amphora-14.0-20190226.1.x86_64 | active |
+--------------------------------------+----------------------------------------+--------+
(overcloud) [stack@undercloud-0 ~]$ openstack image show 2216d407-4193-4c27-b1cc-e0fc23748811
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field            | Value                                                                                                                                                                                                                                                                                                                                                                                                                                         |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| checksum         | 5fbc891c6d7b3fe93cb274415815c7dc                                                                                                                                                                                                                                                                                                                                                                                                              |
| container_format | bare                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| created_at       | 2019-03-04T12:40:15Z                                                                                                                                                                                                                                                                                                                                                                                                                          |
| disk_format      | qcow2                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| file             | /v2/images/2216d407-4193-4c27-b1cc-e0fc23748811/file                                                                                                                                                                                                                                                                                                                                                                                          |
| id               | 2216d407-4193-4c27-b1cc-e0fc23748811                                                                                                                                                                                                                                                                                                                                                                                                          |
| min_disk         | 0                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| min_ram          | 0                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| name             | octavia-amphora-14.0-20190226.1.x86_64                                                                                                                                                                                                                                                                                                                                                                                                        |
| owner            | 639c47aecfcc4929ab0d347849b64b09                                                                                                                                                                                                                                                                                                                                                                                                              |
| properties       | direct_url='rbd://c735da94-3e6a-11e9-811c-525400ff7314/images/2216d407-4193-4c27-b1cc-e0fc23748811/snap', locations='[{u'url': u'rbd://c735da94-3e6a-11e9-811c-525400ff7314/images/2216d407-4193-4c27-b1cc-e0fc23748811/snap', u'metadata': {}}]', os_hash_algo='sha512', os_hash_value='013791bf0dc1c82354bab7c4c24c78ca86df4b28b40acad1a9990d4fe38969ef2c7829da3649f193d71ebd5fccf2376d2dc70505be2a252a45b2c2740a84d8bd', os_hidden='False' |
| protected        | False                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| schema           | /v2/schemas/image                                                                                                                                                                                                                                                                                                                                                                                                                             |
| size             | 837156864                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| status           | active                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| tags             | amphora-image                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| updated_at       | 2019-03-04T12:40:28Z                                                                                                                                                                                                                                                                                                                                                                                                                          |
| virtual_size     | None                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| visibility       | shared                                                                                                                                                                                                                                                                                                                                                                                                                                        |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack image show 2216d407-4193-4c27-b1cc-e0fc23748811 -f json
{
  "min_ram": 0, 
  "status": "active", 
  "virtual_size": null, 
  "name": "octavia-amphora-14.0-20190226.1.x86_64", 
  "tags": "amphora-image", 
  "checksum": "5fbc891c6d7b3fe93cb274415815c7dc", 
  "created_at": "2019-03-04T12:40:15Z", 
  "disk_format": "qcow2", 
  "updated_at": "2019-03-04T12:40:28Z", 
  "visibility": "shared", 
  "properties": "direct_url='rbd://c735da94-3e6a-11e9-811c-525400ff7314/images/2216d407-4193-4c27-b1cc-e0fc23748811/snap', locations='[{u'url': u'rbd://c735da94-3e6a-11e9-811c-525400ff7314/images/2216d407-4193-4c27-b1cc-e0fc23748811/snap', u'metadata': {}}]', os_hash_algo='sha512', os_hash_value='013791bf0dc1c82354bab7c4c24c78ca86df4b28b40acad1a9990d4fe38969ef2c7829da3649f193d71ebd5fccf2376d2dc70505be2a252a45b2c2740a84d8bd', os_hidden='False'", 
  "owner": "639c47aecfcc4929ab0d347849b64b09", 
  "protected": false, 
  "file": "/v2/images/2216d407-4193-4c27-b1cc-e0fc23748811/file", 
  "container_format": "bare", 
  "min_disk": 0, 
  "size": 837156864, 
  "id": "2216d407-4193-4c27-b1cc-e0fc23748811", 
  "schema": "/v2/schemas/image"

Comment 37 Itzik Brown 2019-03-06 15:31:22 UTC
It fails for me.

Comment 41 errata-xmlrpc 2019-03-18 13:03:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0446