Bug 1690726 - [RHOSP-13]Uploading RAW image to glance (CEPH backed) is very slow
Summary: [RHOSP-13]Uploading RAW image to glance (CEPH backed) is very slow
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-glance-store
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Linux
high
high
Target Milestone: beta
: 16.2 (Train on RHEL 8.4)
Assignee: Cyril Roelandt
QA Contact: Mike Abrams
Tana
URL:
Whiteboard:
Depends On:
Blocks: 1899709
TreeView+ depends on / blocked
 
Reported: 2019-03-20 06:04 UTC by rohit londhe
Modified: 2023-10-06 18:11 UTC (History)
10 users (show)

Fixed In Version: python-glance-store-1.0.2-2.20210205034919.bc62bb4.el8ost
Doc Type: Bug Fix
Doc Text:
Before this update, writing an image to RBD could be very slow. This update improves the process for writing an image to RBD, which improves the time it takes for images to be written to RBD.
Clone Of:
: 1899709 (view as bug list)
Environment:
Last Closed: 2021-09-15 07:07:46 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1792710 0 None None None 2019-04-03 11:49:13 UTC
OpenStack gerrit 602577 0 'None' NEW Specify image size to avoid glance backend resize 2021-02-21 10:13:00 UTC
OpenStack gerrit 746579 0 None MERGED Ramp up rbd resize to avoid excessive calls 2021-02-21 10:13:00 UTC
Red Hat Issue Tracker OSP-1543 0 None None None 2022-08-30 14:24:25 UTC
Red Hat Knowledge Base (Solution) 5105981 0 None None None 2020-05-26 15:38:38 UTC
Red Hat Product Errata RHEA-2021:3483 0 None None None 2021-09-15 07:08:28 UTC

Description rohit londhe 2019-03-20 06:04:58 UTC
Description of problem:

Getting Warning message for glance create process: "resize-before-write for each chunk" this is slowing down the image creation process.

Version-Release number of selected component (if applicable):

openstack-glance-16.0.1-5.el7ost.noarch                     
puppet-glance-12.5.0-4.el7ost.noarch                        
python2-glanceclient-2.10.0-1.el7ost.noarch                
python2-glance-store-0.23.1-0.20180213060248.ad7df98.el7ost.noarch 
python-glance-16.0.1-5.el7ost.noarch                        


How reproducible:
Yes

Steps to Reproduce:
Execute:

$ date && time glance --os-image-api-version 2 image-create --progress --visibility public --file <RAW_FILE_PATH> --name <IMAGE_NAME> --container-format bare --disk-format raw


Actual results:

Every time, the image is getting "resize-before-write" which we believe slowing down the process.

From looking at glance log (/var/log/containers/glance/api.log), I've noticed the following messages each time i upload a new image:

2019-03-11 15:30:07.332 80 WARNING glance_store._drivers.rbd [req-305d8a77-657e-4d9c-8bf8-1a5834ca6f68 6e0fef500dcd4c3ba94dee6036e3e978 7ef5effdfbc145daa3104dd985b93cbc - default default] since image size is zero we will be doing resize-before-write for each chunk which will be considerably slower than normal


[root@controller-0 ~]# time openstack image create "test" --file /root/images/test.raw --disk-format raw --container-format bare --public
+------------------+--------------------------------------------------------------------------------------------------------------------+
| Field            | Value                                                                                                              |
+------------------+--------------------------------------------------------------------------------------------------------------------+
| checksum         | 97598938c7d4acfb7a71bb97a956a53d                                                                                   |
| container_format | bare                                                                                                               |
| created_at       | 2019-03-12T08:59:28Z                                                                                               |
| disk_format      | raw                                                                                                                |
| file             | /v2/images/538ce5ca-4ab6-415f-b0e4-184a2a94b970/file                                                               |
| id               | 538ce5ca-4ab6-415f-b0e4-184a2a94b970                                                                               |
| min_disk         | 0                                                                                                                  |
| min_ram          | 0                                                                                                                  |
| name             | test                                                                                                   |
| owner            | 7ef5effdfbc145daa3104dd985b93cbc                                                                                   |
| properties       | direct_url='rbd://30095f51-c118-47ea-8841-e2796c2bad40/devops_os_images/538ce5ca-4ab6-415f-b0e4-184a2a94b970/snap' |
| protected        | False                                                                                                              |
| schema           | /v2/schemas/image                                                                                                  |
| size             | 44023414784                                                                                                        |
| status           | active                                                                                                             |
| tags             |                                                                                                                    |
| updated_at       | 2019-03-12T09:05:24Z                                                                                               |
| virtual_size     | None                                                                                                               |
| visibility       | public                                                                                                             |
+------------------+--------------------------------------------------------------------------------------------------------------------+
real	        5m58.891s
user	0m7.672s
sys		0m20.771s




Expected results:

"resize-before-write" is not expected to happen in API v2

Additional info:

This is similar to [1] but for RHSOP-13 version.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1377796

Comment 8 Erno Kuvaja 2019-04-03 11:49:14 UTC
I added the upstream bug and code review to the external references where this is being addressed.

Comment 9 Cyril Roelandt 2019-04-03 16:17:39 UTC
So apparently, this is no longer an issue when using the new image import process. I'm not sure whether this is going to be fixed upstream, or even easy to backport to 13 :/

Comment 15 Cyril Roelandt 2020-07-15 19:07:09 UTC
@Abhishek: the spec you mentioned has been abandoned. Is https://review.opendev.org/#/c/741121/ a way to solve this bz?

Comment 16 Abhishek Kekane 2020-07-16 05:46:21 UTC
(In reply to Cyril Roelandt from comment #15)
> @Abhishek: the spec you mentioned has been abandoned. Is
> https://review.opendev.org/#/c/741121/ a way to solve this bz?

No the spec you mentioned is to calculate virtual size of image.
We need to check at rbd store whether we can use virtual size to reduce/resolve this issue.

Comment 26 errata-xmlrpc 2021-09-15 07:07:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform (RHOSP) 16.2 enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:3483


Note You need to log in before you can comment on or make changes to this bug.