Bug 1690726

Summary:	[RHOSP-13]Uploading RAW image to glance (CEPH backed) is very slow
Product:	Red Hat OpenStack	Reporter:	rohit londhe <rlondhe>
Component:	python-glance-store	Assignee:	Cyril Roelandt <cyril>
Status:	CLOSED ERRATA	QA Contact:	Mike Abrams <mabrams>
Severity:	high	Docs Contact:	Tana <tberry>
Priority:	high
Version:	13.0 (Queens)	CC:	akekane, apevec, broose, cyril, dhill, eglynn, ekuvaja, gcharot, lhh, mgarciac
Target Milestone:	beta	Keywords:	Triaged, ZStream
Target Release:	16.2 (Train on RHEL 8.4)
Hardware:	Unspecified
OS:	Linux
Whiteboard:
Fixed In Version:	python-glance-store-1.0.2-2.20210205034919.bc62bb4.el8ost	Doc Type:	Bug Fix
Doc Text:	Before this update, writing an image to RBD could be very slow. This update improves the process for writing an image to RBD, which improves the time it takes for images to be written to RBD.	Story Points:	---
Clone Of:
Clones:	1899709 (view as bug list)		Environment:
Last Closed:	2021-09-15 07:07:46 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1899709

Description rohit londhe 2019-03-20 06:04:58 UTC

Description of problem:

Getting Warning message for glance create process: "resize-before-write for each chunk" this is slowing down the image creation process.

Version-Release number of selected component (if applicable):

openstack-glance-16.0.1-5.el7ost.noarch                     
puppet-glance-12.5.0-4.el7ost.noarch                        
python2-glanceclient-2.10.0-1.el7ost.noarch                
python2-glance-store-0.23.1-0.20180213060248.ad7df98.el7ost.noarch 
python-glance-16.0.1-5.el7ost.noarch                        


How reproducible:
Yes

Steps to Reproduce:
Execute:

$ date && time glance --os-image-api-version 2 image-create --progress --visibility public --file <RAW_FILE_PATH> --name <IMAGE_NAME> --container-format bare --disk-format raw


Actual results:

Every time, the image is getting "resize-before-write" which we believe slowing down the process.

From looking at glance log (/var/log/containers/glance/api.log), I've noticed the following messages each time i upload a new image:

2019-03-11 15:30:07.332 80 WARNING glance_store._drivers.rbd [req-305d8a77-657e-4d9c-8bf8-1a5834ca6f68 6e0fef500dcd4c3ba94dee6036e3e978 7ef5effdfbc145daa3104dd985b93cbc - default default] since image size is zero we will be doing resize-before-write for each chunk which will be considerably slower than normal


[root@controller-0 ~]# time openstack image create "test" --file /root/images/test.raw --disk-format raw --container-format bare --public
+------------------+--------------------------------------------------------------------------------------------------------------------+
| Field            | Value                                                                                                              |
+------------------+--------------------------------------------------------------------------------------------------------------------+
| checksum         | 97598938c7d4acfb7a71bb97a956a53d                                                                                   |
| container_format | bare                                                                                                               |
| created_at       | 2019-03-12T08:59:28Z                                                                                               |
| disk_format      | raw                                                                                                                |
| file             | /v2/images/538ce5ca-4ab6-415f-b0e4-184a2a94b970/file                                                               |
| id               | 538ce5ca-4ab6-415f-b0e4-184a2a94b970                                                                               |
| min_disk         | 0                                                                                                                  |
| min_ram          | 0                                                                                                                  |
| name             | test                                                                                                   |
| owner            | 7ef5effdfbc145daa3104dd985b93cbc                                                                                   |
| properties       | direct_url='rbd://30095f51-c118-47ea-8841-e2796c2bad40/devops_os_images/538ce5ca-4ab6-415f-b0e4-184a2a94b970/snap' |
| protected        | False                                                                                                              |
| schema           | /v2/schemas/image                                                                                                  |
| size             | 44023414784                                                                                                        |
| status           | active                                                                                                             |
| tags             |                                                                                                                    |
| updated_at       | 2019-03-12T09:05:24Z                                                                                               |
| virtual_size     | None                                                                                                               |
| visibility       | public                                                                                                             |
+------------------+--------------------------------------------------------------------------------------------------------------------+
real	        5m58.891s
user	0m7.672s
sys		0m20.771s




Expected results:

"resize-before-write" is not expected to happen in API v2

Additional info:

This is similar to [1] but for RHSOP-13 version.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1377796

Comment 8 Erno Kuvaja 2019-04-03 11:49:14 UTC

I added the upstream bug and code review to the external references where this is being addressed.

Comment 9 Cyril Roelandt 2019-04-03 16:17:39 UTC

So apparently, this is no longer an issue when using the new image import process. I'm not sure whether this is going to be fixed upstream, or even easy to backport to 13 :/

Comment 15 Cyril Roelandt 2020-07-15 19:07:09 UTC

@Abhishek: the spec you mentioned has been abandoned. Is https://review.opendev.org/#/c/741121/ a way to solve this bz?

Comment 16 Abhishek Kekane 2020-07-16 05:46:21 UTC

(In reply to Cyril Roelandt from comment #15)
> @Abhishek: the spec you mentioned has been abandoned. Is
> https://review.opendev.org/#/c/741121/ a way to solve this bz?

No the spec you mentioned is to calculate virtual size of image.
We need to check at rbd store whether we can use virtual size to reduce/resolve this issue.

Comment 26 errata-xmlrpc 2021-09-15 07:07:46 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform (RHOSP) 16.2 enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:3483