Bug 2007128

Summary:	Uploading images fails with error "Cannot get 'write' permission without 'resize': Image size is not a multiple of request alignment"
Product:	Container Native Virtualization (CNV)	Reporter:	nijin ashok <nashok>
Component:	Storage	Assignee:	Alexander Wels <awels>
Status:	CLOSED INSUFFICIENT_DATA	QA Contact:	Kevin Alon Goldblatt <kgoldbla>
Severity:	medium	Docs Contact:
Priority:	high
Version:	2.6.10	CC:	alitke, awels, cnv-qe-bugs, fdeutsch, yadu
Target Milestone:	---
Target Release:	4.10.0
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-03-09 13:20:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description nijin ashok 2021-09-23 08:07:39 UTC

Description of problem:

This is similar to bug 1976730 but observed while uploading the image using virtctl and fs overhead is not involved.

An image with a size of 32214352384 bytes was uploaded using virtctl.

~~~
# qemu-img info disk.qcow2
image: disk.qcow2
file format: qcow2
virtual size: 30G (32214352384 bytes)
disk size: 22G
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false

# virtctl image-upload dv dv1 --size=50Gi --image-path=disk.qcow2
~~~

The storage used was "Azure files" and block size is 1 MB. And it is not aligned (32214352384/1048576 = 30722.0004883). So converting to raw failed with the error below.

~~~
E0831 13:24:56.377963       1 prlimit.go:176] qemu-img: Could not open '/data/disk.img': Cannot get 'write' permission without 'resize': Image size is not a multiple of request alignment

E0831 13:24:56.389158       1 data-processor.go:229] exit status 1
qemu-img execution failed
~~~

We were able to upload the image after resizing the image to 32225886208.

~~~
# qemu-img resize -f qcow2 disk.qcow2 32225886208
~~~

Version-Release number of selected component (if applicable):

Openshift Virtualization 2.6.

How reproducible:

100%

Steps to Reproduce:

Use virtctl to upload an image which is not aligned with the storage blocks.

Actual results:

Uploading images fails with error "Cannot get 'write' permission without 'resize': Image size is not a multiple of request alignment"

Expected results:

We may need to resize the disk automatically to align it with block size like we are aligning to 1 MB here  https://github.com/kubevirt/containerized-data-importer/pull/1873/commits/c9029e4fd79920066677387203d1f6eb322d7bd2

Additional info:

Also, I can see that cache is changed to writeback as per https://github.com/kubevirt/containerized-data-importer/pull/1919. I think alignment is only checked with cache=none. So we may already have the fix(?).

Also, there is a related qemu patch https://www.mail-archive.com/qemu-block@nongnu.org/msg88017.html

Comment 1 Adam Litke 2021-09-27 16:58:39 UTC

Alexander, it seems we may need that block size detection code after all.

Comment 2 Alexander Wels 2021-10-01 12:24:51 UTC

I don't know if I want to revive it, it doesn't work on NFS for instance, it only worked for local storage.

Comment 3 Adam Litke 2021-10-05 12:08:33 UTC

Ok, which block size should we align to in order to satisfy all users?

Comment 4 Alexander Wels 2021-10-05 13:10:47 UTC

Looking at the bug description, this is similar but not the same as the alignment bug we fixed. It seems to fail on the initial conversion and not the resize part (which is the part we fixed). Without seeing the entire log I am unsure if it failed on the conversion or resize step.

Comment 5 Alexander Wels 2021-12-01 20:01:49 UTC

So looking at the bug description again, I think I am going to agree with the submitter that we already mitigated the problem when switching cache mode. The problem occurred in the conversion step from the scratch space to the target volume. That is now no longer using direct io, so we should be good. I am going to move this to on qe so it can be retested.

Comment 8 Kevin Alon Goldblatt 2022-01-20 17:57:36 UTC

@nashok Hi, can you give us any additional information regarding the setup of the storage class on 'Azure files'. 
How did they configure the "Azure files" to 1MB block size
Also the output of:
oc get sc
oc get sc <azure-storage> -oyaml

Comment 16 Yan Du 2022-03-09 13:20:17 UTC

we believe we have fixed this issue, but we are unable to verify since we don't have an env where we can reproduce