Bug 1945121

Summary: [CNV][V2V] Migration from RHV fails if using NFS as a target storage
Product: Container Native Virtualization (CNV) Reporter: Igor Braginsky <ibragins>
Component: StorageAssignee: Fabien Dupont <fdupont>
Status: CLOSED ERRATA QA Contact: Igor Braginsky <ibragins>
Severity: urgent Docs Contact: Avital Pinnick <apinnick>
Priority: unspecified    
Version: 2.6.1CC: amastbau, cnv-qe-bugs, dagur, fdupont, istein, mrashish, yadu, ycui
Target Milestone: ---Keywords: Regression, TestBlocker
Target Release: 2.6.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1946177 (view as bug list) Environment:
Last Closed: 2021-05-04 20:09:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1946177    
Attachments:
Description Flags
Pod's log with errors
none
dv_yaml none

Description Igor Braginsky 2021-03-31 11:41:58 UTC
Created attachment 1768054 [details]
Pod's log with errors

Created attachment 1768054 [details]
Pod's log with errors

Description of problem: Migration from RHV fails if using NFS as a target storage


Version-Release number of selected component (if applicable): CNV 2.6.1


How reproducible: 100%


Steps to Reproduce:
1. Start migration wizard as usual
2. When defining target storage - use NFS with filesystem
3. Start migration and take a look on import log

Actual results: Migration will fail on resizing target image


Expected results: Migration succeeded


Additional info:

I0331 08:51:16.058986       1 data-processor.go:239] New phase: Resize
W0331 08:51:16.423118       1 data-processor.go:331] Available space less than requested size, resizing image to available space 4281974784.
I0331 08:51:16.423155       1 data-processor.go:337] Expanding image size to: 4281974784
E0331 08:51:16.438761       1 prlimit.go:174] qemu-img failed output is:
E0331 08:51:16.438789       1 prlimit.go:175] 
E0331 08:51:16.438796       1 prlimit.go:176] qemu-img: Use the --shrink option to perform a shrink operation.
qemu-img: warning: Shrinking an image will delete all data beyond the shrunken image's end. Before performing such an operation, make sure there is no important data there.
E0331 08:51:16.438856       1 data-processor.go:236] exit status 1
qemu-img execution failed

Comment 1 Amos Mastbaum 2021-03-31 12:18:08 UTC
Same behavior with cephs/Filesystem (while cephs/block has no issues)
Maybe this is the issue.

Comment 2 Amos Mastbaum 2021-03-31 12:18:25 UTC
Same behavior with cephs/Filesystem (while cephs/block has no issues)
Maybe this is the issue.

Comment 3 Maya Rashish 2021-03-31 14:24:02 UTC
I backported the PR that introduced this breakage to CDI: https://github.com/kubevirt/containerized-data-importer/pull/1612
I am happy to rip it out of 2.6.1 if it comes to that, there is no strong reason to backport it.
But that will push out the problem to 4.8, not make it go away.

Comment 4 Alexander Wels 2021-03-31 18:58:00 UTC
Can I get a copy of the disk image that is written to the PVC? The PVC is already converted, but not yet resized. That way I can use it to track down the problem. Also I would like a copy of the DV that creates the PVC.

Comment 5 Ilanit Stein 2021-03-31 19:51:29 UTC
Created attachment 1768168 [details]
dv_yaml

Comment 6 Maya Rashish 2021-04-01 16:06:58 UTC
This is the workaround:

oc patch cdi cdi-kubevirt-hyperconverged --type merge -p '{"spec": {"config": {"filesystemOverhead": {"global": "0"}}}}'

This will unset the value:

oc patch cdi cdi-kubevirt-hyperconverged --type merge -p '{"spec": {"config": {"filesystemOverhead": {}}}}'


Does this work around work?

Comment 7 Ilanit Stein 2021-04-01 17:58:51 UTC
Thanks Maya,

The work around works fine. 

I used a source VM fedora32 4G disk and run those steps:

1. Without running the workaround applied yet, 
VM import from RHV to destination storage NFS/Filesystem-> Failed at the resize phase, as expected.

2. Apply the workaround

3. VM import from RHV to destination storage NFS/Filesystem ->PASS, VM started OK.

4. VM import from RHV to destination storage Ceph-RBD/Block ->PASS.

5. Run the command to unset the value - If here the intention was to return the behavior to how it was before applying the work around then this didn't work. VM import from RHV to NFS still PASSES. 
(Looks like the resize phase is still not running, but I didn't catch the exact log last message on this). 

Maya, 
Can you advise on this?

Comment 8 Maya Rashish 2021-04-01 19:19:48 UTC
Add:
oc patch cdi cdi-kubevirt-hyperconverged --type merge -p '{"spec": {"config": {"filesystemOverhead": {"global": "0"}}}}'

Remove:
oc patch cdi cdi-kubevirt-hyperconverged --type=json -p '[{ "op": "remove", "path": "/spec/config/filesystemOverhead" }]'

Comment 9 Ying Cui 2021-04-14 12:30:39 UTC
Fabien, Could we use this bug to track delivery of the size calculation in vm import controller for 2.6.2? If there is a PR, could  you attach it here?

Comment 10 Fabien Dupont 2021-04-14 12:47:05 UTC
Good idea. I've linked the PR.

Comment 11 Ilanit Stein 2021-04-16 15:27:14 UTC
@Fabien,

Can this bug move to ON_QA already?

Comment 12 Fabien Dupont 2021-04-16 16:33:05 UTC
Yes, the latest build should have the fix.

Comment 13 Ilanit Stein 2021-04-20 17:41:03 UTC
Verified on OCP-4.7.7/CNV: iib-67945 hco-v2.6.2-12

importer.log:
I0420 17:36:04.087307       1 importer.go:52] Starting importer
I0420 17:36:04.088055       1 importer.go:134] begin import process
I0420 17:36:04.917211       1 http-datasource.go:214] Attempting to get certs from /certs/ca.pem
I0420 17:36:04.943675       1 data-processor.go:357] Calculating available size
I0420 17:36:04.944865       1 data-processor.go:369] Checking out file system volume size.
I0420 17:36:04.945235       1 data-processor.go:377] Request image size not empty.
I0420 17:36:04.945248       1 data-processor.go:382] Target size 11362347520.
I0420 17:36:04.946643       1 data-processor.go:239] New phase: TransferDataFile
I0420 17:36:04.949269       1 util.go:161] Writing data...
I0420 17:36:05.948957       1 prometheus.go:69] 2.97
I0420 17:36:06.949403       1 prometheus.go:69] 5.95
I0420 17:36:07.949536       1 prometheus.go:69] 9.08
I0420 17:36:08.950759       1 prometheus.go:69] 12.35
I0420 17:36:09.951650       1 prometheus.go:69] 15.78
I0420 17:36:10.951794       1 prometheus.go:69] 19.15
I0420 17:36:11.951951       1 prometheus.go:69] 22.59
I0420 17:36:12.956927       1 prometheus.go:69] 26.09
I0420 17:36:13.957178       1 prometheus.go:69] 29.63
I0420 17:36:14.957385       1 prometheus.go:69] 33.12
I0420 17:36:15.957595       1 prometheus.go:69] 36.72
I0420 17:36:16.957777       1 prometheus.go:69] 40.31
I0420 17:36:17.958021       1 prometheus.go:69] 43.75
I0420 17:36:18.958110       1 prometheus.go:69] 47.27
I0420 17:36:19.958278       1 prometheus.go:69] 50.81
I0420 17:36:20.962524       1 prometheus.go:69] 54.19
I0420 17:36:21.962633       1 prometheus.go:69] 57.74
I0420 17:36:22.962781       1 prometheus.go:69] 61.21
I0420 17:36:23.964158       1 prometheus.go:69] 64.61
I0420 17:36:24.964298       1 prometheus.go:69] 68.05
I0420 17:36:25.964433       1 prometheus.go:69] 71.52
I0420 17:36:26.964517       1 prometheus.go:69] 74.84
I0420 17:36:27.964631       1 prometheus.go:69] 78.25
I0420 17:36:28.964746       1 prometheus.go:69] 81.61
I0420 17:36:29.969447       1 prometheus.go:69] 85.00
I0420 17:36:30.969784       1 prometheus.go:69] 88.10
I0420 17:36:31.969891       1 prometheus.go:69] 91.29
I0420 17:36:32.970111       1 prometheus.go:69] 94.22
I0420 17:36:33.970210       1 prometheus.go:69] 97.52
I0420 17:36:34.970295       1 prometheus.go:69] 100.00
I0420 17:36:37.255459       1 data-processor.go:239] New phase: Resize
W0420 17:36:38.085904       1 data-processor.go:343] Available space less than requested size, resizing image to available space 10737418240.
I0420 17:36:38.085926       1 data-processor.go:346] No need to resize image. Requested size: 11362347520, Image size: 10737418240.
I0420 17:36:38.085936       1 data-processor.go:245] Validating image
I0420 17:36:38.098500       1 data-processor.go:239] New phase: Complete
I0420 17:36:38.098581       1 importer.go:212] Import Complete

Comment 19 errata-xmlrpc 2021-05-04 20:09:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Virtualization 2.6.2 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:1502