Bug 1707372

Created attachment 1565151 [details]
host-imageio.log

Log of host obtained in /var/log/ovirt-imageio-daemon/daemon.log

Created attachment 1565152 [details]
download_disk_snaps.py

Code employed to backup

Created attachment 1565153 [details]
restore_disk.py

Code employed to restore

Created attachment 1565154 [details]
python-NFS.log

Log with a success restore with the attached code. The virtual machine's disks are allocated in NFS Storage

Created attachment 1565155 [details]
python-iSCSI.log

Log with a failed restore with the attached code. The virtual machine's disks are allocated in iSCSI Storage

*** Bug 1707365 has been marked as a duplicate of this bug. ***

Added a suggested fix to the 'upload_disk_snapshots' example in ovirt-engine-sdk:
https://gerrit.ovirt.org/#/c/100351/

The issue was the transfer_service.finalize called without waiting for it to complete
before creating the next image transfer. I.e. the fix is adding the following to
'upload_disk_snapshot' function:

    # Waiting for finalize to complete
    try:
        while transfer_service.get():
            time.sleep(1)
    except sdk.NotFoundError:
         pass

Hello Daniel,
I have tested your modification and the upload process begins. However, it fails when it passes 1Gb in the transfer. In fact the situation is exactly the same than the discussed in ticket: https://bugzilla.redhat.com/show_bug.cgi?id=1707707

I'm attaching the logs I got with my test.

The problem, as it happens with Java SDK, is RHV seems not be considering the initialSize we are indicating in the snapshot creation operation. Even if we are setting a high value, only 1Gb is assigned, so the following upload process fails after that size.

How can we solve this?

Regards,
Fran

Created attachment 1577873 [details]
Python log in fail upload diskSnapshot with more 1G size

Created attachment 1577874 [details]
Data of disks when script python failed

(In reply to francisco.garcia from comment #8)
> Hello Daniel,
> I have tested your modification and the upload process begins. However, it
> fails when it passes 1Gb in the transfer. In fact the situation is exactly
> the same than the discussed in ticket:
> https://bugzilla.redhat.com/show_bug.cgi?id=1707707
> 
> I'm attaching the logs I got with my test.
> 
> The problem, as it happens with Java SDK, is RHV seems not be considering
> the initialSize we are indicating in the snapshot creation operation. Even
> if we are setting a high value, only 1Gb is assigned, so the following
> upload process fails after that size.
> 
> How can we solve this?
> 
> Regards,
> Fran

Daniel , I see Fran tried the fix and it failed for him.
Can this bug be verified or is this depends on bug 1707707 fix?

(In reply to Avihai from comment #11)
> (In reply to francisco.garcia from comment #8)
> > Hello Daniel,
> > I have tested your modification and the upload process begins. However, it
> > fails when it passes 1Gb in the transfer. In fact the situation is exactly
> > the same than the discussed in ticket:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1707707
> > 
> > I'm attaching the logs I got with my test.
> > 
> > The problem, as it happens with Java SDK, is RHV seems not be considering
> > the initialSize we are indicating in the snapshot creation operation. Even
> > if we are setting a high value, only 1Gb is assigned, so the following
> > upload process fails after that size.
> > 
> > How can we solve this?
> > 
> > Regards,
> > Fran
> 
> Daniel , I see Fran tried the fix and it failed for him.
> Can this bug be verified or is this depends on bug 1707707 fix?

There are actually two separate issues:
1. Simultaneous uploads, on same images chain, were allowed in the engine:
not supported yet, hence as part of this BZ, it was validated in engine
and the SDK example has been fixed accordingly. I.e. it can be verified
simply by trying to simultaneously upload disk-snapshots (from the same disk).
2. For block device, the user should be able to specify a custom initial size
(to allow uploading a disk snapshot bigger than 1GiB). It would be covered
as part of bug 1707707.

(In reply to Daniel Erez from comment #12)
> (In reply to Avihai from comment #11)
> > (In reply to francisco.garcia from comment #8)
> > > Hello Daniel,
> > > I have tested your modification and the upload process begins. However, it
> > > fails when it passes 1Gb in the transfer. In fact the situation is exactly
> > > the same than the discussed in ticket:
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1707707
> > > 
> > > I'm attaching the logs I got with my test.
> > > 
> > > The problem, as it happens with Java SDK, is RHV seems not be considering
> > > the initialSize we are indicating in the snapshot creation operation. Even
> > > if we are setting a high value, only 1Gb is assigned, so the following
> > > upload process fails after that size.
> > > 
> > > How can we solve this?
> > > 
> > > Regards,
> > > Fran
> > 
> > Daniel , I see Fran tried the fix and it failed for him.
> > Can this bug be verified or is this depends on bug 1707707 fix?
> 
> There are actually two separate issues:
> 1. Simultaneous uploads, on same images chain, were allowed in the engine:
> not supported yet, hence as part of this BZ, it was validated in engine
> and the SDK example has been fixed accordingly. I.e. it can be verified
> simply by trying to simultaneously upload disk-snapshots (from the same
> disk).
> 2. For block device, the user should be able to specify a custom initial size
> (to allow uploading a disk snapshot bigger than 1GiB). It would be covered
> as part of bug 1707707.

OK to sum up:
1) This fix is to be verified by trying to simultaneously upload disk-snapshots (from the same disk).
2) Fran issue(specify a custom initial size to allow uploading a disk snapshot bigger than 1GiB) is handled at bug 1707707 and this bug fix is not dependant on it.

Please confirm

(In reply to Avihai from comment #13)
> (In reply to Daniel Erez from comment #12)
> > (In reply to Avihai from comment #11)
> > > (In reply to francisco.garcia from comment #8)
> > > > Hello Daniel,
> > > > I have tested your modification and the upload process begins. However, it
> > > > fails when it passes 1Gb in the transfer. In fact the situation is exactly
> > > > the same than the discussed in ticket:
> > > > https://bugzilla.redhat.com/show_bug.cgi?id=1707707
> > > > 
> > > > I'm attaching the logs I got with my test.
> > > > 
> > > > The problem, as it happens with Java SDK, is RHV seems not be considering
> > > > the initialSize we are indicating in the snapshot creation operation. Even
> > > > if we are setting a high value, only 1Gb is assigned, so the following
> > > > upload process fails after that size.
> > > > 
> > > > How can we solve this?
> > > > 
> > > > Regards,
> > > > Fran
> > > 
> > > Daniel , I see Fran tried the fix and it failed for him.
> > > Can this bug be verified or is this depends on bug 1707707 fix?
> > 
> > There are actually two separate issues:
> > 1. Simultaneous uploads, on same images chain, were allowed in the engine:
> > not supported yet, hence as part of this BZ, it was validated in engine
> > and the SDK example has been fixed accordingly. I.e. it can be verified
> > simply by trying to simultaneously upload disk-snapshots (from the same
> > disk).
> > 2. For block device, the user should be able to specify a custom initial size
> > (to allow uploading a disk snapshot bigger than 1GiB). It would be covered
> > as part of bug 1707707.
> 
> OK to sum up:
> 1) This fix is to be verified by trying to simultaneously upload
> disk-snapshots (from the same disk).
> 2) Fran issue(specify a custom initial size to allow uploading a disk
> snapshot bigger than 1GiB) is handled at bug 1707707 and this bug fix is not
> dependant on it.
> 
> Please confirm

Right, that's correct.

This bug cannot be verified due to the missing fix for download_disk_snapshot.py as reported in bug 1728693.

The scenario is:
1. download the VM's disk-snaphosts
2. upload the VM's disk-snapshots (restore)

currently, step 1 cannot be tested.

Currently, only file domain is fixed (partially) and there is no fix for block domain.
The issue was opened on block domain as file domain worded fine.
Therefore, I cannot verify this bug now.

(In reply to Yosi Ben Shimon from comment #16)
> Currently, only file domain is fixed (partially) and there is no fix for
> block domain.
> The issue was opened on block domain as file domain worded fine.
> Therefore, I cannot verify this bug now.

As discussed previously, the issue in block domain is covered by bug 1707707.
This bug address the validation of simultaneous uploads in the engine and the
fix for upload_disk_snapshots.py script - both issues can be verified in this version.
I suggest to fix the title of the BZ and verify accordingly. Moving back to ON_QA.

(In reply to Daniel Erez from comment #17)
> (In reply to Yosi Ben Shimon from comment #16)
> > Currently, only file domain is fixed (partially) and there is no fix for
> > block domain.
> > The issue was opened on block domain as file domain worded fine.
> > Therefore, I cannot verify this bug now.
> 
> As discussed previously, the issue in block domain is covered by bug 1707707.
> This bug address the validation of simultaneous uploads in the engine and the
> fix for upload_disk_snapshots.py script - both issues can be verified in
> this version.
> I suggest to fix the title of the BZ and verify accordingly. Moving back to
> ON_QA.

Sure NP.
Can you please add the fix for download_disk_snapshot.py as well?
That way I can verify it using the steps according to the bug description?

(In reply to Yosi Ben Shimon from comment #18)
> (In reply to Daniel Erez from comment #17)
> > (In reply to Yosi Ben Shimon from comment #16)
> > > Currently, only file domain is fixed (partially) and there is no fix for
> > > block domain.
> > > The issue was opened on block domain as file domain worded fine.
> > > Therefore, I cannot verify this bug now.
> > 
> > As discussed previously, the issue in block domain is covered by bug 1707707.
> > This bug address the validation of simultaneous uploads in the engine and the
> > fix for upload_disk_snapshots.py script - both issues can be verified in
> > this version.
> > I suggest to fix the title of the BZ and verify accordingly. Moving back to
> > ON_QA.
> 
> Sure NP.
> Can you please add the fix for download_disk_snapshot.py as well?
> That way I can verify it using the steps according to the bug description?

Sure, you can try this: https://gerrit.ovirt.org/#/c/101793/

Please retarget this bug to 4.3.6 as we are missing the MERGED code for upload/download disk snapshot in the official 4.3.5-6 build and can not verify this bug with hacks/unofficial code.

Please retarget this bug to 4.3.7 as we are missing the MERGED code for download disk snapshot in the official 4.3.6-6 build and can not verify this bug with hacks/unofficial code.

Missing patch in python-ovirt-engine-sdk4-4.3.2 :
https://gerrit.ovirt.org/gitweb?p=ovirt-engine-sdk.git;a=commit;h=2f198a09c24bcbeaaaaf8032e5e4fe28206b847c

These are the 4.3.2 ovirt-engine-sdk commits:
https://gerrit.ovirt.org/gitweb?p=ovirt-engine-sdk.git;a=shortlog;h=refs/tags/4.3.2

This was merged to master 7 weeks ago but was not merged to python-ovirt-engine-sdk4-4.3.2 which we use at 4.3.6-6.
Since this is the last build in 4.3.6 please retarget to 4.3.7.

Can you please make sure that the missing patch is merged in 4.3.7?

Daniel, can you please add the missing patch in the next python-ovirt-engine-sdk4-4.3.X build ?
We have an additional build in 4.3.6 due to blockers/depndancy issues.

If you can add this patch[1]to the next python-ovirt-engine-sdk4-4.3.X and add this to the new 4.3.6 downstream (rhv4.3.6-7) we can verify this bug in 4.3.6.


[1] https://gerrit.ovirt.org/gitweb?p=ovirt-engine-sdk.git;a=commit;h=2f198a09c24bcbeaaaaf8032e5e4fe28206b847c

I checked and both upload/download_disk_snapshots commits[1][2] which exist on sdk master branch but are missing from sdk_4.3 branch.
As these examples are critical for the customer/bug reproduction (step 1 and 2) they should be in either in this build(4.3.6) or the next one(4.3.7) for us to verify.

As was discussed yesterday, please backport upload/download_disk_snapshots commits[1][2] sdk-examples to sdk_4.3 branch.
As this bug is NOT urgent this can be done for 4.3.7 -> please retarget if this is the case.

[1] https://gerrit.ovirt.org/#/c/100351/   -> upload_disk_snapshots on master branch
[2] https://gerrit.ovirt.org/#/c/101793/   -> download disk snapshots on master branch

Tested on ovirt-engine-4.3.7.0-0.1.el7.noarch

The backup-restore flow works fine using the steps in this bug description.

Moving to VERIFIED

This bugzilla is included in oVirt 4.3.7 release, published on November 21st 2019.

Since the problem described in this bug report should be
resolved in oVirt 4.3.7 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.

Hello,

I have installed Ovirt 4.3.8 to check if this problem was fixed. I'm sorry to tell you that it hasn't been. I re-opened this bug to try fix it.

Description of the problem:
As previous case, I have this environment(ENV):
	1.- A disk with 3GB allocated in iSCSI storage.
	2.- Vm has two snapshots: the first has OS system only (Debian 10), and the second has a file with 1.5 GB.
	3.- Backup of vm with code[Last modified: Nov 7, 2017]: https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/download_vm_ovf.py
  4.- Backup of disks snapshots with code[Last modified: Jul 14, 2019]: https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/download_disk_snapshots.py. 
	5.- Restore with code [Last modified: Sep 4, 2019]: https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/upload_disk_snapshots.py

As previous case, I follow these steps:
	1.- I backup the vm with disk(ENV 1), that contains two snapshots(ENV 2), with backup code (ENV 3) and (ENV 4). It works fine.
	2.- I move the disks snapshots of the backup to local path 'disks/'.
	3.- I run restore code (ENV5) and this restore code create the vm and disk, and after it fails because the disk isn't attached to vm. Pull request: https://github.com/oVirt/ovirt-engine-sdk/pull/23
	4.- I attach the disks manually.
	5.- I modify the code so that it doesn't create either the vm or the disk. And I put manually the vmId and diskId.
	6.- I run the restore code(ENV5) again and it fails when the data uploaded of second disk snapshot reaches 1GB.

The attached files are:
	1.- Host Image-io log(/var/log/ovirt-imageio-daemon/daemon.log): imageio.log
	2.- Engine of Manager(var/log/ovirt-engine/engine.log): engine.log
	3.- Output of the first python script call(Create vm and disk): first-output.txt
	4.- Output of the second python script call(Upload disk data): second-output.txt

Version-Release number of selected component (if applicable):
Ovirt Manager Version: 4.3.8.2-1.el7

Actual results:
Errors, vm is not able to restore

Expected results:
Restore a virtual machine containing a disk with several snapshots larger than 1GB.

Created attachment 1662208 [details]
Host Image IO Log

Created attachment 1662209 [details]
[REOPENED] Python log

Created attachment 1662210 [details]
[REOPENED] Engine log

Created attachment 1662211 [details]
[REOPENED] first-output.txt

Created attachment 1662212 [details]
[REOPENED] second-output.txt

Hi Francisco,

On comment 13 Avihai mentioned that:
"Fran issue(specify a custom initial size to allow uploading a disk snapshot bigger than 1GiB) is handled at bug 1707707 and this bug fix is not dependant on it."

Therefore, I think that those updates should be on bug 1707707, and this one can be re-closed.

Also, I've noted this one:

> 5.- Restore with code [Last modified: Sep 4, 2019]:
> https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/upload_disk_snapshots.py
> 

It seems that the lastest modification date for that script is May 30, 2019.
Can you please compare it to the script you're using (Sep 4, 2019) and share the diffs?

Hello  Shani,
Sorry for delay, I was on holidays and the Spain's situation for the COVID-19 is complicated.

In this moment, the test environment is off and I can't do tests until Wednesday.

In other hand, I don't think that those updates should be on bug 1707707 because I'm using the python code example to minimize the changes and compare behaviours with the same code.

By last, it seems that I mistook the latest modification for that script, sorry. So, the latest modification was May 30, 2019.
I requested a git pull in gerrit, this is the commit:
https://gerrit.ovirt.org/gitweb?p=ovirt-engine-sdk.git;a=commit;h=a9fa1e01dc44e38fa41264c1f04d05f07c2fefa7

On Wednesday I try update the upload with initial size > 1GB.

Thanks for your time :)
Fran

(In reply to francisco.garcia from comment #38)

Thanks, Fran, for your update.

> Sorry for delay, I was on holidays and the Spain's situation for the
> COVID-19 is complicated.
That's fine, hope you're well and keep safe!

> By last, it seems that I mistook the latest modification for that script,
> sorry. So, the latest modification was May 30, 2019.
> I requested a git pull in gerrit, this is the commit:
> https://gerrit.ovirt.org/gitweb?p=ovirt-engine-sdk.git;a=commit;
> h=a9fa1e01dc44e38fa41264c1f04d05f07c2fefa7
I'll comment on the commit :)

> 
> On Wednesday I try update the upload with initial size > 1GB.
As mentioned on comment 17:
"As discussed previously, the issue in block domain is covered by bug 1707707.
This bug address the validation of simultaneous uploads in the engine and the
fix for upload_disk_snapshots.py script"

So IIUC, specify a custom initial size to allow uploading a disk snapshot bigger than 1GiB is handled at bug 1707707, and not related to this fix.

Hi Fransisco,
Any updates on this one?

Hi Shani,

No, sorry, I don't have any updates. I hope this weekend to have some.

Thanks for your time,
Fran

Hi Fran,
Any updates?

The discussion was moved to bug https://bugzilla.redhat.com/1707707.
Closing this bug as the original issue was already fixed and verified.
(https://bugzilla.redhat.com/show_bug.cgi?id=1707372#c27)