RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1501715 - Run 'virsh managedsave' with '--bypass-cache' flag failed.
Summary: Run 'virsh managedsave' with '--bypass-cache' flag failed.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.5
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Jiri Denemark
QA Contact: Yanqiu Zhang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-13 05:35 UTC by lcheng
Modified: 2018-04-10 11:00 UTC (History)
10 users (show)

Fixed In Version: libvirt-3.9.0-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-10 10:59:09 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
libvirtd log (3.51 MB, text/plain)
2017-10-16 03:42 UTC, lcheng
no flags Details
libvirtd log and corresponding QEMU log (1.05 MB, text/plain)
2017-10-16 08:57 UTC, lcheng
no flags Details
corresponding QEMU log (12.17 KB, text/plain)
2017-10-16 09:12 UTC, lcheng
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2018:0704 0 None None None 2018-04-10 11:00:18 UTC

Description lcheng 2017-10-13 05:35:39 UTC
Description of problem:
Run 'virsh managedsave' with '--bypass-cache' flag failed.

Version-Release number of selected component (if applicable):
Both on x86_64 and ppc64le.

libvirt-3.8.0-1.el7.x86_64
libvirt-python-3.8.0-1.el7.x86_64
kernel-3.10.0-723.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Start a guest.

2. Run 'virsh managedsave' without '--bypass-cache' flag.
# virsh managedsave virtlab_test --running

Domain virtlab_test state saved by libvirt

# virsh managedsave virtlab_test --paused

Domain virtlab_test state saved by libvirt

3. Run 'virsh managedsave' with '--bypass-cache' flag.
# virsh managedsave virtlab_test --running --bypass-cache
error: Failed to save domain virtlab_test state
error: operation failed: domain save job: unexpectedly failed

# virsh managedsave virtlab_test --paused --bypass-cache
error: Failed to save domain virtlab_test state
error: operation failed: domain save job: unexpectedly failed


Actual results:
Cannot managed save a guest state with '--bypass-cache' flag.

Expected results:
Can managed save a guest state with '--bypass-cache' flag.

Additional info:
1. The issue also exists when running another virsh cmds with '--bypass-cache'.Such as: 'virsh dump --bypass-cache'.

2. On libvirt-3.7.0-2.el7.x86_64, run managedsave command with '--bypass-cache'. The result is successful.

Comment 3 Jaroslav Suchanek 2017-10-13 14:30:45 UTC
Would you mind attaching libvirt's debug log? Thanks.

Comment 4 lcheng 2017-10-16 03:42:47 UTC
Created attachment 1339026 [details]
libvirtd log

Hi Jaroslav,

Sorry for the late reply. The attachment is following commands test result log.

# virsh managedsave test --running --bypass-cache
error: Failed to save domain test state
error: operation failed: domain save job: unexpectedly failed

# virsh managedsave test --paused --bypass-cache
error: Failed to save domain test state
error: operation failed: domain save job: unexpectedly failed

# virsh dump test /tmp/test.d --bypass-cache
error: Failed to core dump domain test to /tmp/test.d
error: operation failed: domain core dump job: unexpectedly failed

Comment 5 Jiri Denemark 2017-10-16 07:16:49 UTC
Could you also attach the corresponding QEMU log from /var/log/libvirt/qemu/test.log?

Comment 6 lcheng 2017-10-16 08:57:12 UTC
Created attachment 1339109 [details]
libvirtd log and corresponding QEMU log

Hi Jiri,

I rerun the commands. The attachments are libvirtd log and corresponding QEMU log.

Comment 7 lcheng 2017-10-16 09:12:02 UTC
Created attachment 1339110 [details]
corresponding QEMU log

Sorry, I didn't notice only one attachment can be uploaded. In comment 6, the attachment is libvirtd log. This attachment is corresponding QEMU log.

Comment 8 Jiri Denemark 2017-10-17 15:19:00 UTC
Broken by the following commit:

commit 633b699bfda06d9fcdb7f9466e2d2c9b4bc3e63c
Author: Daniel P. Berrange <berrange>
Date:   Wed Sep 20 16:25:56 2017 +0100

iohelper: avoid calling read() with misaligned buffers for O_DIRECT

The iohelper currently calls saferead() to get data from the
underlying file. This has a problem with O_DIRECT when hitting
end-of-file. saferead() is asked to read 1MB, but the first
read() it does may return only a few KB, so it'll try another
read() to fill the remaining buffer. Unfortunately the buffer
pointer passed into this 2nd read() is likely not aligned
to the extent that O_DIRECT requires, so rather than seeing
'0' for end-of-file, we'll get -1 + EINVAL due to misaligned
buffer.

The way the iohelper is currently written, it already handles
getting short reads, so there is actually no need to use
saferead() at all. We can simply call read() directly. The
benefit of this is that we can now write() the data immediately
so when we go into the subsequent reads() we'll always have a
correctly aligned buffer.

Technically the file position ought to be aligned for O_DIRECT
too, but this does not appear to matter when at end-of-file.

Tested-by: Nikolay Shirokovskiy <nshirokovskiy>
Reviewed-by: Eric Blake <eblake>
Signed-off-by: Daniel P. Berrange <berrange>

Comment 9 Jiri Denemark 2017-10-24 09:02:12 UTC
Fixed upstream by

commit 05021e727d80527c4b53debed98b87b565780a16
Refs: v3.8.0-236-g05021e727
Author:     Nikolay Shirokovskiy <nshirokovskiy>
AuthorDate: Thu Sep 28 10:06:47 2017 +0300
Commit:     Jiri Denemark <jdenemar>
CommitDate: Tue Oct 24 10:53:18 2017 +0200

    iohelper: use saferead if later write with O_DIRECT

    One of the usecases of iohelper is to read from pipe and write
    to file with O_DIRECT. As we read from pipe we can have partial
    read and then we fail to write this data because output file
    is open with O_DIRECT and buffer size is not aligned.

    Signed-off-by: Jiri Denemark <jdenemar>

Comment 12 Dan Zheng 2018-01-12 08:55:55 UTC
Please see bz1240494 comment 4. The '0600001' is expected on PPC64 as O_DIRECT will differ between x86 and PPC.

Comment 13 Yanqiu Zhang 2018-01-12 10:55:26 UTC
Verify on x86_64 with latest build:
libvirt-3.9.0-7.el7.x86_64
qemu-kvm-rhev-2.10.0-16.el7.x86_64

Steps:
1.dump
[terminal 1]# virsh dump V7.4-full-2  /tmp/V74.dump --bypass-cache
Domain V7.4-full-2 dumped to /tmp/V74.dump

[terminal 2]# while(true);do cat /proc/$(lsof -w /tmp/V74.dump|awk '/libvirt_i/{print $2}')/fdinfo/1 ;done
...
pos:    489684992
flags:    0140001
mnt_id:    63
...

2.save
# virsh save rhel7.4-2 /tmp/V74.save --bypass-cache
Domain rhel7.4-2 saved to /tmp/V74.save

# while(true);do cat /proc/$(lsof -w /tmp/V74.save|awk '/libvirt_i/{print $2}')/fdinfo/1 ;done
...
pos:    613416960
flags:    0140001
mnt_id:    63
...

3.restore
# virsh restore /tmp/V74.save --bypass-cache
Domain restored from /tmp/V74.save

# while(true);do cat /proc/$(lsof -w /tmp/V74.save|awk '/libvirt_i/{print $2}')/fdinfo/0 ;done
...
pos:    211812352
flags:    0140000
mnt_id:    63
...

4.managedsave
# virsh managedsave V7.4-full-2 --bypass-cache
Domain V7.4-full-2 state saved by libvirt

# while(true);do cat /proc/$(lsof -w /var/lib/libvirt/qemu/save/V7.4-full-2.save|awk '/libvirt_i/{print $2}')/fdinfo/1 ;done
...
pos:    441450496
flags:    0140001
mnt_id:    63
...

5.start
# virsh start V7.4-full-2 --bypass-cache
Domain V7.4-full-2 started

# while(true);do cat /proc/$(lsof -w /var/lib/libvirt/qemu/save/V7.4-full-2.save|awk '/libvirt_i/{print $2}')/fdinfo/0 ;done
...
pos:    463470592
flags:    0140000
mnt_id:    63
...

The operations are successfully executed without error, and the flags are correct.

Comment 17 errata-xmlrpc 2018-04-10 10:59:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0704


Note You need to log in before you can comment on or make changes to this bug.