Bug 1157264 - Allow qemu-img to bypass the host cache (check, compare, convert, rebase, amend)
Summary: Allow qemu-img to bypass the host cache (check, compare, convert, rebase, amend)
Keywords:
Status: CLOSED DUPLICATE of bug 1138690
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 3.5.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ovirt-3.6.3
: 3.6.0
Assignee: Federico Simoncelli
QA Contact: Aharon Canan
URL:
Whiteboard: storage
Depends On: 1116558
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-10-26 19:10 UTC by Federico Simoncelli
Modified: 2016-03-10 06:24 UTC (History)
29 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of: 1116558
Environment:
Last Closed: 2014-12-22 18:42:46 UTC
oVirt Team: Storage
Target Upstream Version:


Attachments (Terms of Use)

Description Federico Simoncelli 2014-10-26 19:10:58 UTC
+++ This bug was initially created as a clone of Bug #1116558 +++

Description of problem:

During snapshot deletion (on a NFS datacenter share) vdsm on SPM issues qemu-img command to handle merging of disk images. This process reads data using the hypervisor page cache. As this data might be hotter than any VM memory the system starts swapping out memory pages of the running VMs. 

With each VM disk being only accessed once during that process the system should avoid using the page cache.

- Either by accessing the files with direct I/O
- Or by clearing the page cache in regular intervals during that operation

Version-Release number of selected component (if applicable):

Ovirt 3.4.2
Fedora 20 hypervisor node (kernel 3.14.8-200.fc20.x86_64)
vdsm-4.14.9-0.fc20

How reproducible:

100%


Steps to Reproduce:
1. Choose OVirt NFS based datacenter
1. Chose VM with a disk that is larger than the free memory on SPM node
2. Stop VM
3. Create snapshot of the large disk (takes a few seconds)
4. Delete snapshot (takes long)
5. watch sapping on SPM node

Actual results:

- page cache usage on SPM increases
- memory fills up to 100%
- paging swaps out memory of VMs

Expected results:

- No paging should occur

Additional info:

- This behaviour can be seen even with swappiness set to 0
- Graphs attached

Comment 1 Federico Simoncelli 2014-10-26 19:23:45 UTC
I am setting the component of this bz to ovirt-engine-backend because if I am not mistaken at the oVirt workshop Markus Stockhausen pointed out that the VMs were migrated out of the host even before memory swapping.

If that's really the case then there's probably an issue on the engine side where we consider cached memory as used and unreclaimable memory.

If that's not the case then we should switch this bz to vdsm to take advantage of the new qemu-img option to use O_DIRECT on read as well.
(We may want to do this regardless but let's wait for feedback first)

Markus can you please clarify what is the exact behavior? Are the VMs migrated out before host starts swapping? Thanks.

Comment 2 Markus Stockhausen 2014-10-26 20:23:26 UTC
Hello,

this is another bug about heavy swapping. What you are referring to (and what I tried to explain at the OVirt workshop), is the the swapping during online migration:

https://bugzilla.redhat.com/show_bug.cgi?id=1153765

But back to this one: The swapping starts during run of qemu-img commands. The reason for this all was that qemu reads thorugh page cache for the above mentioned commands. In between there has been a bugfix for that behaviour. qemu can be passed a parameter to avoid page cache. More details here: 

http://git.qemu.org/?p=qemu.git;a=commit;h=40055951a7afbfc037c6c7351d72a5c5d83ed99b

This parameter should be set from OVirt/RHEV if it finds a patched qemu version.

Comment 3 Federico Simoncelli 2014-10-27 22:07:20 UTC
Thanks for the clarification Markus. Moving to vdsm per comment 2.

Comment 5 Nir Soffer 2014-12-20 20:38:44 UTC
We already have a vdsm bug 1138690 - I think we should close this as duplicate.

Comment 6 Federico Simoncelli 2014-12-22 18:42:46 UTC

*** This bug has been marked as a duplicate of bug 1138690 ***


Note You need to log in before you can comment on or make changes to this bug.