Bug 1186428 - memory balloon doesn't take hotplugged memory into account
Summary: memory balloon doesn't take hotplugged memory into account
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Luiz Capitulino
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 790583
TreeView+ depends on / blocked
 
Reported: 2015-01-27 16:20 UTC by Peter Krempa
Modified: 2015-12-04 16:26 UTC (History)
8 users (show)

Fixed In Version: QEMU2.3
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-12-04 16:26:13 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2546 normal SHIPPED_LIVE qemu-kvm-rhev bug fix and enhancement update 2015-12-04 21:11:56 UTC

Description Peter Krempa 2015-01-27 16:20:38 UTC
Description of problem:
After memory is hotplugged to the guest (or the guest is started with a pc-dimm device added on the commandline) the memory added to the guest OS by the pc-dimm device is not included in the total memory size known by the balloon and thus can't be taken away.

Additionally if the balloon is used in such configuration the guest balloon driver actually uses the memory regions added by the pc-dimm device to take away the memory.

Version-Release number of selected component (if applicable):


How reproducible:
always

Steps to Reproduce:
1. start qemu with a pc-dimm device (or add it later via hotplug) and a memballoon
2. use query-balloon and check the size returned

Actual results:
qemu allows to modify the balloon only to the limit of the initial memory.


Expected results:
One of:
1) the size is updated, and the guest driver notified
2) the memory balloon guest driver operates ONLY on the memory region of the initial memory
3) usage of memballoon and memory hotplug together is forbidden at start of the qemu process. (of course this is the least favorable option)


Additional info:
As the balloon info calculation are based on the "ram_size" global variable and the device itself doesn't seem to actually know the memory size the easiest fix will be either to update "ram_size" to the total memory including the hotplugged or introducing a similar variable that will hold the total memory size.

Comment 1 Igor Mammedov 2015-02-03 10:29:08 UTC
Here is upstream patches about memhp + ballooning
 http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.html

Comment 2 Luiz Capitulino 2015-02-03 22:30:44 UTC
Igor, does that series make it possible to balloon hotplugged memory? If it does, are you OK with this solution?

PS: I guess that's the solution for this BZ, but I still have homework to do.

Comment 3 Igor Mammedov 2015-02-04 07:40:54 UTC
(In reply to Luiz Capitulino from comment #2)
> Igor, does that series make it possible to balloon hotplugged memory? If it
> does, are you OK with this solution?
> 
> PS: I guess that's the solution for this BZ, but I still have homework to do.

I do not know if that's enough, guest side of balloon also needs to be checked to take in account possibility that memory might disappear when it's unplugged and possibly not at the end of available address space but in the middle.

Users probably will try to use ballooning to compensate for the lack of memory unplug in current QEMU.
Also it gives fine grained memory mgmt wrt memory hotplug (at least for linux where minimal increment for hotplug is128Mb, windows is fine even with 2Mb increments, so when unplug is available we potentially can discontinue ballooning drive on windows).

Comment 4 Luiz Capitulino 2015-02-04 14:42:06 UTC
(In reply to Igor Mammedov from comment #3)
> (In reply to Luiz Capitulino from comment #2)
> > Igor, does that series make it possible to balloon hotplugged memory? If it
> > does, are you OK with this solution?
> > 
> > PS: I guess that's the solution for this BZ, but I still have homework to do.
> 
> I do not know if that's enough, guest side of balloon also needs to be
> checked to take in account possibility that memory might disappear when it's
> unplugged and possibly not at the end of available address space but in the
> middle.

Honest question: how does a regular driver work in this case? I mean, you want to hotunplug memory but there's a chunk of that memory in use by a driver. Does hotunplug gracefully fail?

I *guess* that the balloon driver could be considered a regular driver in the hotunplug case, because as far as the guest is considered it's just allocating memory.
 
> Users probably will try to use ballooning to compensate for the lack of
> memory unplug in current QEMU.

Do you consider this to be good or bad? I'd consider it to be good.

But to me, the most important question lurking behind all this discussion is how OpenStack is supposed to make the best use of both features.

> Also it gives fine grained memory mgmt wrt memory hotplug (at least for
> linux where minimal increment for hotplug is128Mb, windows is fine even with
> 2Mb increments, so when unplug is available we potentially can discontinue
> ballooning drive on windows).

Comment 5 Igor Mammedov 2015-02-04 15:45:01 UTC
(In reply to Luiz Capitulino from comment #4)
> (In reply to Igor Mammedov from comment #3)
> > (In reply to Luiz Capitulino from comment #2)
> > > Igor, does that series make it possible to balloon hotplugged memory? If it
> > > does, are you OK with this solution?
> > > 
> > > PS: I guess that's the solution for this BZ, but I still have homework to do.
> > 
> > I do not know if that's enough, guest side of balloon also needs to be
> > checked to take in account possibility that memory might disappear when it's
> > unplugged and possibly not at the end of available address space but in the
> > middle.
> 
> Honest question: how does a regular driver work in this case? I mean, you
> want to hotunplug memory but there's a chunk of that memory in use by a
> driver. Does hotunplug gracefully fail?
I have never had interest in how ballooning driver works, so I do not know answer.
In general linux memory unplug case though, used memory is migrated from unplugged region (region must be in ZONE_MOVABLE) to some still present available memory and only then (on success) guest will eject DIMM device.
As for windows I haven't a slightest idea about what it does before doing eject.

Comment 6 Luiz Capitulino 2015-02-04 18:12:04 UTC
I don't want ballooning to break memory hotunplug. I'll take a detailed look at this situation soonish.

Comment 7 Luiz Capitulino 2015-02-26 20:32:13 UTC
Posted an email upstream for discussion regarding this issue:

http://lists.nongnu.org/archive/html/qemu-devel/2015-02/msg05164.html

Basically, in order to operate correctly, the balloon device in QEMU needs to know about the current memory available to the guest. Before memory hotplug, the only memory available to the guest was the startup memory. With memory hotplug this is different as we have to account for the startup memory _and_ hotplugged memory.

This has two possible solutions, one is simple and has already been posted upstream but it's kind of incomplete. The other is more complex and requires QEMU and guest changes.

More details in my writeup, let's see what upstream says.

Comment 8 Luiz Capitulino 2015-03-02 19:21:10 UTC
Peter,

I have two questions for you:

1. Just for my own education and also to have it documented, can you explain
how this issue affects libvirt support of memory hotplug?

2. Can you confirm that the series bellow does fix the problem for libvirt:

[PATCH v2 0/3] fix bug about balloon working	incorrectly when hotplug memeory
http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.html

I'm going to apply this series. However, in order to memory hotplug to work, you also need to properly configure the guest which has to be done manually for the time being if I'm not mistaken. See bug 1105020 for more information.

Comment 9 Peter Krempa 2015-03-03 09:31:09 UTC
(In reply to Luiz Capitulino from comment #8)
> Peter,
> 
> I have two questions for you:
> 
> 1. Just for my own education and also to have it documented, can you explain
> how this issue affects libvirt support of memory hotplug?

The issue is more or less philosophical in the sense that any fix is currently OK for libvirt as long as we know what the correct and expected behavior is, so that we can either document the limitation or adjust the code.

Libvirt is checking the requested balloon pressure provided by the user, so while implementing memory hotplug I need to make sure that the check is done against the correct value. If all memory is available for ballooning I just need to tweak the check to include the hotplugged memory.

> 
> 2. Can you confirm that the series bellow does fix the problem for libvirt:
> 
> [PATCH v2 0/3] fix bug about balloon working	incorrectly when hotplug memeory
> http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.html
> 
> I'm going to apply this series. However, in order to memory hotplug to work,
> you also need to properly configure the guest which has to be done manually
> for the time being if I'm not mistaken. See bug 1105020 for more information.

I didn't try the series but looking at the code it does exactly what I expected qemu to do in the first place: To allow ballooning also the hotplugged memory.

The memory hotplug code for libvirt that is currently on review is tweaked for this approach so the series is good from my perspective.

As of enabling the hotplugged memory, I'm aware of that limitation, I was testing the code with fedora which is showing symptoms of bug 1105020.

I think that despite 1105020 the approach is correct when considering the way the current memory balloon is implemented, as the guest can arbitrarily decide not to use all memory available and behaving incorrectly when ballooning is applied even without memory hotplug, so this is not making things worse.

Comment 10 Luiz Capitulino 2015-03-03 14:02:02 UTC
Thanks Peter. I've applied zhanghailiang's series to me tree and will backport it to RHEL7.2 as soon as my next pull request gets merged.

Comment 11 Luiz Capitulino 2015-03-09 16:11:51 UTC
Patches are merged upstream:

git log c10b02836ff02fcd09367316260f9368c5c92f28..6adfdc5a6a4c2540a802e10fe0370d3ffb14487b

And posted downstream.

Comment 12 Luiz Capitulino 2015-03-10 17:51:26 UTC
I have been asked to wait for the rebase (vs. posting the patches for review). So, setting "Fixed in Version".

Comment 13 Jeff Nelson 2015-03-10 19:52:14 UTC
Setting status to MODIFIED for the same reason.

Comment 15 Lin Chen 2015-06-23 08:02:25 UTC
reproduce:
host kernel:3.10.0-267.el7.x86_64
host qemu:  qemu-kvm-rhev-2.2.0-8.el7.x86_64

QE retested this bug on the above environment. After hot-plugging memory the balloon size didn't change accordingly.
So, the bug is reproduced on the above environment.


verify:
host kernel:3.10.0-267.el7.x86_64
host qemu:  qemu-kvm-rhev-2.3.0-4.el7.x86_64

QE retested this bug on the above environment. After hot-plugging memory the memory balloon take hotplugged memory into account.
So, the bug fixed.

Comment 16 huiqingding 2015-06-24 03:45:30 UTC
Based on Comment 15, set this bug to VERIFIED.

Best regards,
Huiqing

Comment 18 errata-xmlrpc 2015-12-04 16:26:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2546.html


Note You need to log in before you can comment on or make changes to this bug.