Bug 1186428

Here is upstream patches about memhp + ballooning
 http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.html

Igor, does that series make it possible to balloon hotplugged memory? If it does, are you OK with this solution?

PS: I guess that's the solution for this BZ, but I still have homework to do.

(In reply to Luiz Capitulino from comment #2)
> Igor, does that series make it possible to balloon hotplugged memory? If it
> does, are you OK with this solution?
> 
> PS: I guess that's the solution for this BZ, but I still have homework to do.

I do not know if that's enough, guest side of balloon also needs to be checked to take in account possibility that memory might disappear when it's unplugged and possibly not at the end of available address space but in the middle.

Users probably will try to use ballooning to compensate for the lack of memory unplug in current QEMU.
Also it gives fine grained memory mgmt wrt memory hotplug (at least for linux where minimal increment for hotplug is128Mb, windows is fine even with 2Mb increments, so when unplug is available we potentially can discontinue ballooning drive on windows).

(In reply to Igor Mammedov from comment #3)
> (In reply to Luiz Capitulino from comment #2)
> > Igor, does that series make it possible to balloon hotplugged memory? If it
> > does, are you OK with this solution?
> > 
> > PS: I guess that's the solution for this BZ, but I still have homework to do.
> 
> I do not know if that's enough, guest side of balloon also needs to be
> checked to take in account possibility that memory might disappear when it's
> unplugged and possibly not at the end of available address space but in the
> middle.

Honest question: how does a regular driver work in this case? I mean, you want to hotunplug memory but there's a chunk of that memory in use by a driver. Does hotunplug gracefully fail?

I *guess* that the balloon driver could be considered a regular driver in the hotunplug case, because as far as the guest is considered it's just allocating memory.
 
> Users probably will try to use ballooning to compensate for the lack of
> memory unplug in current QEMU.

Do you consider this to be good or bad? I'd consider it to be good.

But to me, the most important question lurking behind all this discussion is how OpenStack is supposed to make the best use of both features.

> Also it gives fine grained memory mgmt wrt memory hotplug (at least for
> linux where minimal increment for hotplug is128Mb, windows is fine even with
> 2Mb increments, so when unplug is available we potentially can discontinue
> ballooning drive on windows).

(In reply to Luiz Capitulino from comment #4)
> (In reply to Igor Mammedov from comment #3)
> > (In reply to Luiz Capitulino from comment #2)
> > > Igor, does that series make it possible to balloon hotplugged memory? If it
> > > does, are you OK with this solution?
> > > 
> > > PS: I guess that's the solution for this BZ, but I still have homework to do.
> > 
> > I do not know if that's enough, guest side of balloon also needs to be
> > checked to take in account possibility that memory might disappear when it's
> > unplugged and possibly not at the end of available address space but in the
> > middle.
> 
> Honest question: how does a regular driver work in this case? I mean, you
> want to hotunplug memory but there's a chunk of that memory in use by a
> driver. Does hotunplug gracefully fail?
I have never had interest in how ballooning driver works, so I do not know answer.
In general linux memory unplug case though, used memory is migrated from unplugged region (region must be in ZONE_MOVABLE) to some still present available memory and only then (on success) guest will eject DIMM device.
As for windows I haven't a slightest idea about what it does before doing eject.

I don't want ballooning to break memory hotunplug. I'll take a detailed look at this situation soonish.

Posted an email upstream for discussion regarding this issue:

http://lists.nongnu.org/archive/html/qemu-devel/2015-02/msg05164.html

Basically, in order to operate correctly, the balloon device in QEMU needs to know about the current memory available to the guest. Before memory hotplug, the only memory available to the guest was the startup memory. With memory hotplug this is different as we have to account for the startup memory _and_ hotplugged memory.

This has two possible solutions, one is simple and has already been posted upstream but it's kind of incomplete. The other is more complex and requires QEMU and guest changes.

More details in my writeup, let's see what upstream says.

Peter,

I have two questions for you:

1. Just for my own education and also to have it documented, can you explain
how this issue affects libvirt support of memory hotplug?

2. Can you confirm that the series bellow does fix the problem for libvirt:

[PATCH v2 0/3] fix bug about balloon working	incorrectly when hotplug memeory
http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.html

I'm going to apply this series. However, in order to memory hotplug to work, you also need to properly configure the guest which has to be done manually for the time being if I'm not mistaken. See bug 1105020 for more information.

(In reply to Luiz Capitulino from comment #8)
> Peter,
> 
> I have two questions for you:
> 
> 1. Just for my own education and also to have it documented, can you explain
> how this issue affects libvirt support of memory hotplug?

The issue is more or less philosophical in the sense that any fix is currently OK for libvirt as long as we know what the correct and expected behavior is, so that we can either document the limitation or adjust the code.

Libvirt is checking the requested balloon pressure provided by the user, so while implementing memory hotplug I need to make sure that the check is done against the correct value. If all memory is available for ballooning I just need to tweak the check to include the hotplugged memory.

> 
> 2. Can you confirm that the series bellow does fix the problem for libvirt:
> 
> [PATCH v2 0/3] fix bug about balloon working	incorrectly when hotplug memeory
> http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.html
> 
> I'm going to apply this series. However, in order to memory hotplug to work,
> you also need to properly configure the guest which has to be done manually
> for the time being if I'm not mistaken. See bug 1105020 for more information.

I didn't try the series but looking at the code it does exactly what I expected qemu to do in the first place: To allow ballooning also the hotplugged memory.

The memory hotplug code for libvirt that is currently on review is tweaked for this approach so the series is good from my perspective.

As of enabling the hotplugged memory, I'm aware of that limitation, I was testing the code with fedora which is showing symptoms of bug 1105020.

I think that despite 1105020 the approach is correct when considering the way the current memory balloon is implemented, as the guest can arbitrarily decide not to use all memory available and behaving incorrectly when ballooning is applied even without memory hotplug, so this is not making things worse.

Thanks Peter. I've applied zhanghailiang's series to me tree and will backport it to RHEL7.2 as soon as my next pull request gets merged.

Patches are merged upstream:

git log c10b02836ff02fcd09367316260f9368c5c92f28..6adfdc5a6a4c2540a802e10fe0370d3ffb14487b

And posted downstream.

I have been asked to wait for the rebase (vs. posting the patches for review). So, setting "Fixed in Version".

Setting status to MODIFIED for the same reason.

reproduce:
host kernel:3.10.0-267.el7.x86_64
host qemu:  qemu-kvm-rhev-2.2.0-8.el7.x86_64

QE retested this bug on the above environment. After hot-plugging memory the balloon size didn't change accordingly.
So, the bug is reproduced on the above environment.


verify:
host kernel:3.10.0-267.el7.x86_64
host qemu:  qemu-kvm-rhev-2.3.0-4.el7.x86_64

QE retested this bug on the above environment. After hot-plugging memory the memory balloon take hotplugged memory into account.
So, the bug fixed.

Based on Comment 15, set this bug to VERIFIED.

Best regards,
Huiqing

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2546.html