Bug 1300781 - Incorrect memory virtualization in lxc driver
Summary: Incorrect memory virtualization in lxc driver
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: libvirt
Version: 23
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Libvirt Maintainers
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-01-21 17:06 UTC by Avi Kivity
Modified: 2016-03-22 19:53 UTC (History)
13 users (show)

Fixed In Version: libvirt-1.2.18.2-3.fc23
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-03-22 19:53:14 UTC
Type: Bug


Attachments (Terms of Use)

Description Avi Kivity 2016-01-21 17:06:18 UTC
Description of problem:

/proc/meminfo virtualization has incorrect values, leading applications that depend on it to behave incorrectly.


Version-Release number of selected component (if applicable):
libvirt-1.2.18.2-1.fc23.x86_64

How reproducible:

Always

Steps to Reproduce:
1. Run an lxc container with limited memory
2. run 'free' or 'cat /proc/meminfo'

Actual results:

$ virt-install --connect lxc:/// --name fedora23 --memory 2048

Starting install...
Creating domain...                                                                                                                                                                          |    0 B  00:00:00     
Connected to domain fedora23
Escape character is ^]
sh-4.3# ls
bin   dev  home  lib64	     media  opt   root	sbin  sys  usr
boot  etc  lib	 lost+found  mnt    proc  run	srv   tmp  var
sh-4.3# free
              total        used        free      shared  buff/cache   available
Mem:        2097152    -1159568     2096076      956936     1160644    20559700
Swap:      10485756      308704    10177052
sh-4.3# cat /proc/meminfo 
MemTotal:        2097152 kB
MemFree:         2096240 kB
MemAvailable:   20448244 kB
Buffers:               0 kB
Cached:               20 kB
SwapCached:        12436 kB
Active:              652 kB
Inactive:             16 kB
Active(anon):        648 kB
Inactive(anon):        0 kB
Active(file):          4 kB
Inactive(file):       16 kB
Unevictable:           0 kB
Mlocked:           24588 kB
SwapTotal:      10485756 kB
SwapFree:       10177052 kB
Dirty:               120 kB
Writeback:             0 kB
AnonPages:      10714672 kB
Mapped:           745532 kB
Shmem:            939428 kB
Slab:            1160820 kB
SReclaimable:    1010288 kB
SUnreclaim:       150532 kB
KernelStack:       18384 kB
PageTables:       113388 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    26906196 kB
Committed_AS:   20654132 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      435904 kB
VmallocChunk:   34358947836 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:     1329840 kB
DirectMap2M:    32116736 kB
sh-4.3# 

Note free's "used" output is negative, and meminfo's MemAvailable is greater than MemTotal and MemFree.


Expected results:

Something sane

Additional info:

Comment 1 Cole Robinson 2016-01-21 18:43:40 UTC
Thanks for the report Avi, patches posted:

https://www.redhat.com/archives/libvir-list/2016-January/msg00928.html

Comment 2 Avi Kivity 2016-01-21 19:24:08 UTC
Cool, thanks for the quick response. I believe that RHEL suffers from something similar.

Comment 3 Avi Kivity 2016-01-24 13:03:22 UTC
The patches fix the problem for me.

Comment 4 Avi Kivity 2016-01-24 14:46:01 UTC
Strangely, the fix works, then stops working:

-bash-4.3$ free
              total        used        free      shared  buff/cache   available
Mem:        9097152      729024     8082584        4512      285544     8082584
Swap:       8167420           0     8167420
-bash-4.3$ 
-bash-4.3$ free
              total        used        free      shared  buff/cache   available
Mem:    9007199254740991         116 9007199254740875        3972           0 9007199254740875
Swap:       8167420       15348     8152072
-bash-4.3$ cat /proc/meminfo 
MemTotal:       9007199254740991 kB
MemFree:        9007199254740875 kB
MemAvailable:   9007199254740875 kB
Buffers:               0 kB

Some time passed between the two free invocations, but the guest was up the whole time (indeed it's the same ssh session).  So something fishy is going on/

Comment 5 Cole Robinson 2016-01-25 14:33:20 UTC
Hmm. Those patches don't touch the MemTotal/MemFree generation, so probably a pre-existing bug. We pull values from cgroups, so maybe something is getting out of sync

Comment 6 Cole Robinson 2016-03-17 20:09:18 UTC
I'll use this bug to track backporting the upstream patches. If you hit the other issue again please open a separate bug

Comment 7 Fedora Update System 2016-03-17 22:29:56 UTC
libvirt-1.2.18.2-3.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-399269e155

Comment 8 Fedora Update System 2016-03-19 01:23:48 UTC
libvirt-1.2.18.2-3.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-399269e155

Comment 9 Fedora Update System 2016-03-22 19:52:45 UTC
libvirt-1.2.18.2-3.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.