Bug 1300781

Summary: Incorrect memory virtualization in lxc driver
Product: [Fedora] Fedora Reporter: Avi Kivity <avi.kivity>
Component: libvirtAssignee: Libvirt Maintainers <libvirt-maint>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 23CC: agedosier, berrange, clalancette, crobinso, dyuan, hhan, itamar, jforbes, laine, libvirt-maint, veillard, virt-maint, xuzhang
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-1.2.18.2-3.fc23 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-03-22 19:53:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Avi Kivity 2016-01-21 17:06:18 UTC
Description of problem:

/proc/meminfo virtualization has incorrect values, leading applications that depend on it to behave incorrectly.


Version-Release number of selected component (if applicable):
libvirt-1.2.18.2-1.fc23.x86_64

How reproducible:

Always

Steps to Reproduce:
1. Run an lxc container with limited memory
2. run 'free' or 'cat /proc/meminfo'

Actual results:

$ virt-install --connect lxc:/// --name fedora23 --memory 2048

Starting install...
Creating domain...                                                                                                                                                                          |    0 B  00:00:00     
Connected to domain fedora23
Escape character is ^]
sh-4.3# ls
bin   dev  home  lib64	     media  opt   root	sbin  sys  usr
boot  etc  lib	 lost+found  mnt    proc  run	srv   tmp  var
sh-4.3# free
              total        used        free      shared  buff/cache   available
Mem:        2097152    -1159568     2096076      956936     1160644    20559700
Swap:      10485756      308704    10177052
sh-4.3# cat /proc/meminfo 
MemTotal:        2097152 kB
MemFree:         2096240 kB
MemAvailable:   20448244 kB
Buffers:               0 kB
Cached:               20 kB
SwapCached:        12436 kB
Active:              652 kB
Inactive:             16 kB
Active(anon):        648 kB
Inactive(anon):        0 kB
Active(file):          4 kB
Inactive(file):       16 kB
Unevictable:           0 kB
Mlocked:           24588 kB
SwapTotal:      10485756 kB
SwapFree:       10177052 kB
Dirty:               120 kB
Writeback:             0 kB
AnonPages:      10714672 kB
Mapped:           745532 kB
Shmem:            939428 kB
Slab:            1160820 kB
SReclaimable:    1010288 kB
SUnreclaim:       150532 kB
KernelStack:       18384 kB
PageTables:       113388 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    26906196 kB
Committed_AS:   20654132 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      435904 kB
VmallocChunk:   34358947836 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:     1329840 kB
DirectMap2M:    32116736 kB
sh-4.3# 

Note free's "used" output is negative, and meminfo's MemAvailable is greater than MemTotal and MemFree.


Expected results:

Something sane

Additional info:

Comment 1 Cole Robinson 2016-01-21 18:43:40 UTC
Thanks for the report Avi, patches posted:

https://www.redhat.com/archives/libvir-list/2016-January/msg00928.html

Comment 2 Avi Kivity 2016-01-21 19:24:08 UTC
Cool, thanks for the quick response. I believe that RHEL suffers from something similar.

Comment 3 Avi Kivity 2016-01-24 13:03:22 UTC
The patches fix the problem for me.

Comment 4 Avi Kivity 2016-01-24 14:46:01 UTC
Strangely, the fix works, then stops working:

-bash-4.3$ free
              total        used        free      shared  buff/cache   available
Mem:        9097152      729024     8082584        4512      285544     8082584
Swap:       8167420           0     8167420
-bash-4.3$ 
-bash-4.3$ free
              total        used        free      shared  buff/cache   available
Mem:    9007199254740991         116 9007199254740875        3972           0 9007199254740875
Swap:       8167420       15348     8152072
-bash-4.3$ cat /proc/meminfo 
MemTotal:       9007199254740991 kB
MemFree:        9007199254740875 kB
MemAvailable:   9007199254740875 kB
Buffers:               0 kB

Some time passed between the two free invocations, but the guest was up the whole time (indeed it's the same ssh session).  So something fishy is going on/

Comment 5 Cole Robinson 2016-01-25 14:33:20 UTC
Hmm. Those patches don't touch the MemTotal/MemFree generation, so probably a pre-existing bug. We pull values from cgroups, so maybe something is getting out of sync

Comment 6 Cole Robinson 2016-03-17 20:09:18 UTC
I'll use this bug to track backporting the upstream patches. If you hit the other issue again please open a separate bug

Comment 7 Fedora Update System 2016-03-17 22:29:56 UTC
libvirt-1.2.18.2-3.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-399269e155

Comment 8 Fedora Update System 2016-03-19 01:23:48 UTC
libvirt-1.2.18.2-3.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-399269e155

Comment 9 Fedora Update System 2016-03-22 19:52:45 UTC
libvirt-1.2.18.2-3.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.