Bug 1683516

Summary:	Cannot get 'disk_caches' value by memoryStats()
Product:	Red Hat Enterprise Linux Advanced Virtualization	Reporter:	lcheng
Component:	libvirt-python	Assignee:	Pavel Hrdina <phrdina>
Status:	CLOSED ERRATA	QA Contact:	lcheng
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	8.0	CC:	jdenemar, jsuchane, lmen, mtessun, phrdina, xuzhang, yalzhang
Target Milestone:	rc
Target Release:	8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	libvirt-python-5.6.0-2.el8	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-11-06 07:13:36 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description lcheng 2019-02-27 05:32:51 UTC

Description of problem:
Cannot get 'disk_caches' value by memoryStats().

Version-Release number of selected component (if applicable):
python3-libvirt-5.0.0-2.module+el8+2780+89be6baf.x86_64
libvirt-5.0.0-3.module+el8+2807+8bbe0a41.x86_64

How reproducible
100%

Steps to Reproduce:
1. Using virsh dommemstat command can get 'disk_caches' value.
# virsh dommemstat test
actual 2097152
swap_in 0
swap_out 0
major_fault 229
minor_fault 112754
unused 1724684
available 1873044
usable 1678040
last_update 1551160364
disk_caches 54976
rss 573904

2. Using memoryStats() API cann't get 'disk_caches' value.
# python
Python 3.6.8 (default, Jan 11 2019, 02:17:16)
[GCC 8.2.1 20180905 (Red Hat 8.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libvirt
>>> conn = libvirt.open()
>>> dom = conn.lookupByName('test')
>>> dom.memoryStats()
{'actual': 2097152, 'swap_in': 0, 'swap_out': 0, 'major_fault': 229, 'minor_fault': 112754, 'unused': 1724684, 'available': 1873044, 'usable': 1678040, 'last_update': 1551160364, 'rss': 602904}
>>> exit()


Actual results:
Step 2, cann't get 'disk_caches' value.

Expected results:
Step 2, can get 'disk_caches' value.


Additional info:
Don't find VIR_DOMAIN_MEMORY_STAT_DISK_CACHES flag in libvirt-python-5.0.0/libvirt-override.c.

static PyObject *
libvirt_virDomainMemoryStats(PyObject *self ATTRIBUTE_UNUSED,
                             PyObject *args)
{
...
    for (i = 0; i < nr_stats; i++) {
        switch (stats[i].tag) {
        case VIR_DOMAIN_MEMORY_STAT_SWAP_IN:
            key = libvirt_constcharPtrWrap("swap_in");
            break;
        case VIR_DOMAIN_MEMORY_STAT_SWAP_OUT:
            key = libvirt_constcharPtrWrap("swap_out");
            break;
        case VIR_DOMAIN_MEMORY_STAT_MAJOR_FAULT:
            key = libvirt_constcharPtrWrap("major_fault");
            break;
        case VIR_DOMAIN_MEMORY_STAT_MINOR_FAULT:
            key = libvirt_constcharPtrWrap("minor_fault");
            break;
        case VIR_DOMAIN_MEMORY_STAT_UNUSED:
            key = libvirt_constcharPtrWrap("unused");
            break;
        case VIR_DOMAIN_MEMORY_STAT_AVAILABLE:
            key = libvirt_constcharPtrWrap("available");
            break;
        case VIR_DOMAIN_MEMORY_STAT_ACTUAL_BALLOON:
            key = libvirt_constcharPtrWrap("actual");
            break;
        case VIR_DOMAIN_MEMORY_STAT_RSS:
            key = libvirt_constcharPtrWrap("rss");
            break;
#if LIBVIR_CHECK_VERSION(2, 1, 0)
        case VIR_DOMAIN_MEMORY_STAT_USABLE:
            key = libvirt_constcharPtrWrap("usable");
            break;
        case VIR_DOMAIN_MEMORY_STAT_LAST_UPDATE:
            key = libvirt_constcharPtrWrap("last_update");
            break;
#endif /* LIBVIR_CHECK_VERSION(2, 1, 0) */
        default:
            continue;
        }

Comment 1 lcheng 2019-08-19 09:16:04 UTC

Hi Pavel,

I'm testing new flags in python3-libvirt-5.6.0-1.module+el8.1.0+3891+3b51c436.x86_64. Two new flags [1] cann't get values by memoryStats(). It's similar to bug 1683516. But target release flag of bug 1683516 is 8.0. Should I open a new bug for this issue on RHEL 8.1-AV?

[1] VIR_DOMAIN_MEMORY_STAT_HUGETLB_PGALLOC = 11
VIR_DOMAIN_MEMORY_STAT_HUGETLB_PGFAIL = 12


Test steps:
1. Using virsh command can get 'hugetlb_pgalloc' and 'hugetlb_pgfail'.
# virsh dommemstat test
actual 2097152
swap_in 0
swap_out 0
major_fault 197
minor_fault 135461
unused 1705192
available 1873288
usable 1665996
last_update 1565687219
disk_caches 68528
hugetlb_pgalloc 0
hugetlb_pgfail 0
rss 764436

2. Using memoryStats() cann't get 'hugetlb_pgalloc' and 'hugetlb_pgfail'.
>>> dom.memoryStats()
{'actual': 2097152, 'swap_in': 0, 'swap_out': 0, 'major_fault': 197, 'minor_fault': 135461, 'unused': 1705192, 'available': 1873288, 'usable': 1665996, 'last_update': 1565687219, 'rss': 764500}
>>>

Comment 2 Pavel Hrdina 2019-08-19 10:40:41 UTC

Hi Liping,

No need to create a new BUG for this issue.  It can be fixed as part of this BZ which I'll move to RHEL-AV.  This BZ is not relevant to RHEL as all of the new memory stats where introduced after libvirt 4.5.0.

Comment 4 Pavel Hrdina 2019-09-20 09:20:36 UTC

There are two additional stats missing, hugetlb_pgalloc and hugetlb_pgfail which should fixed as well because they were introduced in libvirt 5.4.0.

Comment 5 Pavel Hrdina 2019-09-20 11:01:40 UTC

Upstream commits:

commit d76c27e385ad1f6ed36e17dc7e882767240f69f4
Author: Pavel Hrdina <phrdina>
Date:   Fri Sep 20 11:16:53 2019 +0200

    virDomainMemoryStats: include hugetlb pgalloc and pgfail

commit b0a7747ddd6a2066beea3a6e1162c8f959516174
Author: Pavel Hrdina <phrdina>
Date:   Fri Sep 20 11:14:35 2019 +0200

    virDomainMemoryStats: include disk caches

Comment 11 lcheng 2019-09-26 04:43:42 UTC

Verify with python3-libvirt-5.6.0-2.module+el8.1.0+4286+bc7b622c.x86_64.  

1. Check 'disk_caches', 'hugetlb_pgalloc' and 'hugetlb_pgfail'.
# python
Python 3.6.8 (default, Aug 29 2019, 22:00:26) 
[GCC 8.3.1 20190507 (Red Hat 8.3.1-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libvirt
>>> conn = libvirt.open()
>>> dom = conn.lookupByName('test')
>>> dom.memoryStats()
{'actual': 2097152, 'swap_in': 0, 'swap_out': 0, 'major_fault': 227, 'minor_fault': 127176, 'unused': 1718236, 'available': 1873288, 'usable': 1673088, 'last_update': 1569402804, 'disk_caches': 56700, 'hugetlb_pgalloc': 0, 'hugetlb_pgfail': 0, 'rss': 1027600}

2. Allocate hugepage to guest kernel.
[guest] # echo 1000 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
[guest] # cat /proc/meminfo
...
Buffers:              32 kB
Cached:            61688 kB
SwapCached:          376 kB
...
HugePages_Total:     768
HugePages_Free:      768
...

3. Check 'hugetlb_pgalloc' and 'hugetlb_pgfail' on host.
>>> dom.memoryStats()
{'actual': 2097152, 'swap_in': 312, 'swap_out': 19860, 'major_fault': 785, 'minor_fault': 874329, 'unused': 80304, 'available': 1873288, 'usable': 47856, 'last_update': 1569471940, 'disk_caches': 62100, 'hugetlb_pgalloc': 768, 'hugetlb_pgfail': 1, 'rss': 1043908}

4. Check 'disk_caches' value.
disk_caches = Buffers + Cached + SwapCached

Comment 14 errata-xmlrpc 2019-11-06 07:13:36 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3723