Bug 1809620

Summary:

VM loaded to 100% CPU is shown in engine with 0% CPU

Product:

[oVirt] ovirt-engine

Reporter:

Polina <pagranat>

Component:

BLL.Virt

Assignee:

Milan Zamazal <mzamazal>

Status:

CLOSED CURRENTRELEASE

QA Contact:

Polina <pagranat>

Severity:

medium

Docs Contact:

Priority:

unspecified

Version:

4.4.0

CC:

bugs, michal.skrivanek, rbarry

Target Milestone:

ovirt-4.4.1

Flags:

pm-rhel: ovirt-4.4+

Target Release:

---

Hardware:

x86_64

OS:

Linux

Whiteboard:

Fixed In Version:

rhv-4.4.1-3

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2020-07-08 08:27:28 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

Virt

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

1808940

Bug Blocks:

Attachments:

Description	Flags
logs and screenshot	none
virsh-client VM getStats vmID=""	none
logs	none

Description Polina 2020-03-03 14:35:54 UTC

Created attachment 1667201 [details]
logs and screenshot

Description of problem:
VM loaded to 100% CPU is shown in engine with 0% CPU

Version-Release number of selected component (if applicable):
http://bob-dr.lab.eng.brq.redhat.com/builds/4.4/rhv-4.4.0-23

How reproducible:100%


Steps to Reproduce:
1. Configure VM with 4 CPU (the VM is build on the base of latest-rhel-guest-image-8.1-infra) and run the VM
2. Load CPU on the VM till 100%

Actual results: on the VM the top shows that CPU is loaded but in engineUI it is shown as 0% (please see the attached screenshot)

top - 15:21:58 up 4 min,  1 user,  load average: 4.10, 1.95, 0.77
Tasks: 175 total,   5 running, 170 sleeping,   0 stopped,   0 zombie
%Cpu0  : 99.3 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.7 hi,  0.0 si,  0.0 st
%Cpu1  : 99.0 us,  0.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.7 hi,  0.0 si,  0.0 st
%Cpu2  : 99.3 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.7 hi,  0.0 si,  0.0 st
%Cpu3  : 99.3 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.7 hi,  0.0 si,  0.0 st
MiB Mem :    754.1 total,    376.4 free,    171.4 used,    206.3 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.    454.7 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                       
 1279 root      20   0  222488    236      0 R  99.3   0.0   2:45.29 sh                                                                                                                                            
 1281 root      20   0  222488    236      0 R  99.3   0.0   2:45.25 sh                                                                                                                                            
 1278 root      20   0  222488    236      0 R  99.0   0.0   2:45.31 sh                                                                                                                                            
 1280 root      20   0  222488    236      0 R  99.0   0.0   2:44.78 sh  

Expected results: the cpu load is reported to engine correctly


Additional info:

Comment 1 Polina 2020-03-03 15:14:05 UTC

Created attachment 1667210 [details]
virsh-client VM getStats vmID=""

Comment 2 Michal Skrivanek 2020-03-04 05:53:00 UTC

How long after guest is booted?

Comment 3 Polina 2020-03-04 07:49:31 UTC

I wait about 3 min till the VM has the IP, then start to load CPU.

Comment 4 Milan Zamazal 2020-03-10 12:14:18 UTC

I can't reproduce the bug. What are your libvirt and QEMU versions? And could you please provide Vdsm debug log?

Comment 5 Polina 2020-03-10 23:50:00 UTC

Created attachment 1669132 [details]
logs

[root@lynx01 qemu]# rpm -qa |grep libvirt
libvirt-admin-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-daemon-config-nwfilter-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-daemon-driver-storage-rbd-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-bash-completion-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-lock-sanlock-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-libs-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-daemon-driver-storage-gluster-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-daemon-driver-nodedev-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-daemon-driver-network-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-daemon-driver-storage-disk-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-daemon-driver-storage-logical-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-daemon-driver-storage-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-client-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-daemon-driver-storage-core-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-daemon-config-network-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-daemon-driver-storage-iscsi-direct-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-daemon-driver-storage-scsi-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-daemon-driver-secret-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-daemon-driver-interface-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-daemon-kvm-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-daemon-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-daemon-driver-storage-iscsi-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-daemon-driver-qemu-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
python3-libvirt-6.0.0-1.module+el8.2.0+5453+31b2b136.x86_64
libvirt-daemon-driver-nwfilter-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
libvirt-daemon-driver-storage-mpath-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
[root@lynx01 qemu]# 

[root@lynx01 qemu]# rpm -qa |grep qemu
qemu-kvm-block-curl-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64
qemu-kvm-core-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64
qemu-kvm-block-iscsi-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64
qemu-img-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64
qemu-kvm-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64
qemu-kvm-block-rbd-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64
ipxe-roms-qemu-20181214-5.git133f4c47.el8.noarch
qemu-kvm-common-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64
qemu-kvm-block-ssh-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64
libvirt-daemon-driver-qemu-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
qemu-kvm-block-gluster-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64

the relevant timestamp in engine.log
2020-03-11 01:42:54,229+02 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-89) [] VM 'f1e9524e-53a3-4c75-ba74-0bdc508f2c38'(golden_env_mixed_virtio_1_0) moved from 'PoweringUp' --> 'Up

Comment 6 Milan Zamazal 2020-03-11 14:52:45 UTC

We've found out with Polina that it's apparently this platform bug: https://bugzilla.redhat.com/1808940, which is fixed in systemd-239-28. libvirt couldn't access some CPU stats due to the bug with cgroups access. After upgrading to the given systemd version (and current versions of everything else), CPU usage is shown as expected.

Comment 7 Milan Zamazal 2020-04-29 12:12:51 UTC

Fixed in systemd-239-28.el8, systemd-239-29.el8 is now available in RHEL 8.2.

Comment 8 Polina 2020-06-21 18:08:57 UTC

verified on ovirt-engine-4.4.1.2-0.10.el8ev.noarch, libvirt-6.0.0-24.module+el8.2.1+6997+c666f621.x86_64, qemu-kvm-core-4.2.0-25.module+el8.2.1+6985+9fd9d514.x86_64

Comment 9 Sandro Bonazzola 2020-07-08 08:27:28 UTC

This bugzilla is included in oVirt 4.4.1 release, published on July 8th 2020.

Since the problem described in this bug report should be resolved in oVirt 4.4.1 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.