Bug 2075765

Summary:	[cgroup] Libvirt cannot operate vm control groups after restarting virtqemud
Product:	Red Hat Enterprise Linux 9	Reporter:	yisun
Component:	libvirt	Assignee:	Michal Privoznik <mprivozn>
libvirt sub component:	General	QA Contact:	yisun
Status:	CLOSED ERRATA	Docs Contact:
Severity:	medium
Priority:	medium	CC:	dzheng, jdenemar, jsuchane, lmen, mprivozn, virt-maint, xuzhang, yafu, yalzhang
Version:	9.1	Keywords:	Automation, Regression, Upstream
Target Milestone:	rc	Flags:	pm-rhel: mirror+
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	libvirt-8.3.0-1.el9	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-11-15 10:04:06 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:	8.3.0
Embargoed:

Description yisun 2022-04-15 08:17:37 UTC

Description:
[cgroup][modular_deamon] Libvirt cannot operate vm control groups after restarting virtqemud

Versions:
libvirt-8.2.0-1.el9.x86_64
systemd-250-4.el9.x86_64
kernel-5.14.0-75.el9.x86_64

How reproducible:
100%

Steps:
1. Have a running vm
[root@dell-per730-61 guest_resource_control]# virsh start avocado-vt-vm1
Domain 'avocado-vt-vm1' started

2. Run some SET/GET cgroup related virsh cmd, and nothing run
[root@dell-per730-61 guest_resource_control]# virsh memtune avocado-vt-vm1 --hard-limit=222222222

[root@dell-per730-61 guest_resource_control]# virsh memtune avocado-vt-vm1 
hard_limit     : 222222220
soft_limit     : unlimited
swap_hard_limit: unlimited

[root@dell-per730-61 guest_resource_control]# virsh blkiotune avocado-vt-vm1
weight         : 100
device_weight  : 
device_read_iops_sec: 
device_write_iops_sec: 
device_read_bytes_sec: 
device_write_bytes_sec: 

[root@dell-per730-61 guest_resource_control]# virsh schedinfo avocado-vt-vm1
Scheduler      : posix
cpu_shares     : 100
vcpu_period    : 100000
vcpu_quota     : 17592186044415
emulator_period: 100000
emulator_quota : 17592186044415
global_period  : 100000
global_quota   : 17592186044415
iothread_period: 100000
iothread_quota : 17592186044415

3. Restart virtqemud
[root@dell-per730-61 guest_resource_control]# systemctl restart virtqemud

4. Now the cgroup related virsh cmds will return errors
[root@dell-per730-61 guest_resource_control]# virsh memtune avocado-vt-vm1
error: Unable to get memory parameters
error: Requested operation is not valid: cgroup memory controller is not mounted

[root@dell-per730-61 guest_resource_control]# virsh memtune avocado-vt-vm1 --hard-limit=1111111
error: Unable to change memory parameters
error: Requested operation is not valid: cgroup memory controller is not mounted

[root@dell-per730-61 guest_resource_control]# virsh blkiotune avocado-vt-vm1
error: Unable to get blkio parameters
error: Requested operation is not valid: blkio cgroup isn't mounted

[root@dell-per730-61 guest_resource_control]# virsh schedinfo avocado-vt-vm1
Scheduler      : Unknown
error: Requested operation is not valid: cgroup CPU controller is not mounted

5. The cgroup dirs/files are actually still OK now
[root@dell-per730-61 guest_resource_control]# mount |grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate,memory_recursiveprot)

[root@dell-per730-61 guest_resource_control]# cat /sys/fs/cgroup/machine.slice/machine-qemu\\x2d2\\x2davocado\\x2dvt\\x2dvm1.scope/libvirt/memory.max 
227555553280

Expected result:
Libvirt should be able to operate vm's cgroup params after restarting virtqemud

Comment 2 Michal Privoznik 2022-04-19 15:28:15 UTC

In fact, there's nothing split daemon specific about this. It's a pure typo that reversed one condition during code refactor. Patch posted here:

https://listman.redhat.com/archives/libvir-list/2022-April/230167.html

Comment 3 Michal Privoznik 2022-04-20 07:56:28 UTC

Merged upstream as:

commit eac8de54a68725402ec4785888dddad656db609d
Author:     Michal Prívozník <mprivozn>
AuthorDate: Tue Apr 19 17:22:22 2022 +0200
Commit:     Michal Prívozník <mprivozn>
CommitDate: Wed Apr 20 09:52:56 2022 +0200

    domain_cgroup: Fix a condition in virDomainCgroupConnectCgroup()
    
    While parts of QEMU's CGroup code were moved under hypervisor
    agnostic location (src/hypervisor/) a typo sneaked in. The
    inspiration for virDomainCgroupConnectCgroup() comes from
    qemuConnectCgroup(). The former is called upon reconnecting to a
    running domain (after daemon restart). While the latter returned
    early if the daemon was running unprivileged, the former returns
    early if the daemon runs privileged. This is obviously wrong,
    because root can set up CGroups.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2075765
    Fixes: 788e2b58cb1896f1c25ebbdbde4bafddc5ed4dc9
    Signed-off-by: Michal Privoznik <mprivozn>
    Reviewed-by: Ján Tomko <jtomko>

v8.2.0-193-geac8de54a6

Comment 4 Michal Privoznik 2022-04-20 08:07:21 UTC

*** Bug 2073905 has been marked as a duplicate of this bug. ***

Comment 8 yisun 2022-04-22 12:33:25 UTC

Previrified with:
[root@yisun-mar-fed x86_64]# rpm -qa | grep libvirt-8
libvirt-8.3.0-1.fc34.x86_64

Steps:
1. enable modular daemon manually and check cgroup mounted
#systemctl stop libvirtd.service
#systemctl stop libvirtd{,-ro,-admin,-tcp,-tls}.socket
#systemctl start virtqemud
#systemctl start virtproxyd

[root@yisun-mar-fed x86_64]# mount | grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate,memory_recursiveprot)

2. have a running vm
[root@yisun-mar-fed x86_64]# virsh domstate lmn
running

3. set vm's mem hard limit
[root@yisun-mar-fed x86_64]# virsh memtune lmn --hard-limit=222222222

[root@yisun-mar-fed x86_64]# virsh memtune lmn 
hard_limit     : 222222220
soft_limit     : unlimited
swap_hard_limit: unlimited

4. restart virtqmeud
[root@yisun-mar-fed x86_64]# systemctl restart virtqemud
[root@yisun-mar-fed x86_64]# systemctl status virtqemud
● virtqemud.service - Virtualization qemu daemon
     Loaded: loaded (/usr/lib/systemd/system/virtqemud.service; disabled; vendor preset: disabled)
     Active: active (running) since Fri 2022-04-22 12:29:26 UTC; 9s ago
TriggeredBy: ● virtqemud.socket
             ● virtqemud-admin.socket
             ● virtqemud-ro.socket
       Docs: man:virtqemud(8)
             https://libvirt.org
   Main PID: 251218 (virtqemud)
      Tasks: 19 (limit: 32768)
     Memory: 8.6M
        CPU: 795ms
     CGroup: /system.slice/virtqemud.service
             └─251218 /usr/sbin/virtqemud --timeout 120

Apr 22 12:29:26 yisun-mar-fed systemd[1]: Starting Virtualization qemu daemon...
Apr 22 12:29:26 yisun-mar-fed systemd[1]: Started Virtualization qemu daemon.

5. check the memtune taking effect
[root@yisun-mar-fed x86_64]# virsh memtune lmn 
hard_limit     : 222222220
soft_limit     : unlimited
swap_hard_limit: unlimited

Comment 11 Ján Tomko 2022-04-26 08:58:18 UTC

*** Bug 2078791 has been marked as a duplicate of this bug. ***

Comment 12 yisun 2022-05-09 06:10:01 UTC

test result passed with libvirt-8.3.0-1.el9.x86_64

Comment 16 errata-xmlrpc 2022-11-15 10:04:06 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: libvirt security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8003