Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1810356

Summary:	[cgroup2] When detach a device from vm, the device entry not removed from vm's bpf map
Product:	Red Hat Enterprise Linux Advanced Virtualization	Reporter:	yisun
Component:	libvirt	Assignee:	Pavel Hrdina <phrdina>
Status:	CLOSED ERRATA	QA Contact:	yisun
Severity:	low	Docs Contact:
Priority:	low
Version:	8.2	CC:	jdenemar, jsuchane, lmen, virt-maint, xuzhang, yisun
Target Milestone:	rc	Keywords:	Triaged
Target Release:	8.0	Flags:	pm-rhel: mirror+
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	libvirt-7.0.0-1.el8	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-05-25 06:41:21 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:	7.0.0
Embargoed:

Description yisun 2020-03-05 04:06:00 UTC

Description of problem:
[cgroup2] When detach a device from vm, the device entry not removed from vm's bpf map

Version-Release number of selected component (if applicable):
libvirt-6.0.0-7.module+el8.2.0+5869+c23fe68b.x86_64
systemd-239-23.el8.x86_64

How reproducible:
100%

Steps to Reproduce:
0. Enable cgorup v2
1.
[root@hp-dl320eg8-05 bz1717396]# virsh start vm1
Domain vm1 started

[root@hp-dl320eg8-05 bz1717396]# bpftool cgroup list /sys/fs/cgroup/machine.slice/machine-qemu\\x2d2\\x2dvm1.scope/
ID       AttachType      AttachFlags     Name
18       device

[root@hp-dl320eg8-05 bz1717396]# bpftool prog show id 18
18: cgroup_device  tag b95e404d31962705  gpl
	loaded_at 2020-03-03T23:33:56-0500  uid 0
	xlated 616B  jited 350B  memlock 4096B  map_ids 18

[root@hp-dl320eg8-05 bz1717396]# bpftool map dump id 18
key: 07 00 00 00 01 00 00 00  value: 02 00 06 00
key: ff ff ff ff 88 00 00 00  value: 02 00 06 00
key: 03 00 00 00 01 00 00 00  value: 02 00 06 00
key: 02 00 00 00 05 00 00 00  value: 02 00 06 00
key: 09 00 00 00 01 00 00 00  value: 02 00 06 00
key: 00 00 00 00 fb 00 00 00  value: 02 00 06 00
key: 08 00 00 00 01 00 00 00  value: 02 00 06 00
key: 90 00 00 00 41 00 00 00  value: 01 00 06 00
key: 05 00 00 00 01 00 00 00  value: 02 00 06 00
key: e8 00 00 00 0a 00 00 00  value: 02 00 06 00
Found 10 elements

[root@hp-dl320eg8-05 bz1717396]# cat disk.xml
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/sdbb'/>
      <target dev='vdbd' bus='virtio'/>
    </disk>
[root@hp-dl320eg8-05 bz1717396]# virsh attach-device vm1 disk.xml
Device attached successfully

[root@hp-dl320eg8-05 bz1717396]# bpftool map dump id 18
key: 07 00 00 00 01 00 00 00  value: 02 00 06 00
key: ff ff ff ff 88 00 00 00  value: 02 00 06 00
key: 03 00 00 00 01 00 00 00  value: 02 00 06 00
key: 02 00 00 00 05 00 00 00  value: 02 00 06 00
key: 09 00 00 00 01 00 00 00  value: 02 00 06 00
key: 00 00 00 00 fb 00 00 00  value: 02 00 06 00
key: 08 00 00 00 01 00 00 00  value: 02 00 06 00
key: 90 00 00 00 41 00 00 00  value: 01 00 06 00
key: 50 00 00 00 43 00 00 00  value: 01 00 06 00
key: 05 00 00 00 01 00 00 00  value: 02 00 06 00
key: e8 00 00 00 0a 00 00 00  value: 02 00 06 00
Found 11 elements
<======  NEW ENTRY IS - key: 50 00 00 00 43 00 00 00  value: 01 00 06 00

[root@hp-dl320eg8-05 bz1717396]# python get_bpf_map.py /sys/fs/cgroup/machine.slice/machine-qemu\\x2d2\\x2dvm1.scope/
c 1:7 rw		full
c 136:* rw
c 1:3 rw		null
c 5:2 rw		ptmx
c 1:9 rw		urandom
c 251:0 rw		rtc0
c 1:8 rw		random
b 65:144 rw		sdz
b 67:80 rw		sdbb
c 1:5 rw		zero
c 10:232 rw		kvm

[root@hp-dl320eg8-05 bz1717396]# virsh detach-device vm1 disk.xml
Device detached successfully

[root@hp-dl320eg8-05 bz1717396]# bpftool map dump id 18
key: 07 00 00 00 01 00 00 00  value: 02 00 06 00
key: ff ff ff ff 88 00 00 00  value: 02 00 06 00
key: 03 00 00 00 01 00 00 00  value: 02 00 06 00
key: 02 00 00 00 05 00 00 00  value: 02 00 06 00
key: 09 00 00 00 01 00 00 00  value: 02 00 06 00
key: 00 00 00 00 fb 00 00 00  value: 02 00 06 00
key: 08 00 00 00 01 00 00 00  value: 02 00 06 00
key: 90 00 00 00 41 00 00 00  value: 01 00 06 00
key: 50 00 00 00 43 00 00 00  value: 00 00 00 00
key: 05 00 00 00 01 00 00 00  value: 02 00 06 00
key: e8 00 00 00 0a 00 00 00  value: 02 00 06 00
Found 11 elements
<===== Not back to 10 elements

[root@hp-dl320eg8-05 bz1717396]# python get_bpf_map.py /sys/fs/cgroup/machine.slice/machine-qemu\\x2d2\\x2dvm1.scope/
c 1:7 rw		full
c 136:* rw
c 1:3 rw		null
c 5:2 rw		ptmx
c 1:9 rw		urandom
c 251:0 rw		rtc0
c 1:8 rw		random
b 65:144 rw		sdz
None 67:80
c 1:5 rw		zero
c 10:232 rw		kvm
<==== Now the major:minor==67:80 device still exists in map but value has been set to all zeros
That means, when dump map id=18 following entry still exists, but the value is erase to all zeros
FROM:
# bpftool map dump id 18
key: 50 00 00 00 43 00 00 00  value: 01 00 06 00
...
TO:
# bpftool map dump id 18
key: 50 00 00 00 43 00 00 00  value: 00 00 00 00
...


Actual results:
Device still exists in vm's bpf map after detachment. 

Expected results:
Device entry removed when detach it from vm

Additional info:
This is a minor issue when testing bz1717396, the python file used here can be found in that bug's attachment.

Comment 2 Pavel Hrdina 2020-12-01 11:51:20 UTC

Upstream commit:

commit 0cbcd21b1f981dcca7879130a6653725db87d33a
Author: Pavel Hrdina <phrdina>
Date:   Wed Nov 25 17:21:49 2020 +0100

    vircgroupv2: fix virCgroupV2DenyDevice

Comment 6 yisun 2021-01-17 08:37:55 UTC

Tested with libvirt-7.0.0-1.module+el8.4.0+9464+3e71831a.x86_64
result is : PASS

1. Set selinux to permissive to make sure bpftool can get ebpf info
[root@dell-per740-39 ~]# setenforce 0

2. Get the bpf map info for running vm (vm1)
[root@dell-per740-39 ~]# bpftool cgroup list /sys/fs/cgroup/machine.slice/machine-qemu\\x2d1\\x2dvm1.scope/
ID       AttachType      AttachFlags     Name           
149      device     
<== vm's eBPF program id is 149

[root@dell-per740-39 ~]# bpftool prog show id 149
149: cgroup_device  tag b95e404d31962705  gpl
	loaded_at 2021-01-17T03:13:31-0500  uid 0
	xlated 616B  jited 347B  memlock 4096B  map_ids 149
<== vm's eBPF map id is 149

[root@dell-per740-39 ~]# bpftool map dump id 149
key: 03 00 00 00 01 00 00 00  value: 02 00 06 00
key: 05 00 00 00 01 00 00 00  value: 02 00 06 00
key: 07 00 00 00 01 00 00 00  value: 02 00 06 00
key: 09 00 00 00 01 00 00 00  value: 02 00 06 00
key: ff ff ff ff 88 00 00 00  value: 02 00 06 00
key: 02 00 00 00 05 00 00 00  value: 02 00 06 00
key: e8 00 00 00 0a 00 00 00  value: 02 00 06 00
key: 08 00 00 00 01 00 00 00  value: 02 00 06 00
Found 8 elements
<== Here is 8 elements now, means vm has access right to 8 devices

3. prepare a block device on host, here we use targetcli to prepare a iscsi device (/dev/sdb)
[root@dell-per740-39 ~]# lsscsi
[0:2:0:0]    disk    DELL     PERC H730P Adp   4.30  /dev/sda 
[15:0:0:0]   disk    LIO-ORG  device.logical-  4.0   /dev/sdb 


4. Attach the deivce to vm
[root@dell-per740-39 ~]# cat disk.xml 
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/sdb'/>
      <target dev='vdbd' bus='virtio'/>
    </disk>

[root@dell-per740-39 ~]# virsh attach-device vm1 disk.xml 
Device attached successfully


5. Check the bpf map
[root@dell-per740-39 ~]# bpftool map dump id 149
key: 03 00 00 00 01 00 00 00  value: 02 00 06 00
key: 10 00 00 00 08 00 00 00  value: 01 00 06 00 <==== newly attached device
key: 05 00 00 00 01 00 00 00  value: 02 00 06 00
key: 07 00 00 00 01 00 00 00  value: 02 00 06 00
key: 09 00 00 00 01 00 00 00  value: 02 00 06 00
key: ff ff ff ff 88 00 00 00  value: 02 00 06 00
key: 02 00 00 00 05 00 00 00  value: 02 00 06 00
key: e8 00 00 00 0a 00 00 00  value: 02 00 06 00
key: 08 00 00 00 01 00 00 00  value: 02 00 06 00
Found 9 elements
<== now we found 9 elements, and the newly added one is 'key: 10 00 00 00 08 00 00 00  value: 01 00 06 00'

6. Detach the device from vm
[root@dell-per740-39 ~]# virsh detach-device vm1 disk.xml 
Device detached successfully

7. Check the bpf map
[root@dell-per740-39 ~]# bpftool map dump id 149
key: 03 00 00 00 01 00 00 00  value: 02 00 06 00
key: 05 00 00 00 01 00 00 00  value: 02 00 06 00
key: 07 00 00 00 01 00 00 00  value: 02 00 06 00
key: 09 00 00 00 01 00 00 00  value: 02 00 06 00
key: ff ff ff ff 88 00 00 00  value: 02 00 06 00
key: 02 00 00 00 05 00 00 00  value: 02 00 06 00
key: e8 00 00 00 0a 00 00 00  value: 02 00 06 00
key: 08 00 00 00 01 00 00 00  value: 02 00 06 00
Found 8 elements
<==== As expected, the entry 'key: 10 00 00 00 08 00 00 00  value: 01 00 06 00' removed from the bpf map

Comment 8 errata-xmlrpc 2021-05-25 06:41:21 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2098