Bug 1807090

Summary:	Cannot attach iscsi disk device to domain
Product:	[Fedora] Fedora	Reporter:	Matej Marušák <mmarusak>
Component:	libvirt	Assignee:	Michal Privoznik <mprivozn>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	32	CC:	agedosier, berrange, clalancette, crobinso, itamar, jforbes, laine, libvirt-maint, mprivozn, phrdina, veillard, virt-maint
Target Milestone:	---	Keywords:	Upstream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	libvirt-6.1.0-2.fc32	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-04-01 17:48:16 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Matej Marušák 2020-02-25 15:12:22 UTC

Description of problem:
On Fedora 32 attaching iscsi disk device fails with:
```
error: failed to initialize device BPF map: Operation not permitted
```
This reports comes from Fedora 32 testing image for Cockpit project.

Version-Release number of selected component (if applicable):
libvirt-client-6.0.0-1.fc32.x86_64

How reproducible:
1. Prepare disk definition:
```
$ cat disk.xml 
<disk type="volume" device="disk"><driver name="qemu" cache="default"/><source volume="unit:0:0:0" pool="iscsi-pool"/><target dev="vdh" bus="virtio"/></disk>
```


2.  Have a pool
```
$ virsh pool-info iscsi-pool
Name:           iscsi-pool
UUID:           6590a291-0fbb-48b6-8667-674d7a082b89
State:          running
Persistent:     yes
Autostart:      no
Capacity:       50.00 MiB
Allocation:     50.00 MiB
Available:      0.00 B
```

3. And and a volume
```
$ virsh vol-list iscsi-pool
 Name         Path
---------------------------------------------------------------------------------------
 unit:0:0:0   /dev/disk/by-path/ip-127.0.0.1:3260-iscsi-iqn.2019-09.cockpit.lan-lun-0
```

4. And of course a domain
```
$ sudo virsh dominfo subVmTest1
Id:             1
Name:           subVmTest1
UUID:           90c54d65-a653-44b8-a18f-47878b596b8d
OS Type:        hvm
State:          running
CPU(s):         1
CPU time:       29.1s
Max memory:     262144 KiB
Used memory:    262144 KiB
Persistent:     yes
Autostart:      disable
Managed save:   no
Security model: selinux
Security DOI:   0
Security label: system_u:system_r:svirt_tcg_t:s0:c565,c630 (enforcing)
```

5. Try to attach the disk into the domain
```
$ virsh attach-device --domain subVmTest1 --file disk.xml
error: Failed to attach device from disk.xml
error: failed to initialize device BPF map: Operation not permitted
```

Expected results:
Disk gets attach without error.

I can provide any further information if needed.

Comment 1 Michal Privoznik 2020-02-25 15:28:43 UTC

Matej, anything in the acl log or in journal related to libvirt? What is libvirt trying to do is the following syscall, which fails with EPERM:

https://libvirt.org/git/?p=libvirt.git;a=blob;f=src/util/virbpf.c;h=dec8d0133a1f241eabac202de5b1341417544cdd;hb=HEAD#l48

Comment 2 Matej Marušák 2020-02-25 16:03:32 UTC

`journalctl -f` and executing step 5. produces:
```
Feb 25 11:01:03 localhost.localdomain libvirtd[6469]: failed to initialize device BPF map: Operation not permitted
Feb 25 11:01:03 localhost.localdomain kernel: kauditd_printk_skb: 7 callbacks suppressed
Feb 25 11:01:03 localhost.localdomain kernel: audit: type=2501 audit(1582646463.329:484): pid=6469 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=qemu resrc=cgroup reason=allow vm="subVmTest1" uuid=b517029a-6163-401d-aae0-324b11d3207c cgroup="/sys/fs/cgroup/machine.slice/machine-qemu\x2d1\x2dsubVmTest1.scope/" class=path path="/dev/disk/by-path/ip-127.0.0.1:3260-iscsi-iqn.2019-09.cockpit.lan-lun-0" rdev=08:00 acl=rw exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=failed'
Feb 25 11:01:03 localhost.localdomain audit[6469]: VIRT_RESOURCE pid=6469 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=qemu resrc=cgroup reason=allow vm="subVmTest1" uuid=b517029a-6163-401d-aae0-324b11d3207c cgroup="/sys/fs/cgroup/machine.slice/machine-qemu\x2d1\x2dsubVmTest1.scope/" class=path path="/dev/disk/by-path/ip-127.0.0.1:3260-iscsi-iqn.2019-09.cockpit.lan-lun-0" rdev=08:00 acl=rw exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=failed'
```

Comment 3 Cole Robinson 2020-02-25 16:13:22 UTC

Note there's a similar error that can be hit with libvirt LXC usage: https://bugzilla.redhat.com/show_bug.cgi?id=1770763#c12

Comment 4 Michal Privoznik 2020-02-26 10:14:45 UTC

Looking into the kernel source code there is one possibility. The failing syscall is the following:

syscall(SYS_bpf, BPF_MAP_CREATE, &attr, sizeof(attr));

And I could find the following call trace:

map_create()
find_and_alloc_map()
htab_map_alloc()
bpf_map_charge_init()
bpf_charge_memlock()

The last one checks if the map alloction would not exceed memlock limit. If it does, EPERM is returned. And since the memlock by default is pretty small (64KB), I think there's high chance we are hitting it and thus seeing EPERM.

Comment 5 Pavel Hrdina 2020-02-26 15:11:34 UTC

Upstream patch posted:

https://www.redhat.com/archives/libvir-list/2020-February/msg01086.html

Comment 6 Michal Privoznik 2020-02-27 12:00:33 UTC

Pushed upstream:

b379fee117 daemon: set default memlock limit for systemd service

v6.1.0-rc1-4-gb379fee117

Comment 7 Michal Privoznik 2020-04-01 17:48:16 UTC

The rebased version was pushed into Fedora-32 which contains the fix:

https://bodhi.fedoraproject.org/updates/FEDORA-2020-e2ecf59224