Description of problem: On Fedora 32 attaching iscsi disk device fails with: ``` error: failed to initialize device BPF map: Operation not permitted ``` This reports comes from Fedora 32 testing image for Cockpit project. Version-Release number of selected component (if applicable): libvirt-client-6.0.0-1.fc32.x86_64 How reproducible: 1. Prepare disk definition: ``` $ cat disk.xml <disk type="volume" device="disk"><driver name="qemu" cache="default"/><source volume="unit:0:0:0" pool="iscsi-pool"/><target dev="vdh" bus="virtio"/></disk> ``` 2. Have a pool ``` $ virsh pool-info iscsi-pool Name: iscsi-pool UUID: 6590a291-0fbb-48b6-8667-674d7a082b89 State: running Persistent: yes Autostart: no Capacity: 50.00 MiB Allocation: 50.00 MiB Available: 0.00 B ``` 3. And and a volume ``` $ virsh vol-list iscsi-pool Name Path --------------------------------------------------------------------------------------- unit:0:0:0 /dev/disk/by-path/ip-127.0.0.1:3260-iscsi-iqn.2019-09.cockpit.lan-lun-0 ``` 4. And of course a domain ``` $ sudo virsh dominfo subVmTest1 Id: 1 Name: subVmTest1 UUID: 90c54d65-a653-44b8-a18f-47878b596b8d OS Type: hvm State: running CPU(s): 1 CPU time: 29.1s Max memory: 262144 KiB Used memory: 262144 KiB Persistent: yes Autostart: disable Managed save: no Security model: selinux Security DOI: 0 Security label: system_u:system_r:svirt_tcg_t:s0:c565,c630 (enforcing) ``` 5. Try to attach the disk into the domain ``` $ virsh attach-device --domain subVmTest1 --file disk.xml error: Failed to attach device from disk.xml error: failed to initialize device BPF map: Operation not permitted ``` Expected results: Disk gets attach without error. I can provide any further information if needed.
Matej, anything in the acl log or in journal related to libvirt? What is libvirt trying to do is the following syscall, which fails with EPERM: https://libvirt.org/git/?p=libvirt.git;a=blob;f=src/util/virbpf.c;h=dec8d0133a1f241eabac202de5b1341417544cdd;hb=HEAD#l48
`journalctl -f` and executing step 5. produces: ``` Feb 25 11:01:03 localhost.localdomain libvirtd[6469]: failed to initialize device BPF map: Operation not permitted Feb 25 11:01:03 localhost.localdomain kernel: kauditd_printk_skb: 7 callbacks suppressed Feb 25 11:01:03 localhost.localdomain kernel: audit: type=2501 audit(1582646463.329:484): pid=6469 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=qemu resrc=cgroup reason=allow vm="subVmTest1" uuid=b517029a-6163-401d-aae0-324b11d3207c cgroup="/sys/fs/cgroup/machine.slice/machine-qemu\x2d1\x2dsubVmTest1.scope/" class=path path="/dev/disk/by-path/ip-127.0.0.1:3260-iscsi-iqn.2019-09.cockpit.lan-lun-0" rdev=08:00 acl=rw exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=failed' Feb 25 11:01:03 localhost.localdomain audit[6469]: VIRT_RESOURCE pid=6469 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=qemu resrc=cgroup reason=allow vm="subVmTest1" uuid=b517029a-6163-401d-aae0-324b11d3207c cgroup="/sys/fs/cgroup/machine.slice/machine-qemu\x2d1\x2dsubVmTest1.scope/" class=path path="/dev/disk/by-path/ip-127.0.0.1:3260-iscsi-iqn.2019-09.cockpit.lan-lun-0" rdev=08:00 acl=rw exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=failed' ```
Note there's a similar error that can be hit with libvirt LXC usage: https://bugzilla.redhat.com/show_bug.cgi?id=1770763#c12
Looking into the kernel source code there is one possibility. The failing syscall is the following: syscall(SYS_bpf, BPF_MAP_CREATE, &attr, sizeof(attr)); And I could find the following call trace: map_create() find_and_alloc_map() htab_map_alloc() bpf_map_charge_init() bpf_charge_memlock() The last one checks if the map alloction would not exceed memlock limit. If it does, EPERM is returned. And since the memlock by default is pretty small (64KB), I think there's high chance we are hitting it and thus seeing EPERM.
Upstream patch posted: https://www.redhat.com/archives/libvir-list/2020-February/msg01086.html
Pushed upstream: b379fee117 daemon: set default memlock limit for systemd service v6.1.0-rc1-4-gb379fee117
The rebased version was pushed into Fedora-32 which contains the fix: https://bodhi.fedoraproject.org/updates/FEDORA-2020-e2ecf59224