Created attachment 1947287 [details]
1:qemu log during migration failure

Description of problem:

I tried the steps that are known to work for rdma live migrations on x86_64.
The migration fails with an error.

Version-Release number of selected component (if applicable):
libvirt-9.0.0-7.el9.s390x

How reproducible:
100%


Steps to Reproduce:

0. Use two systems, each with device
     Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
   They must have the same PCHID, which can be checked e.g. via smc_rnics

1. Load necessary kernel modules ib_ipoib mlx5_ib ib_umad rpcrdma ib_srpt ib_iser ib_isert

2. Install necessary packages libibverbs librdmacm infiniband-diags

3. Use ibstat to confirm the ethernet controllers are
     State: Active
     Physical state: LinkUp
     Link layer: Ethernet

4. Assign local address to the network interfaces and confirm they can communicate via ping, e.g.
     ip a add 192.168.100.2/24; ping 192.168.100.3

5. Configure and restart virtqemud, activate all entries for cgroup_device_acl
	"/dev/null", "/dev/full", "/dev/zero",
	"/dev/random", "/dev/urandom",
	"/dev/ptmx", "/dev/kvm", "/dev/kqemu",
	"/dev/rtc","/dev/hpet", "/dev/vfio/vfio",
	"/dev/infiniband/rdma_cm",
	"/dev/infiniband/issm1",
	"/dev/infiniband/umad1",
	"/dev/infiniband/uverbs1"

6. Create shared storage for the VM qcow2 (e.g. NFS: nfs-utils, /etc/export: /nfs *(rw,no_root_squash,async); systemctl restart nfs-server)

7. Turn off firewall

8. Set memory limits on VM
   <memtune>
       <hard_limit unit='KiB'>1048576</hard_limit>
       <swap_hard_limit unit='KiB'>2097152</swap_hard_limit>
   </memtune>

9. Start VM

8. Migrate VM:
   virsh migrate --live --migrateuri rdma://192.168.100.2 rhel7 --listen-address 0 qemu+ssh://192.168.100.2/system --verbose


Actual results:
error: internal error: unable to execute QEMU command 'migrate-incoming': unknown migration protocol: rdma:0:49152

Expected results:
The migration succeeds.

Additional info:
1. The original instructions at hand included more steps that I decided were unnecessary (or didn't work when I tried after building the rpms from srpm)
 a. use opensm to make sure the devices can communicate for step 4; they already could communicate without this
 b. use mstconf to set the link layer type; I couldn't change this so I took the preset Ethernet
2. Attach virtqemud debug log
3. The kernel modules weren't initially built, I'll share a link to the kernel config and my rpms in a private comment