Bug 1939776 - Increase the memlock limit for VM with vDPA devices
Summary: Increase the memlock limit for VM with vDPA devices
Keywords:
Status: ON_QA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: libvirt
Version: 8.4
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 8.5
Assignee: Laine Stump
QA Contact: yalzhang@redhat.com
URL:
Whiteboard:
Depends On:
Blocks: 1955681
TreeView+ depends on / blocked
 
Reported: 2021-03-17 03:04 UTC by jason wang
Modified: 2021-06-16 10:25 UTC (History)
12 users (show)

Fixed In Version: libvirt-7.4.0-1.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
Target Upstream Version: 7.4.0


Attachments (Terms of Use)

Description jason wang 2021-03-17 03:04:44 UTC
Description of problem:

Currently, Qemu will ask vhost-vDPA to pin all guest memory pages in order to have a consistent physical address for device to DMA. This requires the libvirt to increase the memlock limit via prlimit(2) as what has been done for VFIO device. Otherwise, Qemu can't establish correct IOMMU mappings which may break vDPA devices.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Start a VM with vDPA device
2.
3.

Actual results:

Qemu warnings like:

qemu-system-x86_64: failed to write, fd=13, errno=14 (Bad address)
qemu-system-x86_64: vhost vdpa map fail!
qemu-system-x86_64: vhost-vdpa: DMA mapping failed, unable to continue

Expected results:

No warnings at all.


Additional info:

Comment 1 Daniel Berrangé 2021-03-23 19:11:31 UTC
(In reply to jason wang from comment #0)
> Description of problem:
> 
> Currently, Qemu will ask vhost-vDPA to pin all guest memory pages in order
> to have a consistent physical address for device to DMA. This requires the
> libvirt to increase the memlock limit via prlimit(2) as what has been done
> for VFIO device. Otherwise, Qemu can't establish correct IOMMU mappings
> which may break vDPA devices.

snip

> Qemu warnings like:
> 
> qemu-system-x86_64: failed to write, fd=13, errno=14 (Bad address)
> qemu-system-x86_64: vhost vdpa map fail!
> qemu-system-x86_64: vhost-vdpa: DMA mapping failed, unable to continue

If these problems are likely to break vDPA devices, why are they just warnings, as opposed to hard errors from QEMU ?

Comment 2 jason wang 2021-03-24 03:22:44 UTC
(In reply to Daniel Berrangé from comment #1)
> (In reply to jason wang from comment #0)
> > Description of problem:
> > 
> > Currently, Qemu will ask vhost-vDPA to pin all guest memory pages in order
> > to have a consistent physical address for device to DMA. This requires the
> > libvirt to increase the memlock limit via prlimit(2) as what has been done
> > for VFIO device. Otherwise, Qemu can't establish correct IOMMU mappings
> > which may break vDPA devices.
> 
> snip
> 
> > Qemu warnings like:
> > 
> > qemu-system-x86_64: failed to write, fd=13, errno=14 (Bad address)
> > qemu-system-x86_64: vhost vdpa map fail!
> > qemu-system-x86_64: vhost-vdpa: DMA mapping failed, unable to continue
> 
> If these problems are likely to break vDPA devices, why are they just
> warnings, as opposed to hard errors from QEMU ?

Right, this is a bug that needs to be fixed.

Adding Cindy.

Thanks

Comment 3 lulu@redhat.com 2021-03-24 05:32:52 UTC
(In reply to jason wang from comment #2)
> (In reply to Daniel Berrangé from comment #1)
> > (In reply to jason wang from comment #0)
> > > Description of problem:
> > > 
> > > Currently, Qemu will ask vhost-vDPA to pin all guest memory pages in order
> > > to have a consistent physical address for device to DMA. This requires the
> > > libvirt to increase the memlock limit via prlimit(2) as what has been done
> > > for VFIO device. Otherwise, Qemu can't establish correct IOMMU mappings
> > > which may break vDPA devices.
> > 
> > snip
> > 
> > > Qemu warnings like:
> > > 
> > > qemu-system-x86_64: failed to write, fd=13, errno=14 (Bad address)
> > > qemu-system-x86_64: vhost vdpa map fail!
> > > qemu-system-x86_64: vhost-vdpa: DMA mapping failed, unable to continue
> > 
> > If these problems are likely to break vDPA devices, why are they just
> > warnings, as opposed to hard errors from QEMU ?
> 
> Right, this is a bug that needs to be fixed.
> 
> Adding Cindy.
> 
> Thanks

sure, will fix this problem

Comment 4 Laine Stump 2021-03-25 17:47:38 UTC
I pushed these 4 patches upstream which implement the requested functionality:

43d8d16161 conf: new function virDomainDefHasVDPANet()
0789edc86a qemu: simplify qemuDomainGetMemLockLimitBytes()
e7068a0bc2 qemu: account for mdev devices in getPPC64MemLockLimitBytes()
98e22ff749 qemu: increase locked memory limit when a vDPA device is present


They will be in upstream libvirt-7.2.0.

Comment 10 yalzhang@redhat.com 2021-05-14 09:11:50 UTC
Reproduce this bug on libvirt-7.0.0-13.module+el8.4.0+10604+5608c2b4.x86_64:

Packages:
# rpm -q libvirt kernel qemu-kvm
libvirt-7.0.0-13.module+el8.4.0+10604+5608c2b4.x86_64
kernel-4.18.0-305.4.el8.x86_64
qemu-kvm-6.0.0-16.module+el8.5.0+10848+2dccc46d.x86_64

1. On a system with "Mellanox Technologies MT2892 Family [ConnectX-6 Dx]" card, upgrade the firmware to the latest version, then initialize the ovs bridge and tc offloading;
2. Start vm with interface as below:
# virsh dumpxml rhel | grep /interface -B6
    <interface type='vdpa'>
      <source dev='/dev/vhost-vdpa-0'/>
    </interface>
# virsh start rhel 
Domain 'rhel' started

3. vm start successfully, but can not obtain an ip address, check the guest's log:
# cat /var/log/libvirt/qemu/rhel.log
...
qemu-kvm: -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on: warning: short-form boolean option 'disable-ticketing' deprecated
Please use disable-ticketing=on instead
char device redirected to /dev/pts/3 (label charserial0)
2021-05-14T08:56:56.768655Z qemu-kvm: info: Zero hardware mac address detected. Ignoring.
2021-05-14T08:56:56.768768Z qemu-kvm: info: Zero hardware mac address detected. Ignoring.
2021-05-14T08:56:56.768792Z qemu-kvm: info: Zero hardware mac address detected. Ignoring.
2021-05-14T08:56:56.768814Z qemu-kvm: info: Zero hardware mac address detected. Ignoring.
2021-05-14T08:56:56.768836Z qemu-kvm: info: Zero hardware mac address detected. Ignoring.
2021-05-14T08:56:56.768858Z qemu-kvm: info: Zero hardware mac address detected. Ignoring.
2021-05-14T08:56:56.786945Z qemu-kvm: failed to write, fd=20, errno=14 (Bad address)
2021-05-14T08:56:56.786966Z qemu-kvm: vhost vdpa map fail!
2021-05-14T08:56:56.786976Z qemu-kvm: vhost-vdpa: DMA mapping failed, unable to continue
2021-05-14T08:56:56.786989Z qemu-kvm: failed to write, fd=20, errno=14 (Bad address)
2021-05-14T08:56:56.786999Z qemu-kvm: vhost vdpa map fail!
2021-05-14T08:56:56.787008Z qemu-kvm: vhost-vdpa: DMA mapping failed, unable to continue
2021-05-14T08:56:56.787022Z qemu-kvm: failed to write, fd=20, errno=14 (Bad address)
2021-05-14T08:56:56.787031Z qemu-kvm: vhost vdpa map fail!
2021-05-14T08:56:56.787040Z qemu-kvm: vhost-vdpa: DMA mapping failed, unable to continue

there is error "vhost vdpa map fail".

Update libvirt to libvirt-7.3.0-1.module+el8.5.0+11004+f4810536.x86_64, and start the vm again
1. After vm boot successfully, it can get dhcp address and ping outside successfully:
# ifconfig 
enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.73.**.**  netmask 255.255.254.0  broadcast 10.73.**.255
        inet6 fe80::5054:ff:fe9a:e37f  prefixlen 64  scopeid 0x20<link>
        inet6 2620:52:0:4920:5054:ff:fe9a:e37f  prefixlen 64  scopeid 0x0<global>
        ether 52:54:00:9a:e3:7f  txqueuelen 1000  (Ethernet)
        RX packets 84  bytes 11291 (11.0 KiB)
        RX errors 0  dropped 2  overruns 0  frame 0
        TX packets 107  bytes 10553 (10.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
...
# ping www.baidu.com -c 2
PING www.wshifen.com (104.193.88.123) 56(84) bytes of data.
64 bytes from 104.193.88.123 (104.193.88.123): icmp_seq=1 ttl=47 time=165 ms
64 bytes from 104.193.88.123 (104.193.88.123): icmp_seq=2 ttl=47 time=165 ms

--- www.wshifen.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 164.902/164.909/164.917/0.406 ms

2.check the guest's log:
# cat   /var/log/libvirt/qemu/rhel.log
...
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/3 (label charserial0)
2021-05-14T09:07:06.718461Z qemu-kvm: info: Zero hardware mac address detected. Ignoring.
2021-05-14T09:07:06.718609Z qemu-kvm: info: Zero hardware mac address detected. Ignoring.
2021-05-14T09:07:06.718635Z qemu-kvm: info: Zero hardware mac address detected. Ignoring.
2021-05-14T09:07:06.718657Z qemu-kvm: info: Zero hardware mac address detected. Ignoring.
2021-05-14T09:07:06.718680Z qemu-kvm: info: Zero hardware mac address detected. Ignoring.
2021-05-14T09:07:06.718702Z qemu-kvm: info: Zero hardware mac address detected. Ignoring.

there is no error like "vhost vdpa map fail", and the network works well.

Comment 13 yalzhang@redhat.com 2021-05-17 01:18:02 UTC
Hi Laine, When I hotplug a vdpa device, the issue still exists. Start a vm with vdpa device works well, see above comment 10.
1. Start a vm without any interfaces;
2. Hotplug a vdpa interface to the running vm;
3. Check on the vm, the interface can not get ip address, and check the guest's log, there is error:
# cat /var/log/libvirt/qemu/rhel.log
...
2021-05-17T01:13:16.525914Z qemu-kvm: info: Zero hardware mac address detected. Ignoring.
2021-05-17T01:13:16.525936Z qemu-kvm: info: Zero hardware mac address detected. Ignoring.
2021-05-17T01:13:16.525958Z qemu-kvm: info: Zero hardware mac address detected. Ignoring.
2021-05-17T01:13:16.525979Z qemu-kvm: info: Zero hardware mac address detected. Ignoring.
2021-05-17T01:13:16.527951Z qemu-kvm: failed to write, fd=93, errno=14 (Bad address)
2021-05-17T01:13:16.527971Z qemu-kvm: vhost vdpa map fail!
2021-05-17T01:13:16.527982Z qemu-kvm: vhost-vdpa: DMA mapping failed, unable to continue
2021-05-17T01:13:16.527994Z qemu-kvm: failed to write, fd=93, errno=14 (Bad address)
2021-05-17T01:13:16.528005Z qemu-kvm: vhost vdpa map fail!

Comment 14 Laine Stump 2021-05-20 17:29:08 UTC
Yes, I see the problem - although there is a single function that is always called when wanting to know the desired locked memory amount, and that function accounts for all devices that need locked memory, in the case of hotplugging a device only the functions for certain devices actually *call* that function. I'll work up a patch for it.

Comment 16 Laine Stump 2021-05-21 19:12:33 UTC
A fix for the bug Yalan found in Comment 13 has been posted upstream:

https://listman.redhat.com/archives/libvir-list/2021-May/msg00705.html

Comment 17 Laine Stump 2021-05-25 14:38:11 UTC
The following 4 patches resolving the hotplug bug described in Comment 13 have been pushed upstream:

6731a6805d qemu_hotplug.c: don't skip cleanup on failures of qemuDomainAttachNetDevice
a52b74e7a3 conf: new function virDomainNetRemoveByObj()
7b8bec4560 qemu_hotplug.c: add net devices to the domain list earlier
97075dff5c (HEAD -> master, upstream/master, active-bz1939776-memlock-hotplug-vdpa-2) qemu: adjust the maxmemlock limit when hotplugging a vDPA device


Note You need to log in before you can comment on or make changes to this bug.