Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1689876

Summary:	Huge testpmd startup time when realtime-virtual-host profile is enabled
Product:	Red Hat Enterprise Linux 8	Reporter:	Maxime Coquelin <maxime.coquelin>
Component:	tuned	Assignee:	Peter Xu <peterx>
Status:	CLOSED DUPLICATE	QA Contact:	qe-baseos-daemons
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	8.0	CC:	bhu, echaudro, jeder, jskarvad, lcapitulino, olysonek, peterx, thozza
Target Milestone:	rc	Flags:	pm-rhel: mirror+
Target Release:	8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-02-13 22:26:14 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1755139

Description Maxime Coquelin 2019-03-18 10:51:11 UTC

Description of problem:

DPDK testmpd startup time increases a lot when tuned realtime-virtual-host
profile is enabled.

Analysis on testpmd side shows that it's mlockall() syscall duration
that is increased (from 0.04 seconds with tuned disabled to 5+ seconds when
realtime-virtual-host profile is enabled for a single 1GB hugepage provided to testpmd).

mlockall() duration can be measured using strace:
# strace -T -e trace=mlockall ./install/bin/testpmd -l 0,2,3,4,5 -m 1024 -n 4

Version-Release number of selected component (if applicable):

Reproduced with latest RHEL8 kernel.
Also reproduced with latest upstream kernel (and in this case tuned profile
also makes the NMI watchdog to fire while disabled, see https://lkml.org/lkml/2019/3/8/111)

How reproducible:


Steps to Reproduce:
1. tuned-adm off
2. strace -T -e trace=mlockall ./install/bin/testpmd -l 0,2,3,4,5 -m 1024 -n 4
=> mlockall should last few msecs.
3. tuned-adm profile realtime-virtual-host
(e.g. with 2 CPUs out of 8 isolated)
4. strace -T -e trace=mlockall ./install/bin/testpmd -l 0,2,3,4,5 -m 1024 -n 4
=> mlockall should last several secs.


Additional info:

This issue was initially investigated as an upstream DPDK bug:
https://bugs.dpdk.org/show_bug.cgi?id=186

Comment 10 Peter Xu 2019-11-12 17:10:43 UTC

Ok sorry for the late update... Here's something I got.

I noticed that even if I start a simplest testpmd program ("testpmd -m 1024"), the process will try to lock ~200GB memory.  I'm trying with the latest DPDK, which is 19.11.0-rc2:

xz-x1:~ # pgrep -fa testpmd
26679 sudo ./testpmd -m 1024
26681 ./testpmd -m 1024
xz-x1:~ # grep VmLck /proc/26681/status 
VmLck:  200376752 kB

Here's the mapping that may relevant to such huge memory usage:

100033000-100034000 rw-s 00000000 00:19 140686                           /run/dpdk/rte/fbarray_memseg-1048576k-0-0
140000000-180000000 rw-s 00000000 00:45 583071                           /mnt/hugepages-1G/rtemap_0
180000000-940000000 r--p 00000000 00:00 0                                                                              [~32G]
940000000-940001000 rw-s 00000000 00:19 140687                           /run/dpdk/rte/fbarray_memseg-1048576k-0-1
980000000-1180000000 r--p 00000000 00:00 0                                                                             [~32G]
1180000000-1180001000 rw-s 00000000 00:19 140688                         /run/dpdk/rte/fbarray_memseg-1048576k-0-2
11c0000000-19c0000000 r--p 00000000 00:00 0                                                                            [~32G]
19c0000000-19c0001000 rw-s 00000000 00:19 140689                         /run/dpdk/rte/fbarray_memseg-1048576k-0-3
1a00000000-2200000000 r--p 00000000 00:00 0                                                                            [~32G]
2200000000-2200061000 rw-s 00000000 00:19 131690                         /run/dpdk/rte/fbarray_memseg-2048k-0-0
2200200000-2600200000 r--p 00000000 00:00 0                                                                            [~16G]
2600200000-2600261000 rw-s 00000000 00:19 131691                         /run/dpdk/rte/fbarray_memseg-2048k-0-1
2600400000-2a00400000 r--p 00000000 00:00 0                                                                            [~16G]
2a00400000-2a00461000 rw-s 00000000 00:19 131692                         /run/dpdk/rte/fbarray_memseg-2048k-0-2
2a00600000-2e00600000 r--p 00000000 00:00 0                                                                            [~16G]
2e00600000-2e00661000 rw-s 00000000 00:19 131693                         /run/dpdk/rte/fbarray_memseg-2048k-0-3
2e00800000-3200800000 r--p 00000000 00:00 0                                                                            [~16G]

If we add these big chunks up... 32G*4 + 16G*4 = 192G.  That mostly matches.  All these are 4K private anonymous mappings.

IMHO it didn't eat up the system simply because when we try to populate most of the 200G pages they're filled with the zero page so it didn't really consume the host memory (until written, but they seem to be mapped as RO which is even more strange...).  However it's a bit scary, and it also explains on why mlockall() is slow because it'll still need to traverse the whole 4K mappings and build everything up, and, do that for 200G mem took a few seconds.

And why it's fast on RHEL7 and old hosts?  I didn't look very deep into the RHEL7 code, but my wild guess could be that the GUP code changed quite a bit between RHEL7 and RHEL8/upstream so when we used it on RHEL7 before mm_populate() of mlockall() simply failed far earlier, and because mm_populate() does ignores all errors we didn't notice the fact.

Further look into DPDK, it's memseg_primary_init() that calls alloc_va_space() that malloced all these tons of memories.  I've no idea on why they're needed. Maxime, do you know why it needs such huge memory even if with the simplest testpmd? 

Peter

Comment 14 Luiz Capitulino 2020-02-13 22:06:03 UTC

Peter,

Should this BZ be closed as a dupe of bug 1786923.

Comment 15 Peter Xu 2020-02-13 22:17:30 UTC

(In reply to Luiz Capitulino from comment #14)
> Peter,
> 
> Should this BZ be closed as a dupe of bug 1786923.

Agreed.

Comment 16 Luiz Capitulino 2020-02-13 22:26:14 UTC

Closing as a dupe as per comment 14 and comment 15.

*** This bug has been marked as a duplicate of bug 1786923 ***

Comment 17 Red Hat Bugzilla 2023-09-15 00:16:16 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days