RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2027430 - pmseries --load performance improvements
Summary: pmseries --load performance improvements
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: pcp
Version: 9.2
Hardware: All
OS: All
medium
medium
Target Milestone: beta
: 9.2
Assignee: Andreas Gerstmayr
QA Contact: Jan Kurik
Jacob Taylor Valdez
URL:
Whiteboard:
Depends On: 2026726
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-11-29 16:11 UTC by Andreas Gerstmayr
Modified: 2023-05-09 08:16 UTC (History)
9 users (show)

Fixed In Version: pcp-6.0.0-1.el9
Doc Type: No Doc Update
Doc Text:
Clone Of: 2026726
Environment:
Last Closed: 2023-05-09 07:29:44 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-104172 0 None None None 2021-11-29 16:13:33 UTC
Red Hat Product Errata RHBA-2023:2180 0 None None None 2023-05-09 07:29:55 UTC

Description Andreas Gerstmayr 2021-11-29 16:11:53 UTC 
+++ This bug was initially created as a clone of Bug #2026726 +++

Description of problem:

--- Additional comment from Peter Vreman on 2021-11-26 16:21:08 UTC ---

Loadign 2 weeks of data with pmseries --load does not work, it segfaults:
~~~
[cb/Azure] root@li-lc-2635:~# pmseries --load /var/log/pcp/pmlogger/remote/li-ld-2029
Segmentation fault (core dumped)
[cb/Azure] root@li-lc-2635:~#
~~~

The memory usage of pmseries is also huge:
~~~
[cb/Azure] root@li-lc-2635:# ps auxf
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
...
root   1586582 99.1 32.9 10864596 10763968 pts/0 R+ 16:10   7:56  |                       \_ /usr/bin/pmseries --load /var/log/pcp/pmlogger/remote/li-ld-2029
~~~

Is pmseries --load really tested on large archive sets?

--- Additional comment from Peter Vreman on 2021-11-26 16:23:57 UTC ---

I run it in gdb and aborting it after running 10 minutes and showing a backtrace shows there is a recursion issue:

~~~
(gdb) bt
#0  0x00007ffff6b0c2c6 in __pmTimevalCmp () from /lib64/libpcp.so.3
#1  0x00007ffff6b10967 in searchindom () from /lib64/libpcp.so.3
#2  0x00007ffff6b11538 in __pmLogGetInDom () from /lib64/libpcp.so.3
#3  0x00007ffff6af8041 in pmGetInDom () from /lib64/libpcp.so.3
#4  0x00007ffff7b875cd in pmwebapi_add_indom_instances () from /lib64/libpcp_web.so.1
#5  0x00007ffff7b82308 in series_cache_update () from /lib64/libpcp_web.so.1
#6  0x00007ffff7b8261d in server_cache_window () from /lib64/libpcp_web.so.1
#7  0x00007ffff7b81d76 in doneSeriesGetContext () from /lib64/libpcp_web.so.1
#8  0x00007ffff7b8202b in series_cache_update () from /lib64/libpcp_web.so.1
#9  0x00007ffff7b8261d in server_cache_window () from /lib64/libpcp_web.so.1
#10 0x00007ffff7b81d76 in doneSeriesGetContext () from /lib64/libpcp_web.so.1
#11 0x00007ffff7b8202b in series_cache_update () from /lib64/libpcp_web.so.1
#12 0x00007ffff7b8261d in server_cache_window () from /lib64/libpcp_web.so.1
#13 0x00007ffff7b81d76 in doneSeriesGetContext () from /lib64/libpcp_web.so.1
#14 0x00007ffff7b8202b in series_cache_update () from /lib64/libpcp_web.so.1
#15 0x00007ffff7b8261d in server_cache_window () from /lib64/libpcp_web.so.1
#16 0x00007ffff7b81d76 in doneSeriesGetContext () from /lib64/libpcp_web.so.1
#17 0x00007ffff7b8202b in series_cache_update () from /lib64/libpcp_web.so.1
#18 0x00007ffff7b8261d in server_cache_window () from /lib64/libpcp_web.so.1
#19 0x00007ffff7b81d76 in doneSeriesGetContext () from /lib64/libpcp_web.so.1
#20 0x00007ffff7b8202b in series_cache_update () from /lib64/libpcp_web.so.1
#21 0x00007ffff7b8261d in server_cache_window () from /lib64/libpcp_web.so.1
#22 0x00007ffff7b81d76 in doneSeriesGetContext () from /lib64/libpcp_web.so.1
#23 0x00007ffff7b8202b in series_cache_update () from /lib64/libpcp_web.so.1
#24 0x00007ffff7b8261d in server_cache_window () from /lib64/libpcp_web.so.1
#25 0x00007ffff7b81d76 in doneSeriesGetContext () from /lib64/libpcp_web.so.1
#26 0x00007ffff7b8202b in series_cache_update () from /lib64/libpcp_web.so.1
#27 0x00007ffff7b8261d in server_cache_window () from /lib64/libpcp_web.so.1
...
#29498 0x00007ffff7b8202b in series_cache_update () from /lib64/libpcp_web.so.1
#29499 0x00007ffff7b8261d in server_cache_window () from /lib64/libpcp_web.so.1
#29500 0x00007ffff7b81d76 in doneSeriesGetContext () from /lib64/libpcp_web.so.1
#29501 0x00007ffff7b8202b in series_cache_update () from /lib64/libpcp_web.so.1
#29502 0x00007ffff7b8261d in server_cache_window () from /lib64/libpcp_web.so.1
#29503 0x00007ffff7b81d76 in doneSeriesGetContext () from /lib64/libpcp_web.so.1
#29504 0x00007ffff7b8202b in series_cache_update () from /lib64/libpcp_web.so.1
#29505 0x00007ffff7b8261d in server_cache_window () from /lib64/libpcp_web.so.1
#29506 0x00007ffff7b81d76 in doneSeriesGetContext () from /lib64/libpcp_web.so.1
#29507 0x00007ffff7b8202b in series_cache_update () from /lib64/libpcp_web.so.1
#29508 0x00007ffff7b8261d in server_cache_window () from /lib64/libpcp_web.so.1
#29509 0x00007ffff7b81d76 in doneSeriesGetContext () from /lib64/libpcp_web.so.1
#29510 0x00007ffff7b8202b in series_cache_update () from /lib64/libpcp_web.so.1
#29511 0x00007ffff7b8261d in server_cache_window () from /lib64/libpcp_web.so.1
#29512 0x00007ffff7b81d76 in doneSeriesGetContext () from /lib64/libpcp_web.so.1
#29513 0x00007ffff7b8202b in series_cache_update () from /lib64/libpcp_web.so.1
#29514 0x00007ffff7b8261d in server_cache_window () from /lib64/libpcp_web.so.1
#29515 0x00007ffff7b82738 in series_cache_metrics () from /lib64/libpcp_web.so.1
#29516 0x00007ffff7b88f5c in redisSlotsReplyCallback () from /lib64/libpcp_web.so.1
#29517 0x00007ffff7b9d4d0 in redisClusterAsyncCallback () from /lib64/libpcp_web.so.1
#29518 0x00007ffff7b92c95 in redisProcessCallbacks () from /lib64/libpcp_web.so.1
#29519 0x00007ffff7b88539 in redisLibuvPoll () from /lib64/libpcp_web.so.1
#29520 0x00007ffff71cbd15 in uv.io_poll () from /lib64/libuv.so.1
#29521 0x00007ffff71baa74 in uv_run () from /lib64/libuv.so.1
#29522 0x00005555555569e5 in main ()
(gdb)
~~~

Version-Release number of selected component (if applicable):
Package - pcp-5.3.1-5.el8.x86_64.rpm
Comment 1 Peter Vreman 2021-12-08 11:33:06 UTC 
Minimal set of archive files to reproduce it uploaded in customer case 03098682
Comment 3 Nathan Scott 2021-12-13 05:56:10 UTC 
This is fixed in current versions of PCP (i.e. from RHEL 8.6 and 9.0):

$ pmseries --load 20211030.meta 
pmseries: [Info] processed 17575 archive records from [...]/20211030.meta

However, it takes quite a long time to import into Redis (26 minutes @ ~100% CPU for both pmseries and redis-server).  I'm certain we can do better here, so I've adjusted the title of this bug to reflect this remaining work.  For reference it takes about 1 minute for pmlogsummary to process this 3.3Gb archive.  Both pmseries and pmlogsummary use the libpcp pmFetchArchive(3) routine to process the log, so that's kind of a best-case baseline comparison from the PCP side.  I expect there are a number of Redis request-submission-pattern changes we can make to improve the situation.
Comment 29 errata-xmlrpc 2023-05-09 07:29:44 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (pcp bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2180
Note You need to log in before you can comment on or make changes to this bug.