Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
DescriptionOneata Mircea Teodor
2019-04-12 14:27:20 UTC
+++ This bug was initially created as a clone of Bug #1670548 +++
Description of problem:
The following PCP stats for VDO return a message of
"pmval: pmFetch: IPC protocol failure"
PCP Stat Name
vdo.dev.curr_dedupe_queries
vdo.dev.max_dedupe_queries
vdo.dev.dedupe_advice.stale
vdo.dev.dedupe_advice.valid
vdo.dev.read_cache.accesses
vdo.dev.read_cache.data_hits
vdo.dev.read_cache.hits
Four of the VDO stat names have been changed this might be the reason why
they fail.
PCP Stat Name New VDO stat name
vdo.dev.curr_dedupe_queries index_max_dedupe_queries
vdo.dev.max_dedupe_queries index_max_dedupe_queries
vdo.dev.dedupe_advice.stale hash_lock_dedupe_advice_stale
vdo.dev.dedupe_advice.valid hash_lock_dedupe_advice_valid
VDO's read cache was deprecated in 7.6, so these stats do not exist in RHEL8.
Consider removing them from PCP in RHEL 8+
vdo.dev.read_cache.accesses - does not exist in RHEL 8
vdo.dev.read_cache.data_hits - does not exist in RHEL 8
vdo.dev.read_cache.hits - does not exist in RHEL 8
Example of the error message
# pmval -v -s 1 vdo.dev.curr_dedupe_queries
metric: vdo.dev.curr_dedupe_queries
host: host-8-243-15.host.centralci.eng.rdu2.redhat.com
semantics: instantaneous value
units: count
samples: 1
pmval: pmFetch: IPC protocol failure
How to reproduce on RHEL 8:
1. yum -y install vdo pcp pcp-pmda-dm
2. cd /var/lib/pcp/pmdas/dm/
3. ./Install
4. vdo create --name=vdotest --device=/dev/sdb
5. pmval -v -s 1 vdo.dev.curr_dedupe_queries
--- Additional comment from Dennis Keefe on 2019-01-30 22:15:43 UTC ---
This issue exists in 8.0 and if possible, it would be helpful if a fix could be added to RHEL 8.0.
--- Additional comment from Nathan Scott on 2019-01-30 22:21:06 UTC ---
(In reply to Dennis Keefe from comment #1)
> This issue exists in 8.0 and if possible, it would be helpful if a fix could
> be added to RHEL 8.0.
Understood - at this stage we don't intend to update PCP in 8.0 further, but 8.1 is no problem.
cheers.
--- Additional comment from Dennis Keefe on 2019-02-01 23:29:06 UTC ---
Also, if pmval is run 3 times for any of these stats it will throw a segfault
[root@host-8-246-82 coredump]# coredumpctl list
TIME PID UID GID SIG COREFILE EXE
Fri 2019-02-01 18:11:45 EST 15444 0 0 11 present /var/lib/pcp/pmdas/dm/pmdadm
Fri 2019-02-01 18:11:46 EST 21584 0 0 11 present /var/lib/pcp/pmdas/dm/pmdadm
Fri 2019-02-01 18:15:00 EST 21911 0 0 11 present /var/lib/pcp/pmdas/dm/pmdadm
Fri 2019-02-01 18:15:05 EST 21923 0 0 11 present /var/lib/pcp/pmdas/dm/pmdad
[root@host-8-246-82 coredump]# coredumpctl gdb 21923
PID: 21923 (pmdadm)
UID: 0 (root)
GID: 0 (root)
Signal: 11 (SEGV)
Timestamp: Fri 2019-02-01 18:15:05 EST (11min ago)
Command Line: /var/lib/pcp/pmdas/dm/pmdadm -d 129
Executable: /var/lib/pcp/pmdas/dm/pmdadm
Control Group: /system.slice/pmcd.service
Unit: pmcd.service
Slice: system.slice
Boot ID: eab09d88401447e88af95cf704e75710
Machine ID: 591d7280467543a2bd433dccc22b02ef
Hostname: host-8-246-82.host.centralci.eng.rdu2.redhat.com
Storage: /var/lib/systemd/coredump/core.pmdadm.0.eab09d88401447e88af95cf704e75710.21923.1549062905000000.lz4
Message: Process 21923 (pmdadm) of user 0 dumped core.
Stack trace of thread 21923:
#0 0x00007fa03046361a __GI_____strtoull_l_internal (libc.so.6)
#1 0x0000557a85bccbb1 vdo_fetch_ull (pmdadm)
#2 0x0000557a85bccd55 dm_vdodev_fetch (pmdadm)
#3 0x0000557a85bcb486 dm_fetchCallBack (pmdadm)
#4 0x00007fa030cf11f3 pmdaFetch (libpcp_pmda.so.3)
#5 0x0000557a85bcb780 dm_fetch (pmdadm)
#6 0x00007fa030cf4591 __pmdaMainPDU (libpcp_pmda.so.3)
#7 0x00007fa030cf4d38 pmdaMain (libpcp_pmda.so.3)
#8 0x0000557a85bcb2fc main (pmdadm)
#9 0x00007fa03044a813 __libc_start_main (libc.so.6)
#10 0x0000557a85bcb33e _start (pmdadm)
GNU gdb (GDB) Red Hat Enterprise Linux 8.2-5.el8
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /var/lib/pcp/pmdas/dm/pmdadm...Reading symbols from .gnu_debugdata for /var/lib/pcp/pmdas/dm/pmdadm...(no debugging symbols found)...done.
(no debugging symbols found)...done.
[New LWP 21923]
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
Core was generated by `/var/lib/pcp/pmdas/dm/pmdadm -d 129'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fa03046361a in ____strtoull_l_internal () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install pcp-pmda-dm-4.3.0-3.el8.x86_64
--- Additional comment from Dennis Keefe on 2019-02-01 23:32:48 UTC ---
and after installing a bunch of debug packages, here is the output.
[root@host-8-246-82 coredump]# coredumpctl gdb 21923
PID: 21923 (pmdadm)
UID: 0 (root)
GID: 0 (root)
Signal: 11 (SEGV)
Timestamp: Fri 2019-02-01 18:15:05 EST (16min ago)
Command Line: /var/lib/pcp/pmdas/dm/pmdadm -d 129
Executable: /var/lib/pcp/pmdas/dm/pmdadm
Control Group: /system.slice/pmcd.service
Unit: pmcd.service
Slice: system.slice
Boot ID: eab09d88401447e88af95cf704e75710
Machine ID: 591d7280467543a2bd433dccc22b02ef
Hostname: host-8-246-82.host.centralci.eng.rdu2.redhat.com
Storage: /var/lib/systemd/coredump/core.pmdadm.0.eab09d88401447e88af95cf704e75710.21923.1549062905000000.lz4
Message: Process 21923 (pmdadm) of user 0 dumped core.
Stack trace of thread 21923:
#0 0x00007fa03046361a __GI_____strtoull_l_internal (libc.so.6)
#1 0x0000557a85bccbb1 vdo_fetch_ull (pmdadm)
#2 0x0000557a85bccd55 dm_vdodev_fetch (pmdadm)
#3 0x0000557a85bcb486 dm_fetchCallBack (pmdadm)
#4 0x00007fa030cf11f3 pmdaFetch (libpcp_pmda.so.3)
#5 0x0000557a85bcb780 dm_fetch (pmdadm)
#6 0x00007fa030cf4591 __pmdaMainPDU (libpcp_pmda.so.3)
#7 0x00007fa030cf4d38 pmdaMain (libpcp_pmda.so.3)
#8 0x0000557a85bcb2fc main (pmdadm)
#9 0x00007fa03044a813 __libc_start_main (libc.so.6)
#10 0x0000557a85bcb33e _start (pmdadm)
GNU gdb (GDB) Red Hat Enterprise Linux 8.2-5.el8
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /var/lib/pcp/pmdas/dm/pmdadm...Reading symbols from /usr/lib/debug/var/lib/pcp/pmdas/dm/pmdadm-4.3.0-3.el8.x86_64.debug...done.
done.
[New LWP 21923]
Missing separate debuginfo for /lib64/libpcp_pmda.so.3
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/94/092edb4682aad2b939797b05ff3e542bbd8656.debug
warning: Loadable section ".note.gnu.property" outside of ELF segments
Missing separate debuginfo for /lib64/libpcp.so.3
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/3b/3f3f0db700ab806c88b905e0e7f8da9787f4d6.debug
warning: Loadable section ".note.gnu.property" outside of ELF segments
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/var/lib/pcp/pmdas/dm/pmdadm -d 129'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fa03046361a in __GI_____strtoul_l_internal (nptr=0x0, endptr=endptr@entry=0x7fffb41d2350, base=base@entry=10, group=group@entry=0, loc=0x7fa0307e6580 <_nl_global_locale>) at ../stdlib/strtol_l.c:283
283 if (base < 0 || base == 1 || base > 36)
--- Additional comment from Dennis Keefe on 2019-02-01 23:40 UTC ---
Core dump from process 21923 of pmdadm
kernel: 4.18.0-64.el8.x86_64
pcp-selinux-4.3.0-3.el8.x86_64
pcp-pmda-dm-4.3.0-3.el8.x86_64
pcp-debuginfo-4.3.0-3.el8.x86_64
pcp-conf-4.3.0-3.el8.x86_64
pcp-4.3.0-3.el8.x86_64
pcp-debugsource-4.3.0-3.el8.x86_64
pcp-pmda-dm-debuginfo-4.3.0-3.el8.x86_64
pcp-libs-debuginfo-4.3.0-3.el8.x86_64
pcp-libs-4.3.0-3.el8.x86_64
--- Additional comment from Nathan Scott on 2019-02-07 03:22:33 UTC ---
Fixed upstream now (thanks for access to your system, Dennis!)
Author: Nathan Scott <nathans>
Date: Thu Feb 7 14:21:29 2019 +1100
pmdadm: add new metrics, fix handling of missing kernel stats
Several VDO metrics were added, some were moved and their kernel
statistics files went away. This exposed a class of bug in the
way the PMDA handled missing files, causing a sigsegv.
Update qa/1188 to exercise the new style of VDO metrics also.
Resolves Red Hat BZ #1670548.
--- Additional comment from Dennis Keefe on 2019-02-07 16:23:16 UTC ---
Thank you. If possible, we could really use this fix back-ported to 8.0.1.
This will allow users to collect all VDO stats without running into problems.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2019:3364