Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1670548

Summary:

PCP stats for VDO need to be updated

Product:

Red Hat Enterprise Linux 8

Reporter:

Dennis Keefe <dkeefe>

Component:

pcp

Assignee:

Nathan Scott <nathans>

Status:

CLOSED ERRATA

QA Contact:

Michal Kolar <mkolar>

Severity:

low

Docs Contact:

Priority:

unspecified

Version:

8.1

CC:

bgollahe, fche, lberk, mcermak, mgoodwin, mkolar, mnewsome, nathans, patrickm, toneata

Target Milestone:

Keywords:

ZStream

Target Release:

8.0

Flags:

pm-rhel: mirror+

Hardware:

All

OS:

Linux

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Clones:

1699377 (view as bug list)

Environment:

Last Closed:

2019-06-18 17:20:27 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

1699377, 1701002

Attachments:

Description	Flags
Core dump	none

Description Dennis Keefe 2019-01-29 19:17:42 UTC

Description of problem:
The following PCP stats for VDO return a message of
 "pmval: pmFetch: IPC protocol failure"

PCP Stat Name                       

vdo.dev.curr_dedupe_queries  
vdo.dev.max_dedupe_queries  
vdo.dev.dedupe_advice.stale   
vdo.dev.dedupe_advice.valid  
vdo.dev.read_cache.accesses 
vdo.dev.read_cache.data_hits 
vdo.dev.read_cache.hits  


Four of the VDO stat names have been changed this might be the reason why 
they fail.


PCP Stat Name                        New VDO stat name

vdo.dev.curr_dedupe_queries     index_max_dedupe_queries
vdo.dev.max_dedupe_queries      index_max_dedupe_queries
vdo.dev.dedupe_advice.stale     hash_lock_dedupe_advice_stale
vdo.dev.dedupe_advice.valid     hash_lock_dedupe_advice_valid


VDO's read cache was deprecated in 7.6, so these stats do not exist in RHEL8.
Consider removing them from PCP in RHEL 8+

vdo.dev.read_cache.accesses    - does not exist in RHEL 8
vdo.dev.read_cache.data_hits   - does not exist in RHEL 8
vdo.dev.read_cache.hits        - does not exist in RHEL 8


Example of the error message

# pmval -v  -s 1 vdo.dev.curr_dedupe_queries

metric:    vdo.dev.curr_dedupe_queries
host:      host-8-243-15.host.centralci.eng.rdu2.redhat.com
semantics: instantaneous value
units:     count
samples:   1

pmval: pmFetch: IPC protocol failure


How to reproduce on RHEL 8:
1. yum -y install vdo pcp pcp-pmda-dm
2. cd /var/lib/pcp/pmdas/dm/
3. ./Install
4. vdo create --name=vdotest --device=/dev/sdb
5. pmval -v  -s 1 vdo.dev.curr_dedupe_queries

Comment 1 Dennis Keefe 2019-01-30 22:15:43 UTC

This issue exists in 8.0 and if possible, it would be helpful if a fix could be added to RHEL 8.0.

Comment 2 Nathan Scott 2019-01-30 22:21:06 UTC

(In reply to Dennis Keefe from comment #1)
> This issue exists in 8.0 and if possible, it would be helpful if a fix could
> be added to RHEL 8.0.

Understood - at this stage we don't intend to update PCP in 8.0 further, but 8.1 is no problem.

cheers.

Comment 3 Dennis Keefe 2019-02-01 23:29:06 UTC

Also, if pmval is run 3 times for any of these stats it will throw a segfault

[root@host-8-246-82 coredump]# coredumpctl list
TIME                            PID   UID   GID SIG COREFILE  EXE
Fri 2019-02-01 18:11:45 EST   15444     0     0  11 present   /var/lib/pcp/pmdas/dm/pmdadm
Fri 2019-02-01 18:11:46 EST   21584     0     0  11 present   /var/lib/pcp/pmdas/dm/pmdadm
Fri 2019-02-01 18:15:00 EST   21911     0     0  11 present   /var/lib/pcp/pmdas/dm/pmdadm
Fri 2019-02-01 18:15:05 EST   21923     0     0  11 present   /var/lib/pcp/pmdas/dm/pmdad

[root@host-8-246-82 coredump]# coredumpctl gdb 21923
           PID: 21923 (pmdadm)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Fri 2019-02-01 18:15:05 EST (11min ago)
  Command Line: /var/lib/pcp/pmdas/dm/pmdadm -d 129
    Executable: /var/lib/pcp/pmdas/dm/pmdadm
 Control Group: /system.slice/pmcd.service
          Unit: pmcd.service
         Slice: system.slice
       Boot ID: eab09d88401447e88af95cf704e75710
    Machine ID: 591d7280467543a2bd433dccc22b02ef
      Hostname: host-8-246-82.host.centralci.eng.rdu2.redhat.com
       Storage: /var/lib/systemd/coredump/core.pmdadm.0.eab09d88401447e88af95cf704e75710.21923.1549062905000000.lz4
       Message: Process 21923 (pmdadm) of user 0 dumped core.
                
                Stack trace of thread 21923:
                #0  0x00007fa03046361a __GI_____strtoull_l_internal (libc.so.6)
                #1  0x0000557a85bccbb1 vdo_fetch_ull (pmdadm)
                #2  0x0000557a85bccd55 dm_vdodev_fetch (pmdadm)
                #3  0x0000557a85bcb486 dm_fetchCallBack (pmdadm)
                #4  0x00007fa030cf11f3 pmdaFetch (libpcp_pmda.so.3)
                #5  0x0000557a85bcb780 dm_fetch (pmdadm)
                #6  0x00007fa030cf4591 __pmdaMainPDU (libpcp_pmda.so.3)
                #7  0x00007fa030cf4d38 pmdaMain (libpcp_pmda.so.3)
                #8  0x0000557a85bcb2fc main (pmdadm)
                #9  0x00007fa03044a813 __libc_start_main (libc.so.6)
                #10 0x0000557a85bcb33e _start (pmdadm)

GNU gdb (GDB) Red Hat Enterprise Linux 8.2-5.el8
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /var/lib/pcp/pmdas/dm/pmdadm...Reading symbols from .gnu_debugdata for /var/lib/pcp/pmdas/dm/pmdadm...(no debugging symbols found)...done.
(no debugging symbols found)...done.
[New LWP 21923]

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments
Core was generated by `/var/lib/pcp/pmdas/dm/pmdadm -d 129'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fa03046361a in ____strtoull_l_internal () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install pcp-pmda-dm-4.3.0-3.el8.x86_64

Comment 4 Dennis Keefe 2019-02-01 23:32:48 UTC

and after installing a bunch of debug packages, here is the output.

[root@host-8-246-82 coredump]# coredumpctl gdb 21923
           PID: 21923 (pmdadm)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Fri 2019-02-01 18:15:05 EST (16min ago)
  Command Line: /var/lib/pcp/pmdas/dm/pmdadm -d 129
    Executable: /var/lib/pcp/pmdas/dm/pmdadm
 Control Group: /system.slice/pmcd.service
          Unit: pmcd.service
         Slice: system.slice
       Boot ID: eab09d88401447e88af95cf704e75710
    Machine ID: 591d7280467543a2bd433dccc22b02ef
      Hostname: host-8-246-82.host.centralci.eng.rdu2.redhat.com
       Storage: /var/lib/systemd/coredump/core.pmdadm.0.eab09d88401447e88af95cf704e75710.21923.1549062905000000.lz4
       Message: Process 21923 (pmdadm) of user 0 dumped core.
                
                Stack trace of thread 21923:
                #0  0x00007fa03046361a __GI_____strtoull_l_internal (libc.so.6)
                #1  0x0000557a85bccbb1 vdo_fetch_ull (pmdadm)
                #2  0x0000557a85bccd55 dm_vdodev_fetch (pmdadm)
                #3  0x0000557a85bcb486 dm_fetchCallBack (pmdadm)
                #4  0x00007fa030cf11f3 pmdaFetch (libpcp_pmda.so.3)
                #5  0x0000557a85bcb780 dm_fetch (pmdadm)
                #6  0x00007fa030cf4591 __pmdaMainPDU (libpcp_pmda.so.3)
                #7  0x00007fa030cf4d38 pmdaMain (libpcp_pmda.so.3)
                #8  0x0000557a85bcb2fc main (pmdadm)
                #9  0x00007fa03044a813 __libc_start_main (libc.so.6)
                #10 0x0000557a85bcb33e _start (pmdadm)

GNU gdb (GDB) Red Hat Enterprise Linux 8.2-5.el8
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /var/lib/pcp/pmdas/dm/pmdadm...Reading symbols from /usr/lib/debug/var/lib/pcp/pmdas/dm/pmdadm-4.3.0-3.el8.x86_64.debug...done.
done.
[New LWP 21923]
Missing separate debuginfo for /lib64/libpcp_pmda.so.3
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/94/092edb4682aad2b939797b05ff3e542bbd8656.debug

warning: Loadable section ".note.gnu.property" outside of ELF segments
Missing separate debuginfo for /lib64/libpcp.so.3
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/3b/3f3f0db700ab806c88b905e0e7f8da9787f4d6.debug

warning: Loadable section ".note.gnu.property" outside of ELF segments
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/var/lib/pcp/pmdas/dm/pmdadm -d 129'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fa03046361a in __GI_____strtoul_l_internal (nptr=0x0, endptr=endptr@entry=0x7fffb41d2350, base=base@entry=10, group=group@entry=0, loc=0x7fa0307e6580 <_nl_global_locale>) at ../stdlib/strtol_l.c:283
283	  if (base < 0 || base == 1 || base > 36)

Comment 5 Dennis Keefe 2019-02-01 23:40:40 UTC

Created attachment 1526106 [details]
Core dump

Core dump from process 21923 of pmdadm
kernel: 4.18.0-64.el8.x86_64

pcp-selinux-4.3.0-3.el8.x86_64
pcp-pmda-dm-4.3.0-3.el8.x86_64
pcp-debuginfo-4.3.0-3.el8.x86_64
pcp-conf-4.3.0-3.el8.x86_64
pcp-4.3.0-3.el8.x86_64
pcp-debugsource-4.3.0-3.el8.x86_64
pcp-pmda-dm-debuginfo-4.3.0-3.el8.x86_64
pcp-libs-debuginfo-4.3.0-3.el8.x86_64
pcp-libs-4.3.0-3.el8.x86_64

Comment 6 Nathan Scott 2019-02-07 03:22:33 UTC

Fixed upstream now (thanks for access to your system, Dennis!)

Author: Nathan Scott <nathans>
Date:   Thu Feb 7 14:21:29 2019 +1100

    pmdadm: add new metrics, fix handling of missing kernel stats
    
    Several VDO metrics were added, some were moved and their kernel
    statistics files went away.  This exposed a class of bug in the
    way the PMDA handled missing files, causing a sigsegv.
    
    Update qa/1188 to exercise the new style of VDO metrics also.
    
    Resolves Red Hat BZ #1670548.

Comment 7 Dennis Keefe 2019-02-07 16:23:16 UTC

Thank you.  If possible, we could really use this fix back-ported to 8.0.1.
This will allow users to collect all VDO stats without running into problems.

Comment 12 Michal Kolar 2019-05-30 11:51:18 UTC

Reproduced against pcp-4.3.0-3.el8 and verified against pcp-4.3.0-4.el8_0.

Comment 14 errata-xmlrpc 2019-06-18 17:20:27 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1525