RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1780003 - pmie does not shut-down gracefully
Summary: pmie does not shut-down gracefully
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: pcp
Version: 8.2
Hardware: All
OS: Linux
unspecified
low
Target Milestone: rc
: 8.0
Assignee: Mark Goodwin
QA Contact: Jan Kurik
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-12-05 08:40 UTC by Jan Kurik
Modified: 2023-12-15 17:02 UTC (History)
9 users (show)

Fixed In Version: pcp-5.0.2
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-04-28 15:40:22 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-30900 0 None None None 2023-10-06 18:57:13 UTC
Red Hat Product Errata RHBA-2020:1628 0 None None None 2020-04-28 15:41:05 UTC

Internal Links: 1806428

Description Jan Kurik 2019-12-05 08:40:07 UTC 
Description of problem:
When shutting down PMIE using "systemctl" command, the final status is "failed" instead of "inactive". 

Version-Release number of selected component (if applicable):
PCP: pcp-5.0.1-1.el8
RHEL: RHEL-8.2.0-20191203.0

How reproducible:
Always (on all supported platforms)

Steps to Reproduce:
1. Install PCP on RHEL-8
2. Start PMCD and PMIE using command "systemctl start pmcd pmie"
3. Stop PMIE using command "systemctl stop pmie"
4. Check the status of PMIE using command "systemctl status pmie"

Actual results:
Status is:
    Active: failed (Result: exit-code)

Expected results:
Status is:
    Active: inactive

Additional info:
Full output of "systemctl status pmie":

● pmie.service - Performance Metrics Inference Engine
   Loaded: loaded (/usr/lib/systemd/system/pmie.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2019-12-04 10:21:54 EST; 20min ago
     Docs: man:pmie(1)
  Process: 107871 ExecStop=/usr/share/pcp/lib/pmie stop (code=exited, status=0/SUCCESS)
  Process: 107403 ExecStart=/usr/share/pcp/lib/pmie start (code=exited, status=0/SUCCESS)
 Main PID: 107760 (code=exited, status=15)

Dec 04 10:21:54 ibm-p8-firestone-02.pnr.lab.eng.rdu2.redhat.com pmie[107403]:     # /usr/bin/systemctl enable pmie.service
Dec 04 10:21:54 ibm-p8-firestone-02.pnr.lab.eng.rdu2.redhat.com pmie[107403]: Starting pmie ...
Dec 04 10:21:54 ibm-p8-firestone-02.pnr.lab.eng.rdu2.redhat.com systemd[1]: pmie.service: Can't open PID file /run/pcp/pmie.pid (yet?) after start: No such file or directory
Dec 04 10:21:54 ibm-p8-firestone-02.pnr.lab.eng.rdu2.redhat.com systemd[1]: pmie.service: Supervising process 107760 which is not our child. We'll most likely not notice when it exits.
Dec 04 10:21:54 ibm-p8-firestone-02.pnr.lab.eng.rdu2.redhat.com systemd[1]: Started Performance Metrics Inference Engine.
Dec 04 10:21:54 ibm-p8-firestone-02.pnr.lab.eng.rdu2.redhat.com systemd[1]: Stopping Performance Metrics Inference Engine...
Dec 04 10:21:54 ibm-p8-firestone-02.pnr.lab.eng.rdu2.redhat.com systemd[1]: pmie.service: Main process exited, code=exited, status=15/n/a
Dec 04 10:21:54 ibm-p8-firestone-02.pnr.lab.eng.rdu2.redhat.com pmie[107871]: Waiting for pmie process(es) to terminate ...
Dec 04 10:21:54 ibm-p8-firestone-02.pnr.lab.eng.rdu2.redhat.com systemd[1]: pmie.service: Failed with result 'exit-code'.
Dec 04 10:21:54 ibm-p8-firestone-02.pnr.lab.eng.rdu2.redhat.com systemd[1]: Stopped Performance Metrics Inference Engine.



I have been trying to collect more information increasing verbosity level of systemd and journald to "debug". Unfortunately no more info, related to this case, has been captured. Please let me know if there is anything I can do more to provide you with relevant information.
Comment 1 Mark Goodwin 2019-12-06 01:40:55 UTC 
The systemd service file calls /usr/share/pcp/lib/pmie stop  to shutdown the service. That script calls pmie_check -s .. which fails to kill the process (reason unknown at this stage). So after a timeout, systemd kills it with signal 15 (SIGTERM) and reports it as 'failed' rather than 'inactive'.

Adding this BZ to the long list of service/rc/script updates planned for pcp-5.0.3 or pcp-5.1.0 or later (not just for pmie, but pmlogger and other services too).
Comment 2 Mark Goodwin 2019-12-09 07:13:52 UTC 
On receipt of a signal 15 (SIGTERM), pmie exits with status 15, despite this being a "normal" exit as killed by pmie_check. Systemd considers this a failure exit status and reports it as such.

This patch fixes it, but will need more testing in case there are any other ramifications of changing the pmie exit status.

diff --git a/src/pmie/src/dstruct.c b/src/pmie/src/dstruct.c
index a5c2c6e1e..63c753ccf 100644
--- a/src/pmie/src/dstruct.c
+++ b/src/pmie/src/dstruct.c
@@ -262,7 +262,7 @@ sleepTight(Task *t)
                sts = nanosleep(&ts, &tleft);
                /* deferred signal handling done immediately */
                if (doexit)
-                   exit(doexit);
+                   exit(doexit == 15 ? 0 : doexit);
                if (dorotate) {
                    logRotate();
                    dorotate = 0;
Comment 6 Mark Goodwin 2019-12-09 22:12:51 UTC 
Resolved upstream for pcp-5.0.2

commit c0211c039e6f3714c0a6534bc213aad4f0c359d5 (HEAD -> master, upstream-master/master, upstream-goodwinos/master)
Author: Mark Goodwin <mgoodwin>
Date:   Tue Dec 10 08:56:49 2019 +1100

    pmie: on SIGTERM, exit 0 for a normal shutdown rather than exit 15
    
    On receipt of a SIGTERM (15) as issued by pmie_check -s, pmie exits
    with status 15, which systemd considers a failure. Exit 0 in this
    case for a normal shutdown.
    
    Resolves #807
    Resolves RHBZ#1780003
Comment 11 errata-xmlrpc 2020-04-28 15:40:22 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1628
Note You need to log in before you can comment on or make changes to this bug.