Bug 1929556

Right after boot, pmlogger.service is not running, and /var/log/pcp/pmlogger/ just contains an empty .NeedRewrite stamp file.

Unfortunately pcp doesn't really use systemd, so drilling down into the startup scripts:

# sh -x /usr/libexec/pcp/lib/pmlogger start-systemd
[...lots of relatively uninteresting startup/variable init noise ...]
+ pmlogger_check --quick -V
# sits here for some 10s
 bgstatus=0
+ '[' -s /var/tmp/pmlogger_rc_start.2TxNhEGJu/pmcheck ']'
+ return 0
+ estatus=true
+ '[' start-systemd = start-systemd ']'
+ true
+ _RC_STATUS
+ _rc_status=0
+ '[' '' = -v ']'
+ return 0
+ status=0
+ false
+ rm -rf /var/tmp/pmlogger_rc.sMfd2t9AE /var/tmp/pmlogger_rc_start.2TxNhEGJu
+ exit 0


After that, there is *no* pcp/pm* related process running, so despite claiming success (exit 0), it did not really start something. As systemd never gets a notify, it eventually declares the unit failed as there is no process left.

I now have a new log file:

# cat /var/log/pcp/pmlogger/pmlogger_check.log
pmlogger_check [/etc/pcp/pmlogger/control.d/local:26]
Warning: creating directory (/var/log/pcp/pmlogger/m1.cockpit.lan) for PCP archive files
Wed Feb 17 13:17:48 EST 2021
Arrgghhh ... pmcd at localhost failed to start after 5 seconds
=== failing pmprobes ===
pmprobe: Cannot connect to PMCD on host "local:": Connection refused

It also created /var/log/pcp/pmlogger/m1.cockpit.lan/ but that is empty.

pmcd.service is indeed not running. So trying this again:

systemctl start pmcd  # succeeds
systemctl start pmlogger

This now succees as well.

So apparently the change is that pmlogger now started to depend on pmcd.service (it was not before), but does not declare that dependency with Requires/After=pmcd.service ?

(In reply to Martin Pitt from comment #5)
> So apparently the change is that pmlogger now started to depend on
> pmcd.service (it was not before)

Are you using the default pmlogger config, which includes a "primary logger" which connects to a local pmcd (configured in /etc/pcp/pmlogger/control.d/local)?
If so, then you need to start pmcd before pmlogger, because pmlogger is configured to connect and log metrics from the local pmcd.

> but does not declare that dependency with
> Requires/After=pmcd.service ?

pmlogger in general does not depend on a local pmcd, that's why we've removed the Require= systemd dependency (it used to be there before).
For example, if you want to log remote pmcds only, there is no need to have a local pmcd daemon running. See upstream discussion [1] and the PR [2] for more context.

fwiw, we still have a After=pmcd.service dependency in pmlogger.service, to make sure that pmlogger starts after pmcd. But there is no Require= anymore, because it depends on the pmlogger configuration if a local pmcd should be running or not.


[1] https://github.com/performancecopilot/pcp/issues/1200
[2] https://github.com/performancecopilot/pcp/pull/1206

(In reply to Andreas Gerstmayr from comment #6)
> Are you using the default pmlogger config, which includes a "primary logger"
> which connects to a local pmcd (configured in
> /etc/pcp/pmlogger/control.d/local)?

This is a bog standard "dnf install pcp on a cloud image" install, I didn't touch any pcp config. Here it is:

# cat /etc/pcp/pmlogger/control.d/local 
#
# PCP archive logging configuration/control
#
# see ../control for a description of the format
#

# === VARIABLE ASSIGNMENTS ===
#
# DO NOT REMOVE OR EDIT THE FOLLOWING LINE
$version=1.1

# Uncomment one of the lines below to enable/disable compression behaviour
# that is different to the pmlogger_daily default.
# Value is days before compressing archives, 0 is immediate compression,
# "never" or "forever" suppresses compression.
#
#$PCP_COMPRESSAFTER=0
#$PCP_COMPRESSAFTER=3
#$PCP_COMPRESSAFTER=never

# === LOGGER CONTROL SPECIFICATIONS ===
#
#Host		P?  S?	directory			args

# local primary logger
LOCALHOSTNAME	y   n	PCP_ARCHIVE_DIR/LOCALHOSTNAME	-r -T24h10m -c config.default -v 100Mb

That's exactly the file that is shipped in pcp.rpm.

> If so, then you need to start pmcd before pmlogger, because pmlogger is
> configured to connect and log metrics from the local pmcd.

If "you" is "me", then no -- that is just plain a regression. With my user's hat on, I don't really care what "pmcd" is, I just want pcp to collect local metrics. I started/enabled pmlogger.service, upgrade to 8.4, and it starts to  fail.

If "you" means "pcp's startup scripts", then yes, that would do it.

> pmlogger in general does not depend on a local pmcd, that's why we've
> removed the Require= systemd dependency (it used to be there before).

That does seem more efficient indeed. So what I'd suggest is:

 * Replace the static Requires=pmcd with a dynamic check in /usr/libexec/pcp/lib/pmlogger that starts pmcd.service if necessary (with systemctl --no-block, as you are in the middle of a systemd transaction)
 * Reconsider if the default config should really require pmcd if you want it off by default.

Thanks for clarifying!

After chatting with Andreas today, we're going to back out the (one-line) unit file change and rethink our options in the 8.5 timeframe.

These two BZs are one and the same problem, viewed from two different angles (via pmie and pmlogger - its one patch and one git-revert to fix both BZs.

Since this will now need to go through the exception review process for 8.4 I'm streamlining this into one BZ to save everyone time.

*** This bug has been marked as a duplicate of bug 1930284 ***