Bug 2275408 - pmlogger.service fails to start with missing /var/log directories
Summary: pmlogger.service fails to start with missing /var/log directories
Keywords:
Status: ASSIGNED
Alias: None
Product: Fedora
Classification: Fedora
Component: pcp
Version: 39
Hardware: Unspecified
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Nathan Scott
QA Contact: Fedora Extras Quality Assurance
URL: https://cockpit-logs.us-east-1.linode...
Whiteboard: CockpitTest
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-04-17 05:44 UTC by Martin Pitt
Modified: 2024-04-23 10:08 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Martin Pitt 2024-04-17 05:44:56 UTC
The recent update to pcp-6.2.1-1.fc39 [1] has caused a regression in the service's data initialization, which now causes pmlogger.service to crash.

[1] https://bodhi.fedoraproject.org/updates/FEDORA-2024-4c09d22f7e

Reproducible: Always

Steps to Reproduce:
dnf update --enablerepo=updates-testing pcp
systemctl stop pmlogger
rm -rf /var/log/pcp/pmlogger
systemctl start pmlogger
Actual Results:  
Starting pmlogger.service - Performance Metrics Archive Logger...
/etc/pcp/pmlogger/rc: Warning: Performance Co-Pilot archive logger(s) not permanently enabled.
    To enable pmlogger, run the following as root:
    # /usr/bin/systemctl enable pmlogger.service
mv: cannot move '/var/tmp/pmlogger_rc_start.oWntjy6pf/pmcheck' to '/var/log/pcp/pmlogger/pmlogger_check.log': No such file or directory
pmlogger.service: Failed with result 'protocol'.
Failed to start pmlogger.service - Performance Metrics Archive Logger.


Expected Results:  
pmlogger creates its log directory like before (possibly using systemd-tmpfiles)

Comment 1 Nathan Scott 2024-04-17 06:06:25 UTC
| rm -rf /var/log/pcp/pmlogger

Hmm?  That directory is an installed part of the pcp rpm, we can't just blow it away and expect everything to be fine.

It did work previously, but that was just good fortune thanks to optimistically helpful PCP start scripts.  Unfortunately, for security reasons (CVE-2023-6917) we are no longer are able to be quite so helpful anymore.

We probably want something more like this in Cockpit CI:

rm -rf /var/log/pcp/pmlogger/*

I guess?  Or use "find | xargs rm" or something more along those lines.

Comment 2 Martin Pitt 2024-04-17 06:12:08 UTC
Yes, that's what I did in https://github.com/cockpit-project/cockpit/pull/20328 and this isn't major/critical, but more like minor. But I still consider it a good behaviour of services to be able to deal with a completely empty /var/log (or /var/tmp/ etc.). That's why systemd-tmpfiles exists, to support a "system reset". pcp already makes extensive usage of that, so it'd be really nice to add the log directories there as well.

Comment 3 Nathan Scott 2024-04-17 06:51:10 UTC
+1 ... I'll add it to the to-do list, thanks Martin.

Comment 4 Jan Kurik 2024-04-18 08:37:44 UTC
Today I just realized this is an issue with PCP container. Before it was possible to map /var/log/pcp to a PCP container at once. i.e.:
"-v /var/tmp/pcp-log:/var/log/pcp"

Now it is necessary to map every sub-directory in /var/log/pcp explicitly.i.e.
"-v /var/tmp/pcp-log/pmcd:/var/log/pcp/pmcd -v /var/tmp/pcp-log/pmfind:/var/log/pcp/pmfind -v /var/tmp/pcp-log/pmie:/var/log/pcp/pmie -v /var/tmp/pcp-log/pmlogger:/var/log/pcp/pmlogger -v /var/tmp/pcp-log/pmproxy:/var/log/pcp/pmproxy"

This can be seen as an regression.

Comment 5 Nathan Scott 2024-04-18 23:51:02 UTC
| This can be seen as an regression.

Chatting to Jan last night, he tried it out and found the tmpfiles.d entries solve this.

Comment 6 Nathan Scott 2024-04-23 10:08:57 UTC
Fixed upstream by commit below:

commit 80609476a0357f7f7d1479439967c7bdaaaa0a54
Author: Nathan Scott <nathans>
Date:   Tue Apr 23 16:48:43 2024 +1000

    tmpfiles.d: extend the tmpfiles.d directory set to logdirs
    
    Resolves Fedora BZ #2275408


Note You need to log in before you can comment on or make changes to this bug.