Bug 2298058

Summary: netdata-1.45.6-1 systemd-journal.plugin cant access journal
Product: [Fedora] Fedora Reporter: Craig Donnelly <cdonnell>
Component: netdataAssignee: Didier Fabert (tartare) <didier.fabert>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 40CC: didier.fabert, go-sig
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: netdata-1.47.0-1.fc40 netdata-1.47.0-1.fc39 netdata-1.47.0-1.el9 netdata-1.47.0-4.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-09-04 02:23:01 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Craig Donnelly 2024-07-16 01:35:48 UTC
After installing netdata, starting the service results in spam from failure to open a journal path (that appears to have been created by this service).

I get the same behavior on multiple F40 systems.

root@srv:~# rpm -q netdata
netdata-1.45.6-1.fc40.x86_64

root@srv:~# systemctl start netdata.service

root@srv:~# systemctl status netdata.service -l --no-pager
● netdata.service - Real time performance monitoring
     Loaded: loaded (/usr/lib/systemd/system/netdata.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (running) since Mon 2024-07-15 07:51:35 EDT; 1min 37s ago
    Process: 322999 ExecStartPre=/bin/mkdir -p /var/cache/netdata (code=exited, status=0/SUCCESS)
    Process: 323001 ExecStartPre=/bin/chown -R netdata /var/cache/netdata (code=exited, status=0/SUCCESS)
    Process: 323003 ExecStartPre=/bin/mkdir -p /run/netdata (code=exited, status=0/SUCCESS)
    Process: 323005 ExecStartPre=/bin/chown -R netdata /run/netdata (code=exited, status=0/SUCCESS)
   Main PID: 323007 (netdata)
      Tasks: 130 (limit: 19143)
     Memory: 96.4M (peak: 97.2M)
        CPU: 10.077s
     CGroup: /system.slice/netdata.service
             ├─323007 /usr/sbin/netdata -P /run/netdata/netdata.pid -D
             ├─323011 /usr/sbin/netdata --special-spawn-server
             ├─323219 /usr/libexec/netdata/plugins.d/apps.plugin 1
             ├─323226 /usr/bin/bash /usr/libexec/netdata/plugins.d/tc-qos-helper.sh 1
             ├─323257 /usr/libexec/netdata/plugins.d/network-viewer.plugin 1
             ├─323259 /usr/libexec/netdata/plugins.d/nfacct.plugin 1
             ├─323279 /usr/libexec/netdata/plugins.d/go.d.plugin 1
             └─323284 /usr/libexec/netdata/plugins.d/systemd-journal.plugin 1

Jul 15 07:53:11 srv systemd-journal.plugin[323284]: JOURNAL: cannot open file '/var/log/journal/00f22a4ba90941658197e67eac79f138.netdata/system.journal' to update msg_ut
Jul 15 07:53:11 srv systemd-journal.plugin[323284]: JOURNAL: cannot open file '/var/log/journal/00f22a4ba90941658197e67eac79f138.netdata/system.journal' to update msg_ut
Jul 15 07:53:11 srv systemd-journal.plugin[323284]: JOURNAL: cannot open file '/var/log/journal/00f22a4ba90941658197e67eac79f138.netdata/system.journal' to update msg_ut
Jul 15 07:53:11 srv systemd-journal.plugin[323284]: JOURNAL: cannot open file '/var/log/journal/00f22a4ba90941658197e67eac79f138.netdata/system.journal' to update msg_ut
Jul 15 07:53:12 srv systemd-journal.plugin[323284]: JOURNAL: cannot open file '/var/log/journal/00f22a4ba90941658197e67eac79f138.netdata/system.journal' to update msg_ut
Jul 15 07:53:12 srv systemd-journal.plugin[323284]: JOURNAL: cannot open file '/var/log/journal/00f22a4ba90941658197e67eac79f138.netdata/system.journal' to update msg_ut
Jul 15 07:53:12 srv systemd-journal.plugin[323284]: JOURNAL: cannot open file '/var/log/journal/00f22a4ba90941658197e67eac79f138.netdata/system.journal' to update msg_ut
Jul 15 07:53:12 srv systemd-journal.plugin[323284]: JOURNAL: cannot open file '/var/log/journal/00f22a4ba90941658197e67eac79f138.netdata/system.journal' to update msg_ut
Jul 15 07:53:13 srv systemd-journal.plugin[323284]: JOURNAL: cannot open file '/var/log/journal/00f22a4ba90941658197e67eac79f138.netdata/system.journal' to update msg_ut
Jul 15 07:53:13 srv systemd-journal.plugin[323284]: JOURNAL: cannot open file '/var/log/journal/00f22a4ba90941658197e67eac79f138.netdata/system.journal' to update msg_ut

root@srv:~# ll /var/log/journal/
total 16
drwxr-sr-x+ 2 root systemd-journal 12288 Jul 11 19:18 00f22a4ba90941658197e67eac79f138
drwxr-sr-x+ 2 root systemd-journal    28 Jul 15 07:15 00f22a4ba90941658197e67eac79f138.netdata

root@srv:~# ll /var/log/journal/00f22a4ba90941658197e67eac79f138.netdata/
total 8196
-rw-r-----+ 1 root systemd-journal 8388608 Jul 15 07:55 system.journal



Reproducible: Always

Steps to Reproduce:
1. Install netdata
2. `systemctl enable --now netdata`
3. `systemctl status netdata`
Actual Results:  
Error message spams constantly, multiple times per second in the unit status.

Jul 15 07:53:13 srv systemd-journal.plugin[323284]: JOURNAL: cannot open file '/var/log/journal/00f22a4ba90941658197e67eac79f138.netdata/system.journal' to update msg_ut

Expected Results:  
No systemd-journal.plugin errors.

Comment 1 Didier Fabert (tartare) 2024-08-22 07:24:31 UTC
Hi Craig,

I can confirm the problem and the fix is very easy

sudo usermod -aG systemd-journal netdata

Next netdata packages will include this fix (unfortunately netdata-1.46.3-4 is already in testing repo)

Thanks for your report

Comment 2 Fedora Update System 2024-08-26 14:16:43 UTC
FEDORA-2024-21fa367bbc (netdata-1.47.0-1.fc39) has been submitted as an update to Fedora 39.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-21fa367bbc

Comment 3 Fedora Update System 2024-08-26 14:16:47 UTC
FEDORA-2024-03f5e57063 (netdata-1.47.0-1.fc40) has been submitted as an update to Fedora 40.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-03f5e57063

Comment 4 Fedora Update System 2024-08-26 14:16:52 UTC
FEDORA-EPEL-2024-2480bf2932 (netdata-1.47.0-4.el8) has been submitted as an update to Fedora EPEL 8.
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2024-2480bf2932

Comment 5 Fedora Update System 2024-08-26 14:16:57 UTC
FEDORA-EPEL-2024-9335483c08 (netdata-1.47.0-1.el9) has been submitted as an update to Fedora EPEL 9.
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2024-9335483c08

Comment 6 Fedora Update System 2024-08-27 02:34:21 UTC
FEDORA-2024-21fa367bbc has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf install --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-21fa367bbc \*`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-21fa367bbc

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 7 Fedora Update System 2024-08-27 03:49:03 UTC
FEDORA-EPEL-2024-9335483c08 has been pushed to the Fedora EPEL 9 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2024-9335483c08

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 8 Fedora Update System 2024-08-27 04:03:06 UTC
FEDORA-EPEL-2024-2480bf2932 has been pushed to the Fedora EPEL 8 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2024-2480bf2932

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 9 Fedora Update System 2024-08-27 16:46:49 UTC
FEDORA-2024-03f5e57063 has been pushed to the Fedora 40 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf install --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-03f5e57063 \*`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-03f5e57063

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 10 Fedora Update System 2024-09-04 02:23:01 UTC
FEDORA-2024-03f5e57063 (netdata-1.47.0-1.fc40) has been pushed to the Fedora 40 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 11 Fedora Update System 2024-09-04 03:53:14 UTC
FEDORA-2024-21fa367bbc (netdata-1.47.0-1.fc39) has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 12 Fedora Update System 2024-09-04 04:03:19 UTC
FEDORA-EPEL-2024-9335483c08 (netdata-1.47.0-1.el9) has been pushed to the Fedora EPEL 9 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 13 Fedora Update System 2024-09-04 04:17:03 UTC
FEDORA-EPEL-2024-2480bf2932 (netdata-1.47.0-4.el8) has been pushed to the Fedora EPEL 8 stable repository.
If problem still persists, please make note of it in this bug report.