RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2043495 - abrt-xorg.service fails to start with error Cannot read journal data
Summary: abrt-xorg.service fails to start with error Cannot read journal data
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: abrt
Version: 8.5
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: rc
: ---
Assignee: abrt
QA Contact: CS System Management SST QE
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-21 11:10 UTC by Pavol Babinčák
Modified: 2023-06-30 08:17 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-06-30 08:17:33 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1433652 1 urgent CLOSED abrt-dump-journal-oops: Cannot read journal data 2022-12-13 15:12:02 UTC
Red Hat Bugzilla 1482230 1 None None None 2022-01-21 11:12:41 UTC
Red Hat Bugzilla 1804993 1 None None None 2022-01-21 11:12:41 UTC
Red Hat Issue Tracker RHELPLAN-109084 0 None None None 2022-01-21 11:13:19 UTC

Description Pavol Babinčák 2022-01-21 11:10:26 UTC 
Description of problem:
After some time abrt-dump-journal-xorg fails to start due to error Cannot read journal data.

Version-Release number of selected component (if applicable):
2.10.9-21.el8

How reproducible:
Always, possibly after corruption of journal.

Steps to Reproduce:
1. (re)start a service: sudo systemctl restart abrt-xorg.service
2. check if the service is running: systemctl status abrt-xorg.service

Actual results:
Service is "failed". In my case:

$ systemctl status abrt-xorg.service
● abrt-xorg.service - ABRT Xorg log watcher
   Loaded: loaded (/usr/lib/systemd/system/abrt-xorg.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2022-01-20 10:07:16 CET; 1 day 1h ago
 Main PID: 1719 (code=exited, status=1/FAILURE)

Jan 20 10:07:15 yumba systemd[1]: Started ABRT Xorg log watcher.
Jan 20 10:07:16 yumba abrt-dump-journal-xorg[1719]: abrt-dump-journal-xorg: Cannot read journal data.
Jan 20 10:07:16 yumba systemd[1]: abrt-xorg.service: Main process exited, code=exited, status=1/FAILURE
Jan 20 10:07:16 yumba systemd[1]: abrt-xorg.service: Failed with result 'exit-code'.


Expected results:
Service is "running", ignoring bad journal entries.

Additional info:

Getting ExecStart from /usr/lib/systemd/system/abrt-xorg.service with additional verbosity:

$ sudo /usr/bin/abrt-dump-journal-xorg -fxtD -vvv                                  
abrt-dump-journal-xorg: Loaded option 'MaxCrashReportsSize' = '5000'
abrt-dump-journal-xorg: Loaded option 'DeleteUploaded' = 'no'
abrt-dump-journal-xorg: Loaded option 'AutoreportingEvent' = 'report_uReport'
abrt-dump-journal-xorg: Loaded option 'AutoreportingEnabled' = 'no'
abrt-dump-journal-xorg: Loaded option 'MaxCrashReportsSize' = '5000'
abrt-dump-journal-xorg: Loaded option 'DeleteUploaded' = 'no'
abrt-dump-journal-xorg: Loaded option 'AutoreportingEvent' = 'report_uReport'
abrt-dump-journal-xorg: Loaded option 'AutoreportingEnabled' = 'no'
abrt-dump-journal-xorg: Loading settings from 'xorg.conf'
abrt-dump-journal-xorg: Loaded option 'BlacklistedXorgModules' = 'nvidia, fglrx, vboxvideo'
abrt-dump-journal-xorg: Loaded option 'JournalFilters' = '_COMM=gdm-x-session, _COMM=gnome-shell'
abrt-dump-journal-xorg: Loaded option 'BlacklistedXorgModules' = 'nvidia, fglrx, vboxvideo'
abrt-dump-journal-xorg: Loaded option 'JournalFilters' = '_COMM=gdm-x-session, _COMM=gnome-shell'
abrt-dump-journal-xorg: Loaded 'xorg.conf'
abrt-dump-journal-xorg: Using journal filter from conf file xorg.conf
abrt-dump-journal-xorg: Using journal match: '_COMM=gdm-x-session'
abrt-dump-journal-xorg: Using journal match: '_COMM=gnome-shell'
abrt-dump-journal-xorg: Failed to read 'MESSAGE' field: Bad message
abrt-dump-journal-xorg: Cannot read journal data.


Running it through ltrace I see last entries as:

strncpy(0x7ffe0746f2e0, "[308042:308075:1119/053235.521890:ERROR:connection_factory_impl.cc(429)] Failed to connect to MCS endpoint with error -106", 122) = 0x7ffe0746f2e0
strstr("[308042:308075:1119/053235.521890:ERROR:connection_factory_impl.cc(429)] Failed to connect to MCS endpoint with error -106", "Backtrace:") = nil
sd_journal_next(0x55b8f082eb50, 0x55b8eefe0ddc, 0x18000000000, 0x7ffe0746f35a)                                                    = 1
sd_journal_get_data(0x55b8f082eb50, 0x55b8eefe165e, 0x7ffe0746f2b0, 0x7ffe0746f2a8)                                               = 0xffffffb6
strerror(74)                                                                                                                      = "Bad message"


With command:

$ sudo journalctl -u abrt-xorg.service

I found that error appeared for the first time on November 19:

...

-- Reboot --
Nov 09 12:19:36 yumba systemd[1]: Started ABRT Xorg log watcher.
-- Reboot --
Nov 19 10:48:13 yumba systemd[1]: Started ABRT Xorg log watcher.
Nov 19 10:48:14 yumba abrt-dump-journal-xorg[1645]: abrt-dump-journal-xorg: Cannot read journal data.
Nov 19 10:48:14 yumba systemd[1]: abrt-xorg.service: Main process exited, code=exited, status=1/FAILURE
Nov 19 10:48:14 yumba systemd[1]: abrt-xorg.service: Failed with result 'exit-code'.

....

And I was able to narrow down actual journal file that is corrupted:


$ journalctl --verify --file /var/log/journal/be37cfed3f434d0cb8d83feafdfaa26b/user-18031~     
1314e88: Invalid entry item (20/25 offset: 000000                                                                                                                  
1314e88: Invalid object contents: Bad message                                                                                                                      
File corruption detected at /var/log/journal/be37cfed3f434d0cb8d83feafdfaa26b/user-18031~:1314e88 (of 25165824 bytes, 79%).
FAIL: /var/log/journal/be37cfed3f434d0cb8d83feafdfaa26b/user-18031~ (Bad message)

---

From what I understand in https://bugs.freedesktop.org/show_bug.cgi?id=64116 journal files which have been identified as corrupted are rotated and there is no plan to get those fixed nor there is a recommendation to remove them ("? Why would you want to throw-away the good parts in the journal files?").
Note You need to log in before you can comment on or make changes to this bug.