Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
After some time abrt-dump-journal-xorg fails to start due to error Cannot read journal data.
Version-Release number of selected component (if applicable):
2.10.9-21.el8
How reproducible:
Always, possibly after corruption of journal.
Steps to Reproduce:
1. (re)start a service: sudo systemctl restart abrt-xorg.service
2. check if the service is running: systemctl status abrt-xorg.service
Actual results:
Service is "failed". In my case:
$ systemctl status abrt-xorg.service
● abrt-xorg.service - ABRT Xorg log watcher
Loaded: loaded (/usr/lib/systemd/system/abrt-xorg.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2022-01-20 10:07:16 CET; 1 day 1h ago
Main PID: 1719 (code=exited, status=1/FAILURE)
Jan 20 10:07:15 yumba systemd[1]: Started ABRT Xorg log watcher.
Jan 20 10:07:16 yumba abrt-dump-journal-xorg[1719]: abrt-dump-journal-xorg: Cannot read journal data.
Jan 20 10:07:16 yumba systemd[1]: abrt-xorg.service: Main process exited, code=exited, status=1/FAILURE
Jan 20 10:07:16 yumba systemd[1]: abrt-xorg.service: Failed with result 'exit-code'.
Expected results:
Service is "running", ignoring bad journal entries.
Additional info:
Getting ExecStart from /usr/lib/systemd/system/abrt-xorg.service with additional verbosity:
$ sudo /usr/bin/abrt-dump-journal-xorg -fxtD -vvv
abrt-dump-journal-xorg: Loaded option 'MaxCrashReportsSize' = '5000'
abrt-dump-journal-xorg: Loaded option 'DeleteUploaded' = 'no'
abrt-dump-journal-xorg: Loaded option 'AutoreportingEvent' = 'report_uReport'
abrt-dump-journal-xorg: Loaded option 'AutoreportingEnabled' = 'no'
abrt-dump-journal-xorg: Loaded option 'MaxCrashReportsSize' = '5000'
abrt-dump-journal-xorg: Loaded option 'DeleteUploaded' = 'no'
abrt-dump-journal-xorg: Loaded option 'AutoreportingEvent' = 'report_uReport'
abrt-dump-journal-xorg: Loaded option 'AutoreportingEnabled' = 'no'
abrt-dump-journal-xorg: Loading settings from 'xorg.conf'
abrt-dump-journal-xorg: Loaded option 'BlacklistedXorgModules' = 'nvidia, fglrx, vboxvideo'
abrt-dump-journal-xorg: Loaded option 'JournalFilters' = '_COMM=gdm-x-session, _COMM=gnome-shell'
abrt-dump-journal-xorg: Loaded option 'BlacklistedXorgModules' = 'nvidia, fglrx, vboxvideo'
abrt-dump-journal-xorg: Loaded option 'JournalFilters' = '_COMM=gdm-x-session, _COMM=gnome-shell'
abrt-dump-journal-xorg: Loaded 'xorg.conf'
abrt-dump-journal-xorg: Using journal filter from conf file xorg.conf
abrt-dump-journal-xorg: Using journal match: '_COMM=gdm-x-session'
abrt-dump-journal-xorg: Using journal match: '_COMM=gnome-shell'
abrt-dump-journal-xorg: Failed to read 'MESSAGE' field: Bad message
abrt-dump-journal-xorg: Cannot read journal data.
Running it through ltrace I see last entries as:
strncpy(0x7ffe0746f2e0, "[308042:308075:1119/053235.521890:ERROR:connection_factory_impl.cc(429)] Failed to connect to MCS endpoint with error -106", 122) = 0x7ffe0746f2e0
strstr("[308042:308075:1119/053235.521890:ERROR:connection_factory_impl.cc(429)] Failed to connect to MCS endpoint with error -106", "Backtrace:") = nil
sd_journal_next(0x55b8f082eb50, 0x55b8eefe0ddc, 0x18000000000, 0x7ffe0746f35a) = 1
sd_journal_get_data(0x55b8f082eb50, 0x55b8eefe165e, 0x7ffe0746f2b0, 0x7ffe0746f2a8) = 0xffffffb6
strerror(74) = "Bad message"
With command:
$ sudo journalctl -u abrt-xorg.service
I found that error appeared for the first time on November 19:
...
-- Reboot --
Nov 09 12:19:36 yumba systemd[1]: Started ABRT Xorg log watcher.
-- Reboot --
Nov 19 10:48:13 yumba systemd[1]: Started ABRT Xorg log watcher.
Nov 19 10:48:14 yumba abrt-dump-journal-xorg[1645]: abrt-dump-journal-xorg: Cannot read journal data.
Nov 19 10:48:14 yumba systemd[1]: abrt-xorg.service: Main process exited, code=exited, status=1/FAILURE
Nov 19 10:48:14 yumba systemd[1]: abrt-xorg.service: Failed with result 'exit-code'.
....
And I was able to narrow down actual journal file that is corrupted:
$ journalctl --verify --file /var/log/journal/be37cfed3f434d0cb8d83feafdfaa26b/user-18031~
1314e88: Invalid entry item (20/25 offset: 000000
1314e88: Invalid object contents: Bad message
File corruption detected at /var/log/journal/be37cfed3f434d0cb8d83feafdfaa26b/user-18031~:1314e88 (of 25165824 bytes, 79%).
FAIL: /var/log/journal/be37cfed3f434d0cb8d83feafdfaa26b/user-18031~ (Bad message)
---
From what I understand in https://bugs.freedesktop.org/show_bug.cgi?id=64116 journal files which have been identified as corrupted are rotated and there is no plan to get those fixed nor there is a recommendation to remove them ("? Why would you want to throw-away the good parts in the journal files?").
Description of problem: After some time abrt-dump-journal-xorg fails to start due to error Cannot read journal data. Version-Release number of selected component (if applicable): 2.10.9-21.el8 How reproducible: Always, possibly after corruption of journal. Steps to Reproduce: 1. (re)start a service: sudo systemctl restart abrt-xorg.service 2. check if the service is running: systemctl status abrt-xorg.service Actual results: Service is "failed". In my case: $ systemctl status abrt-xorg.service ● abrt-xorg.service - ABRT Xorg log watcher Loaded: loaded (/usr/lib/systemd/system/abrt-xorg.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Thu 2022-01-20 10:07:16 CET; 1 day 1h ago Main PID: 1719 (code=exited, status=1/FAILURE) Jan 20 10:07:15 yumba systemd[1]: Started ABRT Xorg log watcher. Jan 20 10:07:16 yumba abrt-dump-journal-xorg[1719]: abrt-dump-journal-xorg: Cannot read journal data. Jan 20 10:07:16 yumba systemd[1]: abrt-xorg.service: Main process exited, code=exited, status=1/FAILURE Jan 20 10:07:16 yumba systemd[1]: abrt-xorg.service: Failed with result 'exit-code'. Expected results: Service is "running", ignoring bad journal entries. Additional info: Getting ExecStart from /usr/lib/systemd/system/abrt-xorg.service with additional verbosity: $ sudo /usr/bin/abrt-dump-journal-xorg -fxtD -vvv abrt-dump-journal-xorg: Loaded option 'MaxCrashReportsSize' = '5000' abrt-dump-journal-xorg: Loaded option 'DeleteUploaded' = 'no' abrt-dump-journal-xorg: Loaded option 'AutoreportingEvent' = 'report_uReport' abrt-dump-journal-xorg: Loaded option 'AutoreportingEnabled' = 'no' abrt-dump-journal-xorg: Loaded option 'MaxCrashReportsSize' = '5000' abrt-dump-journal-xorg: Loaded option 'DeleteUploaded' = 'no' abrt-dump-journal-xorg: Loaded option 'AutoreportingEvent' = 'report_uReport' abrt-dump-journal-xorg: Loaded option 'AutoreportingEnabled' = 'no' abrt-dump-journal-xorg: Loading settings from 'xorg.conf' abrt-dump-journal-xorg: Loaded option 'BlacklistedXorgModules' = 'nvidia, fglrx, vboxvideo' abrt-dump-journal-xorg: Loaded option 'JournalFilters' = '_COMM=gdm-x-session, _COMM=gnome-shell' abrt-dump-journal-xorg: Loaded option 'BlacklistedXorgModules' = 'nvidia, fglrx, vboxvideo' abrt-dump-journal-xorg: Loaded option 'JournalFilters' = '_COMM=gdm-x-session, _COMM=gnome-shell' abrt-dump-journal-xorg: Loaded 'xorg.conf' abrt-dump-journal-xorg: Using journal filter from conf file xorg.conf abrt-dump-journal-xorg: Using journal match: '_COMM=gdm-x-session' abrt-dump-journal-xorg: Using journal match: '_COMM=gnome-shell' abrt-dump-journal-xorg: Failed to read 'MESSAGE' field: Bad message abrt-dump-journal-xorg: Cannot read journal data. Running it through ltrace I see last entries as: strncpy(0x7ffe0746f2e0, "[308042:308075:1119/053235.521890:ERROR:connection_factory_impl.cc(429)] Failed to connect to MCS endpoint with error -106", 122) = 0x7ffe0746f2e0 strstr("[308042:308075:1119/053235.521890:ERROR:connection_factory_impl.cc(429)] Failed to connect to MCS endpoint with error -106", "Backtrace:") = nil sd_journal_next(0x55b8f082eb50, 0x55b8eefe0ddc, 0x18000000000, 0x7ffe0746f35a) = 1 sd_journal_get_data(0x55b8f082eb50, 0x55b8eefe165e, 0x7ffe0746f2b0, 0x7ffe0746f2a8) = 0xffffffb6 strerror(74) = "Bad message" With command: $ sudo journalctl -u abrt-xorg.service I found that error appeared for the first time on November 19: ... -- Reboot -- Nov 09 12:19:36 yumba systemd[1]: Started ABRT Xorg log watcher. -- Reboot -- Nov 19 10:48:13 yumba systemd[1]: Started ABRT Xorg log watcher. Nov 19 10:48:14 yumba abrt-dump-journal-xorg[1645]: abrt-dump-journal-xorg: Cannot read journal data. Nov 19 10:48:14 yumba systemd[1]: abrt-xorg.service: Main process exited, code=exited, status=1/FAILURE Nov 19 10:48:14 yumba systemd[1]: abrt-xorg.service: Failed with result 'exit-code'. .... And I was able to narrow down actual journal file that is corrupted: $ journalctl --verify --file /var/log/journal/be37cfed3f434d0cb8d83feafdfaa26b/user-18031~ 1314e88: Invalid entry item (20/25 offset: 000000 1314e88: Invalid object contents: Bad message File corruption detected at /var/log/journal/be37cfed3f434d0cb8d83feafdfaa26b/user-18031~:1314e88 (of 25165824 bytes, 79%). FAIL: /var/log/journal/be37cfed3f434d0cb8d83feafdfaa26b/user-18031~ (Bad message) --- From what I understand in https://bugs.freedesktop.org/show_bug.cgi?id=64116 journal files which have been identified as corrupted are rotated and there is no plan to get those fixed nor there is a recommendation to remove them ("? Why would you want to throw-away the good parts in the journal files?").