Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Created attachment 1906893[details]
virt-install.log with the traceback
Description of problem:
Very rarely, as a flake in Anaconda kickstart tests installation fails with this traceback:
01:50:00,167 INFO lvmdbusd:2439:2439 - MThreadRunner: exception
01:50:00,167 INFO lvmdbusd: Traceback (most recent call last):
01:50:00,167 INFO lvmdbusd: File "/usr/lib/python3.9/site-packages/lvmdbusd/utils.py", line 667, in _run
01:50:00,167 INFO lvmdbusd: self.rc = self.f(*self.args)
01:50:00,167 INFO lvmdbusd: File "/usr/lib/python3.9/site-packages/lvmdbusd/fetch.py", line 40, in _main_thread_load
01:50:00,167 INFO lvmdbusd: (lv_changes, remove) = load_lvs(
01:50:00,167 INFO lvmdbusd: File "/usr/lib/python3.9/site-packages/lvmdbusd/lv.py", line 143, in load_lvs
01:50:00,167 INFO lvmdbusd: return common(
01:50:00,167 INFO lvmdbusd: File "/usr/lib/python3.9/site-packages/lvmdbusd/loader.py", line 37, in common
01:50:00,167 INFO lvmdbusd: objects = retrieve(search_keys, cache_refresh=False)
01:50:00,167 INFO lvmdbusd: File "/usr/lib/python3.9/site-packages/lvmdbusd/lv.py", line 95, in lvs_state_retrieve
01:50:00,167 INFO lvmdbusd: l['vdo_operating_mode'],
01:50:00,167 INFO lvmdbusd:KeyError: 'vdo_operating_mode'
Kickstart-tests issue filed for the problem: https://github.com/rhinstaller/kickstart-tests/issues/724
Version-Release number of selected component (if applicable):
lvm2-2.03.16-3.el9.x86_64
but we've seen it also on latest RHEL9 in May and earlier.
How reproducible:
Very rarely, seems like ~ 1/20000 test runs.
Steps to Reproduce:
1. Run many anaconda kickstart tests and wait for a race condition?
2.
3.
Actual results:
Anaconda kickstart test fails with traceback
Expected results:
No traceback during the test.
Additional info:
The issue is causing Anaconda kickstart-tests 'flakes' (2nd run succeeds).
Seems extremely hard to reproduce.
It is possible that the installation itself would finish successfully. The log monitor of the test is killing the installation when it sees the Traceback in the log.
Vojto, do you think it makes sense to follow-up on this report (reassign to lvm) given its extremely low frequency and hard reproducibility? Also it is not sure if the crash is fatal for the installation. Or is it maybe a known issue?
@rvykydal Any chance of getting `/tmp/lvm.log` from the kickstart tests run?
@tasleson Can you take a look at this? If it really happens only in 1/20000 runs I think we can ignore it (especially because both lvmdbusd and anaconda/blivet seems to be ok with that).
(In reply to Vojtech Trefny from comment #2)
> @rvykydal Any chance of getting `/tmp/lvm.log` from the kickstart
> tests run?
Unfortunately we don't gather this log on regular daily runs of kickstart tests so we'd need to try to reproduce with tests modified to get the log. The reproducing may take some month(s) using significant portion of our testing resources (~1/5) in Upshift. If lvm team consider the issue worth pursuing following we would consider setting up the reproducing.
(In reply to Vojtech Trefny from comment #2)
> @tasleson Can you take a look at this? If it really happens only
> in 1/20000 runs I think we can ignore it (especially because both lvmdbusd
> and anaconda/blivet seems to be ok with that).
The root cause of this error is lvm not returning a key in the JSON output.
Lvmdbusd retrieves the state of lvm from a separate thread. This tread will loop 5 times waiting a second between
each attempt before it shuts the daemon down if it encounters an exception. This
was added because lvm has a history of not returning all the requested data or returning malformed JSON.
This error is not actually reported to the dbus API user and is effectively hidden from the user unless it
persists for more than 5 seconds. This traceback is informational, however it's causing the test to fail
because:
"The log monitor of the test is killing the installation when it sees the Traceback in the log.
I'll move this output so that it will only report if we fail to retrieve the state of lvm after all
the attempts are exhausted and the daemon is essentially shutting down.
Marking Verifed:Tested (SanityOnly) now that initial regression testing has finished on the latest 9.2 rpms.
kernel-5.14.0-201.kpq1.el9 BUILT: Wed Nov 23 02:13:08 PM CET 2022
lvm2-2.03.17-2.el9 BUILT: Wed Nov 30 09:56:30 AM CET 2022
lvm2-libs-2.03.17-2.el9 BUILT: Wed Nov 30 09:56:30 AM CET 2022
Marking VERIFIED (SanityOnly) with vdo regression test results for the latest 9.2 build as well.
kernel-5.14.0-231.el9 BUILT: Mon Jan 9 08:32:41 PM CET 2023
lvm2-2.03.17-4.el9 BUILT: Tue Jan 10 06:40:12 PM CET 2023
lvm2-libs-2.03.17-4.el9 BUILT: Tue Jan 10 06:40:12 PM CET 2023
kmod-kvdo-8.2.1.3-66.el9 BUILT: Wed Jan 4 10:39:27 PM CET 2023
Comment 14Jones5S
2023-02-11 06:28:15 UTC
Comment hidden (spam)
This comment was flagged a spam, view the edit history to see the original text if required.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (lvm2 bug fix and enhancement update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2023:2544