RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2120267 - Anaconda kickstart test is failing due Traceback lvmdbusd:KeyError: 'vdo_operating_mode'
Summary: Anaconda kickstart test is failing due Traceback lvmdbusd:KeyError: 'vdo_oper...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: lvm2
Version: 9.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: LVM and device-mapper development team
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-08-22 11:51 UTC by Radek Vykydal
Modified: 2023-05-09 10:37 UTC (History)
10 users (show)

Fixed In Version: lvm2-2.03.17-1.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-05-09 08:23:45 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
virt-install.log with the traceback (534.66 KB, text/plain)
2022-08-22 11:51 UTC, Radek Vykydal 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker CLUSTERQE-6170 0 None None None 2022-11-28 20:42:39 UTC
Red Hat Issue Tracker RHELPLAN-131832 0 None None None 2022-08-22 12:01:58 UTC
Red Hat Product Errata RHBA-2023:2544 0 None None None 2023-05-09 08:24:04 UTC

Description Radek Vykydal 2022-08-22 11:51:40 UTC 
Created attachment 1906893 [details]
virt-install.log with the traceback

Description of problem:

Very rarely, as a flake in Anaconda kickstart tests installation fails with this traceback:

01:50:00,167 INFO lvmdbusd:2439:2439 - MThreadRunner: exception
01:50:00,167 INFO lvmdbusd: Traceback (most recent call last):
01:50:00,167 INFO lvmdbusd:  File "/usr/lib/python3.9/site-packages/lvmdbusd/utils.py", line 667, in _run
01:50:00,167 INFO lvmdbusd:    self.rc = self.f(*self.args)
01:50:00,167 INFO lvmdbusd:  File "/usr/lib/python3.9/site-packages/lvmdbusd/fetch.py", line 40, in _main_thread_load
01:50:00,167 INFO lvmdbusd:    (lv_changes, remove) = load_lvs(
01:50:00,167 INFO lvmdbusd:  File "/usr/lib/python3.9/site-packages/lvmdbusd/lv.py", line 143, in load_lvs
01:50:00,167 INFO lvmdbusd:    return common(
01:50:00,167 INFO lvmdbusd:  File "/usr/lib/python3.9/site-packages/lvmdbusd/loader.py", line 37, in common
01:50:00,167 INFO lvmdbusd:    objects = retrieve(search_keys, cache_refresh=False)
01:50:00,167 INFO lvmdbusd:  File "/usr/lib/python3.9/site-packages/lvmdbusd/lv.py", line 95, in lvs_state_retrieve
01:50:00,167 INFO lvmdbusd:    l['vdo_operating_mode'],
01:50:00,167 INFO lvmdbusd:KeyError: 'vdo_operating_mode'

Kickstart-tests issue filed for the problem: https://github.com/rhinstaller/kickstart-tests/issues/724

Version-Release number of selected component (if applicable):

lvm2-2.03.16-3.el9.x86_64
but we've seen it also on latest RHEL9 in May and earlier.

How reproducible:

Very rarely, seems like ~ 1/20000 test runs.

Steps to Reproduce:
1. Run many anaconda kickstart tests and wait for a race condition?
2.
3.

Actual results:

Anaconda kickstart test fails with traceback

Expected results:

No traceback during the test.

Additional info:

The issue is causing Anaconda kickstart-tests 'flakes' (2nd run succeeds).
Seems extremely hard to reproduce.
It is possible that the installation itself would finish successfully. The log monitor of the test is killing the installation when it sees the Traceback in the log.
Comment 1 Radek Vykydal 2022-08-22 11:54:28 UTC 
Vojto, do you think it makes sense to follow-up on this report (reassign to lvm) given its extremely low frequency and hard reproducibility? Also it is not sure if the crash is fatal for the installation. Or is it maybe a known issue?
Comment 2 Vojtech Trefny 2022-08-24 05:43:49 UTC 
@rvykydal Any chance of getting `/tmp/lvm.log` from the kickstart tests run?

@tasleson Can you take a look at this? If it really happens only in 1/20000 runs I think we can ignore it (especially because both lvmdbusd and anaconda/blivet seems to be ok with that).
Comment 3 Radek Vykydal 2022-08-24 07:38:26 UTC 
(In reply to Vojtech Trefny from comment #2)
> @rvykydal Any chance of getting `/tmp/lvm.log` from the kickstart
> tests run?

Unfortunately we don't gather this log on regular daily runs of kickstart tests so we'd need to try to reproduce with tests modified to get the log. The reproducing may take some month(s) using significant portion of our testing resources (~1/5) in Upshift. If lvm team consider the issue worth pursuing following we would consider setting up the reproducing.
Comment 4 Tony Asleson 2022-08-25 14:34:37 UTC 
(In reply to Vojtech Trefny from comment #2)
> @tasleson Can you take a look at this? If it really happens only
> in 1/20000 runs I think we can ignore it (especially because both lvmdbusd
> and anaconda/blivet seems to be ok with that).

The root cause of this error is lvm not returning a key in the JSON output.

Lvmdbusd retrieves the state of lvm from a separate thread.  This tread will loop 5 times waiting a second between
each attempt before it shuts the daemon down if it encounters an exception.  This
was added because lvm has a history of not returning all the requested data or returning malformed JSON.
This error is not actually reported to the dbus API user and is effectively hidden from the user unless it
persists for more than 5 seconds.  This traceback is informational, however it's causing the test to fail
because:

"The log monitor of the test is killing the installation when it sees the Traceback in the log.

I'll move this output so that it will only report if we fail to retrieve the state of lvm after all
the attempts are exhausted and the daemon is essentially shutting down.
Comment 5 Tony Asleson 2022-10-17 16:04:23 UTC 
Correction posted upstream: https://sourceware.org/git/?p=lvm2.git;a=commit;h=d42bdb07def72d68c05241823979b28952978c05
Comment 8 Corey Marthaler 2022-12-07 16:34:44 UTC 
Marking Verifed:Tested (SanityOnly) now that initial regression testing has finished on the latest 9.2 rpms.

kernel-5.14.0-201.kpq1.el9    BUILT: Wed Nov 23 02:13:08 PM CET 2022
lvm2-2.03.17-2.el9    BUILT: Wed Nov 30 09:56:30 AM CET 2022
lvm2-libs-2.03.17-2.el9    BUILT: Wed Nov 30 09:56:30 AM CET 2022
Comment 13 Corey Marthaler 2023-01-18 18:00:27 UTC 
Marking VERIFIED (SanityOnly) with vdo regression test results for the latest 9.2 build as well.

kernel-5.14.0-231.el9    BUILT: Mon Jan  9 08:32:41 PM CET 2023
lvm2-2.03.17-4.el9    BUILT: Tue Jan 10 06:40:12 PM CET 2023
lvm2-libs-2.03.17-4.el9    BUILT: Tue Jan 10 06:40:12 PM CET 2023
kmod-kvdo-8.2.1.3-66.el9    BUILT: Wed Jan  4 10:39:27 PM CET 2023
Comment 14 Jones5S 2023-02-11 06:28:15 UTC Comment hidden (spam)
Comment 16 errata-xmlrpc 2023-05-09 08:23:45 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (lvm2 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2544
Note You need to log in before you can comment on or make changes to this bug.