Bug 1931002

Created attachment 1758331 [details]
/var/log

Description of problem:
Upgrade RHVH via RHVM. There is an exception related to db5 in imgbased, then an error occurred while removing the new layer.


Version-Release number of selected component (if applicable):
redhat-virtualization-host-4.4.5-20210215.0.el8_3
imgbased-1.2.17-0.1.el8ev.noarch

How reproducible:
not 100%

Steps to Reproduce:
1. Install RHVH-4.4-20210202.0-RHVH-x86_64-dvd1.iso
2.  Login to RHVH, setup local repos and point to "redhat-virtualization-host-4.4.5-20210215.0.el8_3"
3. Add the host to RHVM
4. Upgrade the host via RHVM

Actual results:
Host upgrade failed.

Expected results:
RHVH upgrade successfully.

Additional info:
The error in imgbased.log is as follows:
~~~~~~
2021-02-20 02:19:24,331 [DEBUG] (MainThread) Calling: (['rpm', '-U', '--quiet', '--justdb', '--root', '/tmp/mnt.hAcsB//', '/var/cache/dnf/rhel-7-server-rhvh-4-rpms-21ea416fbe4cd14c/packages/redhat-virtualization-host-image-update-latest.rpm'],) {'close_fds': True, 'stderr': -2}
2021-02-20 02:19:24,468 [DEBUG] (MainThread) Exception! b'error: db5 error(-30969) from dbenv->open: BDB0091 DB_VERSION_MISMATCH: Database environment version mismatch\nerror: cannot open Packages index using db5 -  (-30969)\nerror: cannot open Packages database in /tmp/mnt.hAcsB/var/lib/rpm\n'
2021-02-20 02:19:24,469 [DEBUG] (MainThread) Calling: (['umount', '-l', '/tmp/mnt.hAcsB'],) {'close_fds': True, 'stderr': -2}
2021-02-20 02:19:24,575 [DEBUG] (MainThread) Calling: (['rmdir', '/tmp/mnt.hAcsB'],) {'close_fds': True, 'stderr': -2}
2021-02-20 02:19:24,580 [DEBUG] (MainThread) Calling: (['umount', '-l', '/tmp/mnt.H8TNL'],) {'close_fds': True, 'stderr': -2}
2021-02-20 02:19:24,625 [DEBUG] (MainThread) Calling: (['rmdir', '/tmp/mnt.H8TNL'],) {'close_fds': True, 'stderr': -2}
2021-02-20 02:19:24,630 [DEBUG] (MainThread) Calling: (['umount', '-l', '/tmp/mnt.EWcoj'],) {'close_fds': True, 'stderr': -2}
2021-02-20 02:19:24,654 [DEBUG] (MainThread) Calling: (['rmdir', '/tmp/mnt.EWcoj'],) {'close_fds': True, 'stderr': -2}
2021-02-20 02:19:24,659 [ERROR] (MainThread) Update failed, resetting registered LVs
2021-02-20 02:19:24,659 [DEBUG] (MainThread) Calling: (['sync'],) {'close_fds': True, 'stderr': -2}
2021-02-20 02:19:24,679 [DEBUG] (MainThread) Calling: (['lvs', '--noheadings', '--ignoreskippedcluster', '-olv_dm_path', 'rhvh_dell-per515-01/rhvh-4.4.5.3-0.20210215.0'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-02-20 02:19:24,747 [DEBUG] (MainThread) Returned: b'/dev/mapper/rhvh_dell--per515--01-rhvh--4.4.5.3--0.20210215.0'
2021-02-20 02:19:24,747 [DEBUG] (MainThread) Calling: (['lvremove', '-ff', 'rhvh_dell-per515-01/rhvh-4.4.5.3-0.20210215.0'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-02-20 02:19:24,909 [DEBUG] (MainThread) Returned: b'Logical volume "rhvh-4.4.5.3-0.20210215.0" successfully removed'
2021-02-20 02:19:24,910 [DEBUG] (MainThread) Calling: (['lvs', '--noheadings', '--ignoreskippedcluster', '-olv_dm_path', 'rhvh_dell-per515-01/rhvh-4.4.5.3-0.20210215.0+1'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-02-20 02:19:24,969 [DEBUG] (MainThread) Returned: b'/dev/mapper/rhvh_dell--per515--01-rhvh--4.4.5.3--0.20210215.0+1'
2021-02-20 02:19:24,969 [DEBUG] (MainThread) Calling: (['lvremove', '-ff', 'rhvh_dell-per515-01/rhvh-4.4.5.3-0.20210215.0+1'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-02-20 02:19:25,236 [DEBUG] (MainThread) Returned: b'Logical volume "rhvh-4.4.5.3-0.20210215.0+1" successfully removed'
Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/tmp.H0sNFrZqyn/usr/lib/python3.6/site-packages/imgbased/__main__.py", line 53, in <module>
    CliApplication()
  File "/tmp/tmp.H0sNFrZqyn/usr/lib/python3.6/site-packages/imgbased/__init__.py", line 82, in CliApplication
    app.hooks.emit("post-arg-parse", args)
  File "/tmp/tmp.H0sNFrZqyn/usr/lib/python3.6/site-packages/imgbased/hooks.py", line 120, in emit
    cb(self.context, *args)
  File "/tmp/tmp.H0sNFrZqyn/usr/lib/python3.6/site-packages/imgbased/plugins/update.py", line 75, in post_argparse
    six.reraise(*exc_info)
  File "/tmp/tmp.H0sNFrZqyn/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
    raise value
  File "/tmp/tmp.H0sNFrZqyn/usr/lib/python3.6/site-packages/imgbased/plugins/update.py", line 66, in post_argparse
    base, _ = LiveimgExtractor(app.imgbase).extract(args.FILENAME)
  File "/tmp/tmp.H0sNFrZqyn/usr/lib/python3.6/site-packages/imgbased/plugins/update.py", line 148, in extract
    "%s" % size, nvr)
  File "/tmp/tmp.H0sNFrZqyn/usr/lib/python3.6/site-packages/imgbased/plugins/update.py", line 128, in add_base_with_tree
    new_layer_lv = self.imgbase.add_layer(new_base)
  File "/tmp/tmp.H0sNFrZqyn/usr/lib/python3.6/site-packages/imgbased/imgbase.py", line 213, in add_layer
    self.hooks.emit("new-layer-added", prev_lv, new_lv)
  File "/tmp/tmp.H0sNFrZqyn/usr/lib/python3.6/site-packages/imgbased/hooks.py", line 120, in emit
    cb(self.context, *args)
  File "/tmp/tmp.H0sNFrZqyn/usr/lib/python3.6/site-packages/imgbased/plugins/osupdater.py", line 118, in on_new_layer
    postprocess(new_lv)
  File "/tmp/tmp.H0sNFrZqyn/usr/lib/python3.6/site-packages/imgbased/plugins/osupdater.py", line 253, in postprocess
    _install_update_rpm()
  File "/tmp/tmp.H0sNFrZqyn/usr/lib/python3.6/site-packages/imgbased/plugins/osupdater.py", line 245, in _install_update_rpm
    new_fs.path("/"), rpms[0]])
  File "/tmp/tmp.H0sNFrZqyn/usr/lib/python3.6/site-packages/imgbased/utils.py", line 373, in rpm
    return self.call(["rpm"] + args, **kwargs)
  File "/tmp/tmp.H0sNFrZqyn/usr/lib/python3.6/site-packages/imgbased/utils.py", line 324, in call
    stdout = command.call(*args, **kwargs)
  File "/tmp/tmp.H0sNFrZqyn/usr/lib/python3.6/site-packages/imgbased/command.py", line 14, in call
    return subprocess.check_output(*args, **kwargs).strip()
  File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib64/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['rpm', '-U', '--quiet', '--justdb', '--root', '/tmp/mnt.hAcsB//', '/var/cache/dnf/rhel-7-server-rhvh-4-rpms-21ea416fbe4cd14c/packages/redhat-virtualization-host-image-update-latest.rpm']' returned non-zero exit status 1.
~~~~~~