Bug 1143866

Summary: virt-v2v fails with error: cannot open Packages index using db5
Product: Red Hat Enterprise Linux 7 Reporter: tingting zheng <tzheng>
Component: libguestfsAssignee: Richard W.M. Jones <rjones>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: medium    
Version: 7.1CC: codong, dyuan, juzhou, mbooth, mzhan, ptoscano, rjones, tzheng
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: V2V
Fixed In Version: libguestfs-1.27.50-1.1.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-03-05 13:45:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Debug info about rpmdb error none

Description tingting zheng 2014-09-18 07:39:30 UTC
Created attachment 938770 [details]
Debug info about rpmdb error

Description
virt-v2v fails with error: cannot open Packages index using db5

Version:
libguestfs-1.27.48-1.1.el7.x86_64
virt-v2v-1.27.48-1.1.el7.x86_64

How reproducible:
50%

Steps to Reproduce:
1.Use virt-v2v to convert some guests,met the below error info.
#virt-v2v -o rhev -os 10.66.90.115:/vol/v2v_auto/auto_export --network rhevm rhel7-rc-graphics -v -x |& tee /tmp/v2v-rpmdb.log
virt-v2v: error: libguestfs error: command_lines: error: rpmdb: BDB0113 
Thread/process 12680/139878611654464 failed: BDB1507 Thread died in 
Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal 
error, run database recovery
error: cannot open Packages index using db5 -  (-30973)
error: cannot open Packages database in /var/lib/rpm
error: rpmdb: BDB0113 Thread/process 12680/139878611654464 failed: BDB1507 
Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal 
error, run database recovery
error: cannot open Packages database in /var/lib/rpm

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Actual results:
virt-v2v fails with error: cannot open Packages index using db5

Expected results:
virt-v2v should convert guests successfully without such error info.

Additional info:
The above issue can not be reproduced every time,sometimes I met similar error:
# virt-v2v -ic xen+ssh://10.66.106.64 -os default  xen-pv-rhel5.8-i386
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.66.106.64 xen-pv-rhel5.8-i386
[  16.0] Creating an overlay to protect the source from being modified
[  31.0] Opening the overlay
[ 103.0] Initializing the target -o libvirt -os default
[ 103.0] Inspecting the overlay
[ 116.0] Checking for sufficient free disk space in the guest
[ 116.0] Estimating space required on target for each disk
[ 116.0] Converting Red Hat Enterprise Linux Server release 5.8 (Tikanga) to run on KVM
virt-v2v: error: libguestfs error: command_lines: rpmdb: Program version 
4.3 doesn't match environment version
error: db4 error(-30974) from dbenv->open: DB_VERSION_MISMATCH: Database 
environment version mismatch
error: cannot open Packages index using db3 -  (-30974)
error: cannot open Packages database in /var/lib/rpm

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Comment 2 Richard W.M. Jones 2014-09-18 13:03:05 UTC
The error is that we run a very simple rpm command in the guest
and it fails rather badly:

# rpm -ql kernel
error: rpmdb: BDB0113 Thread/process 12680/139878611654464 failed: BDB1507 Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db5 -  (-30973)
error: cannot open Packages database in /var/lib/rpm
error: rpmdb: BDB0113 Thread/process 12680/139878611654464 failed: BDB1507 Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages database in /var/lib/rpm

Is the rhel7-rc-graphics guest OK?  Obviously it cannot be
running because we check for that.

You could try booting up the guest and running various simple
rpm commands like 'rpm -ql kernel' and 'rpm -qa' and just check
that the RPM database really isn't corrupt.

Comment 3 tingting zheng 2014-09-19 02:59:33 UTC
(In reply to Richard W.M. Jones from comment #2)
> The error is that we run a very simple rpm command in the guest
> and it fails rather badly:
> 
> # rpm -ql kernel
> error: rpmdb: BDB0113 Thread/process 12680/139878611654464 failed: BDB1507
> Thread died in Berkeley DB library
> error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal
> error, run database recovery
> error: cannot open Packages index using db5 -  (-30973)
> error: cannot open Packages database in /var/lib/rpm
> error: rpmdb: BDB0113 Thread/process 12680/139878611654464 failed: BDB1507
> Thread died in Berkeley DB library
> error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal
> error, run database recovery
> error: cannot open Packages database in /var/lib/rpm
> 
> Is the rhel7-rc-graphics guest OK?  Obviously it cannot be
> running because we check for that.

Yes,it is ok,after conversion,I can boot it successfully.

> You could try booting up the guest and running various simple
> rpm commands like 'rpm -ql kernel' and 'rpm -qa' and just check
> that the RPM database really isn't corrupt.

I tried these commands,it worked fine.

btw,I can not reproduce this error everytime for the same guest,there is a rhel6 bug 806714 in old v2v for reference.

Comment 4 Richard W.M. Jones 2014-09-19 11:38:12 UTC
I think I see what's going on.  I have pushed these
patches which I hope will fix things (virt-v2v >= 1.27.50):

https://github.com/libguestfs/libguestfs/commit/6db2ff52f9557bc3644700386320dc7a3fa54830
https://github.com/libguestfs/libguestfs/commit/4aa38bba2ca77cf0ff7271a8f419182c57f59f7a

Comment 6 tingting zheng 2014-09-22 07:45:47 UTC
Reproduce this bug with:
libguestfs-1.27.49-1.1.el7.x86_64
virt-v2v-1.27.49-1.1.el7.x86_64

1.Prepare a guest,do some operations to make the rpmdb corrupted.
# yum install libvirt
error: rpmdb: BDB0113 Thread/process 3350/140483719948096 failed: BDB1507 Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db5 -  (-30973)
error: cannot open Packages database in /var/lib/rpm
CRITICAL:yum.main:

Error: rpmdb open failed

2.Use virt-v2v to convert the guest,error shows:
# virt-v2v -ic xen+ssh://10.66.106.64 -o rhev  -os 10.66.90.115:/vol/v2v_auto/auto_export --network rhevm  rhel7-rc-graphics -of raw
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.66.106.64 rhel7-rc-graphics
[  16.0] Creating an overlay to protect the source from being modified
[  30.0] Opening the overlay
[ 230.0] Initializing the target -o rhev -os 10.66.90.115:/vol/v2v_auto/auto_export
[ 230.0] Inspecting the overlay
[ 245.0] Checking for sufficient free disk space in the guest
[ 245.0] Estimating space required on target for each disk
[ 245.0] Converting Red Hat Enterprise Linux Server release 7.0 (Maipo) to run on KVM
virt-v2v: error: libguestfs error: command_lines: error: rpmdb: BDB0113 
Thread/process 3350/140483719948096 failed: BDB1507 Thread died in Berkeley 
DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal 
error, run database recovery
error: cannot open Packages index using db5 -  (-30973)
error: cannot open Packages database in /var/lib/rpm
error: rpmdb: BDB0113 Thread/process 3350/140483719948096 failed: BDB1507 
Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal 
error, run database recovery
error: cannot open Packages database in /var/lib/rpm

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Tested with latest virt-v2v version:
libguestfs-1.27.50-1.1.el7.x86_64
virt-v2v-1.27.50-1.1.el7.x86_64

Repeat the above steps,no such error shows.

Refer to the above comments,move the bug to VERIFIED.

Comment 8 errata-xmlrpc 2015-03-05 13:45:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0303.html