Bug 448587

Created attachment 306812 [details]
rhm start strace

rhm starts if I comment out the following line from /etc/rhm.conf:

load-module=libbdbstore.so.0

This error usually occurs when the broker does not have the permissions it needs
to either read or write to the directory where the store is located. The default
location is /var/lib/qpidd, and the bdb files are initialized in
/var/lib/qpidd/rhm/dat. Perhaps this dir and/or its subdirs have root ownership
after install?

This problem has been observed on RHEL4 as well. The /var/lib/qpidd/rhm dir has
root ownership, preventing it from being written into by the broker which runs
as user qpidd.

Workaround: Remove directory /var/lib/qpidd/rhm. The broker will recreate it
with the correct permissions when it is started with the store module.

*** Bug 446896 has been marked as a duplicate of this bug. ***

There seems to be an additional problem with rhm on F8 (once the permissions
issue is out of the way):

[root@nsantos ~]# service rhmd start
Starting RHM daemon: Exception: Daemon startup failed: Error opening environment
(BdbMessageStore.cpp:140): DbEnv::open: DB_VERSION_MISMATCH: Database
environment version mismatch
                                                           [FAILED]
[root@nsantos ~]# rpm -q db4
db4-4.6.21-1.fc8

Assigning to Kim for troubleshooting of bdb versions in the store.

(In reply to comment #4)
> This problem has been observed on RHEL4 as well. The /var/lib/qpidd/rhm dir has
> root ownership, preventing it from being written into by the broker which runs
> as user qpidd.
> 
> Workaround: Remove directory /var/lib/qpidd/rhm. The broker will recreate it
> with the correct permissions when it is started with the store module.

qpidd-0.2.656926-9 and qpidc-0.2.656926-9 no longer seem to exhibit this problem.

(In reply to comment #6)
> There seems to be an additional problem with rhm on F8 (once the permissions
> issue is out of the way):
> 
> [root@nsantos ~]# service rhmd start
> Starting RHM daemon: Exception: Daemon startup failed: Error opening environment
> (BdbMessageStore.cpp:140): DbEnv::open: DB_VERSION_MISMATCH: Database
> environment version mismatch
>                                                            [FAILED]
> [root@nsantos ~]# rpm -q db4
> db4-4.6.21-1.fc8
> 
> Assigning to Kim for troubleshooting of bdb versions in the store.

This is likely caused by installing RHEL5 rpms on an F8 system. RHEL5 systems
use 4.3.29-9.fc6, while F8 uses 4.6.21-1.fc8. Since the binaries in the RHEL5
rpms are compiled using the earlier version, this error would not be surprising.

I think we need to use F8-compiled rpms to avoid this error. Note that this
error would affect the store only; it may be possible to avoid it by not loading
the store - at the expense of durability, of course.

I installed F8 RPMS that nuno built in koji, so I don't think it's a RHEL5 RPM
on F8 issue.

Tested fresh install of F8 rpms:
yum install qpidc-devel rhm amqp python-qpid
...
Installed: python-qpid.noarch 0:0.2.657115-12.fc8 qpidc-devel.i386
0:0.2.656926-1.fc8 rhm.i386 0:0.2.2060-2.fc8
Dependency Installed: amqp.noarch 0:1.0.656025-5.fc8 qpidc.i386
0:0.2.656926-1.fc8 qpidd.i386 0:0.2.656926-1.fc8
Complete!

[root@busy-beaver ~]# service rhmd start
Starting RHM daemon:                                       [  OK  ]

I'll try using RHEL5 rpms next and see if I can reproduce this problem.

OK, the theory of installing RHEL5 rpms on F8 is busted - there are *way* too
many dependency problems for this to be a realistic "mistake" (unless a --nodeps
was used...)

rpm -ivh amqp-1.0.656025-5.el5.noarch.rpm
python-qpid-0.2.657115-12.el5.noarch.rpm qpidc-0.2.656926-1.el5.i386.rpm
qpidc-devel-0.2.656926-1.el5.i386.rpm qpidd-0.2.656926-1.el5.i386.rpm
rhm-0.2.2060-2.el5.i386.rpm 
warning: amqp-1.0.656025-5.el5.noarch.rpm: Header V3 DSA signature: NOKEY, key
ID 897da07a
error: Failed dependencies:
        python(abi) = 2.4 is needed by python-qpid-0.2.657115-12.el5.noarch
        libboost_filesystem.so.2 is needed by qpidc-0.2.656926-1.el5.i386
        libboost_program_options.so.2 is needed by qpidc-0.2.656926-1.el5.i386
        libxerces-c.so.28 is needed by qpidc-0.2.656926-1.el5.i386
        libboost_filesystem.so.2 is needed by qpidd-0.2.656926-1.el5.i386
        libboost_program_options.so.2 is needed by qpidd-0.2.656926-1.el5.i386
        libxerces-c.so.28 is needed by qpidd-0.2.656926-1.el5.i386
        libdb_cxx-4.3.so is needed by rhm-0.2.2060-2.el5.i386

Taking a moment to peek into 4.6.21 source code, another possible scenario for
this error is the opening/recovery of a database created on a different version
of db4. This may be a testing artifact in this case. If this is true, and since
no databases are present in /var/lib/qpidd, this should not happen on a fresh
install.

When I installed I did not want to use the RHEL5 pacakges with --nodeps so I
waited for nuno to build the F8 packages.  I did not see the DB VERSION mismatch
problem because rhm wouldn't start, but when running qpidd alone I did see an
error and a a suggested option for how to resolve it (although it said it
mention it would destroy the previous data, or something to that effect).  I am
unable to reproduce it now unfortunately.

The only case I have seen in which this problem has occurred is where a database
from a previous or old installation exists. I believe BDB ship utilities which
allow an existing database from an older version of BDB to be parsed and
"upgraded" so it can be used with a newer version. This raises the question of
whether the installing RPM should contain a script to do just this when updating. 

I will change the status of this to NEEDINFO so that it does not fall between
the cracks... but this question should be solved prior to the first update after
GA, otherwise support issues will ensue.

From BDB docs:
DbEnv::open():
...
Errors:
The DbEnv::open method may fail and throw DbException, encapsulating one of the
following non-zero errors, or return one of the following non-zero errors: 
...
DB_VERSION_MISMATCH
    The version of the Berkeley DB library doesn't match the version that
created the database environment.
...

The solution for the moment to this is to capture this (and perhaps some of the
other open error messages) and give a more detailed error description with
perhaps some hint on how to solve the problem.

Note also that db4 supports an upgrade API call, but under what conditions it
could/should be safely used should a matter of future discussion.

I have been unable to reproduce this by copying databases from earlier versions
of db4 to F8. I copied databases from RHEL4.6 (4.2.52) and RHEL5.1 (F8 currently
uses 4.6.21), but instead of giving the DB_VERSION_MISMATCH error, I get:

Database handles still open at environment close
Error opening environment (BdbMessageStore.cpp:144): DbEnv::open: Invalid argument

I can resolve the error by running db_recover on the database (which requires
the db4-utils package to be installed). I'm not ready to hard-wire a response to
this error into the code until I know more - I'll release-note this instead.

r.2146: If BdEnv::open() throws DB_VERSION_MISMATCH, a message explaining some
options to solve the problem is now displayed.

qpidc-0.2.667603-1.el5, qpidc-perftest-0.2.667603-1.el5, qpidd-0.2.667603-1.el5, and rhm-0.2.2153-1.el5 have been pushed to the staging repo for testing