Bug 448587 - Unable to start rhm on F8
Unable to start rhm on F8
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp (Show other bugs)
beta
All Linux
urgent Severity high
: ---
: ---
Assigned To: Kim van der Riet
Kim van der Riet
:
: 446896 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-05-27 14:18 EDT by Robert Rati
Modified: 2009-05-07 16:09 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-12-02 11:06:12 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
rhm start strace (28.32 KB, text/plain)
2008-05-27 14:18 EDT, Robert Rati
no flags Details

  None (edit)
Description Robert Rati 2008-05-27 14:18:10 EDT
Description of problem:
Using the recently built BT4 packages for F8, I am unable to start the rhm
daemon.  The following error is spit out:

Starting RHM daemon: Exception: Daemon startup failed: Error opening environment
(BdbMessageStore.cpp:140): DbEnv::open: Permission denied
                                                           [FAILED]

I have attached an strace as well.

Version-Release number of selected component (if applicable):
rhm-0.2.2060-2

How reproducible:
Very

Steps to Reproduce:
1. Install or upgrade to the BT4 release on F8 using the x86_64 packages.
2. Run /etc/init.d/rhm start
3.
  
Actual results:


Expected results:


Additional info:
Comment 1 Robert Rati 2008-05-27 14:18:10 EDT
Created attachment 306812 [details]
rhm start strace
Comment 2 Robert Rati 2008-05-27 15:13:00 EDT
rhm starts if I comment out the following line from /etc/rhm.conf:

load-module=libbdbstore.so.0
Comment 3 Kim van der Riet 2008-05-27 16:04:40 EDT
This error usually occurs when the broker does not have the permissions it needs
to either read or write to the directory where the store is located. The default
location is /var/lib/qpidd, and the bdb files are initialized in
/var/lib/qpidd/rhm/dat. Perhaps this dir and/or its subdirs have root ownership
after install?
Comment 4 Kim van der Riet 2008-05-27 16:42:31 EDT
This problem has been observed on RHEL4 as well. The /var/lib/qpidd/rhm dir has
root ownership, preventing it from being written into by the broker which runs
as user qpidd.

Workaround: Remove directory /var/lib/qpidd/rhm. The broker will recreate it
with the correct permissions when it is started with the store module.
Comment 5 Kim van der Riet 2008-05-28 07:27:22 EDT
*** Bug 446896 has been marked as a duplicate of this bug. ***
Comment 6 Nuno Santos 2008-05-29 15:11:06 EDT
There seems to be an additional problem with rhm on F8 (once the permissions
issue is out of the way):

[root@nsantos ~]# service rhmd start
Starting RHM daemon: Exception: Daemon startup failed: Error opening environment
(BdbMessageStore.cpp:140): DbEnv::open: DB_VERSION_MISMATCH: Database
environment version mismatch
                                                           [FAILED]
[root@nsantos ~]# rpm -q db4
db4-4.6.21-1.fc8

Assigning to Kim for troubleshooting of bdb versions in the store.
Comment 7 Jeff Needle 2008-05-29 16:00:21 EDT
(In reply to comment #4)
> This problem has been observed on RHEL4 as well. The /var/lib/qpidd/rhm dir has
> root ownership, preventing it from being written into by the broker which runs
> as user qpidd.
> 
> Workaround: Remove directory /var/lib/qpidd/rhm. The broker will recreate it
> with the correct permissions when it is started with the store module.

qpidd-0.2.656926-9 and qpidc-0.2.656926-9 no longer seem to exhibit this problem.
Comment 8 Kim van der Riet 2008-05-30 09:44:20 EDT
(In reply to comment #6)
> There seems to be an additional problem with rhm on F8 (once the permissions
> issue is out of the way):
> 
> [root@nsantos ~]# service rhmd start
> Starting RHM daemon: Exception: Daemon startup failed: Error opening environment
> (BdbMessageStore.cpp:140): DbEnv::open: DB_VERSION_MISMATCH: Database
> environment version mismatch
>                                                            [FAILED]
> [root@nsantos ~]# rpm -q db4
> db4-4.6.21-1.fc8
> 
> Assigning to Kim for troubleshooting of bdb versions in the store.

This is likely caused by installing RHEL5 rpms on an F8 system. RHEL5 systems
use 4.3.29-9.fc6, while F8 uses 4.6.21-1.fc8. Since the binaries in the RHEL5
rpms are compiled using the earlier version, this error would not be surprising.

I think we need to use F8-compiled rpms to avoid this error. Note that this
error would affect the store only; it may be possible to avoid it by not loading
the store - at the expense of durability, of course.
Comment 9 Robert Rati 2008-05-30 09:56:04 EDT
I installed F8 RPMS that nuno built in koji, so I don't think it's a RHEL5 RPM
on F8 issue.
Comment 10 Kim van der Riet 2008-05-30 11:12:48 EDT
Tested fresh install of F8 rpms:
yum install qpidc-devel rhm amqp python-qpid
...
Installed: python-qpid.noarch 0:0.2.657115-12.fc8 qpidc-devel.i386
0:0.2.656926-1.fc8 rhm.i386 0:0.2.2060-2.fc8
Dependency Installed: amqp.noarch 0:1.0.656025-5.fc8 qpidc.i386
0:0.2.656926-1.fc8 qpidd.i386 0:0.2.656926-1.fc8
Complete!

[root@busy-beaver ~]# service rhmd start
Starting RHM daemon:                                       [  OK  ]

I'll try using RHEL5 rpms next and see if I can reproduce this problem.
Comment 11 Kim van der Riet 2008-05-30 11:28:14 EDT
OK, the theory of installing RHEL5 rpms on F8 is busted - there are *way* too
many dependency problems for this to be a realistic "mistake" (unless a --nodeps
was used...)

rpm -ivh amqp-1.0.656025-5.el5.noarch.rpm
python-qpid-0.2.657115-12.el5.noarch.rpm qpidc-0.2.656926-1.el5.i386.rpm
qpidc-devel-0.2.656926-1.el5.i386.rpm qpidd-0.2.656926-1.el5.i386.rpm
rhm-0.2.2060-2.el5.i386.rpm 
warning: amqp-1.0.656025-5.el5.noarch.rpm: Header V3 DSA signature: NOKEY, key
ID 897da07a
error: Failed dependencies:
        python(abi) = 2.4 is needed by python-qpid-0.2.657115-12.el5.noarch
        libboost_filesystem.so.2 is needed by qpidc-0.2.656926-1.el5.i386
        libboost_program_options.so.2 is needed by qpidc-0.2.656926-1.el5.i386
        libxerces-c.so.28 is needed by qpidc-0.2.656926-1.el5.i386
        libboost_filesystem.so.2 is needed by qpidd-0.2.656926-1.el5.i386
        libboost_program_options.so.2 is needed by qpidd-0.2.656926-1.el5.i386
        libxerces-c.so.28 is needed by qpidd-0.2.656926-1.el5.i386
        libdb_cxx-4.3.so is needed by rhm-0.2.2060-2.el5.i386
Comment 12 Kim van der Riet 2008-05-30 12:13:55 EDT
Taking a moment to peek into 4.6.21 source code, another possible scenario for
this error is the opening/recovery of a database created on a different version
of db4. This may be a testing artifact in this case. If this is true, and since
no databases are present in /var/lib/qpidd, this should not happen on a fresh
install.
Comment 13 Robert Rati 2008-05-30 15:18:26 EDT
When I installed I did not want to use the RHEL5 pacakges with --nodeps so I
waited for nuno to build the F8 packages.  I did not see the DB VERSION mismatch
problem because rhm wouldn't start, but when running qpidd alone I did see an
error and a a suggested option for how to resolve it (although it said it
mention it would destroy the previous data, or something to that effect).  I am
unable to reproduce it now unfortunately.
Comment 14 Kim van der Riet 2008-06-06 07:44:32 EDT
The only case I have seen in which this problem has occurred is where a database
from a previous or old installation exists. I believe BDB ship utilities which
allow an existing database from an older version of BDB to be parsed and
"upgraded" so it can be used with a newer version. This raises the question of
whether the installing RPM should contain a script to do just this when updating. 

I will change the status of this to NEEDINFO so that it does not fall between
the cracks... but this question should be solved prior to the first update after
GA, otherwise support issues will ensue.
Comment 15 Kim van der Riet 2008-06-09 15:02:24 EDT
From BDB docs:
DbEnv::open():
...
Errors:
The DbEnv::open method may fail and throw DbException, encapsulating one of the
following non-zero errors, or return one of the following non-zero errors: 
...
DB_VERSION_MISMATCH
    The version of the Berkeley DB library doesn't match the version that
created the database environment.
...

The solution for the moment to this is to capture this (and perhaps some of the
other open error messages) and give a more detailed error description with
perhaps some hint on how to solve the problem.

Note also that db4 supports an upgrade API call, but under what conditions it
could/should be safely used should a matter of future discussion.
Comment 16 Kim van der Riet 2008-06-10 10:24:04 EDT
I have been unable to reproduce this by copying databases from earlier versions
of db4 to F8. I copied databases from RHEL4.6 (4.2.52) and RHEL5.1 (F8 currently
uses 4.6.21), but instead of giving the DB_VERSION_MISMATCH error, I get:

Database handles still open at environment close
Error opening environment (BdbMessageStore.cpp:144): DbEnv::open: Invalid argument

I can resolve the error by running db_recover on the database (which requires
the db4-utils package to be installed). I'm not ready to hard-wire a response to
this error into the code until I know more - I'll release-note this instead.
Comment 17 Kim van der Riet 2008-06-10 10:32:39 EDT
r.2146: If BdEnv::open() throws DB_VERSION_MISMATCH, a message explaining some
options to solve the problem is now displayed.
Comment 18 Mike Bonnet 2008-06-19 23:52:45 EDT
qpidc-0.2.667603-1.el5, qpidc-perftest-0.2.667603-1.el5, qpidd-0.2.667603-1.el5, and rhm-0.2.2153-1.el5 have been pushed to the staging repo for testing

Note You need to log in before you can comment on or make changes to this bug.