Bug 160317

I did a quick search with google and it looks like there is likely a way to get
db 4.3 to work. You run some commands with db 4.2 against the database files,
then upgrade to 4.3.

http://mail-index.netbsd.org/tech-pkg/2005/01/27/0031.html

I am very interested in your results. I have multiple mail servers using
cyrus-imapd under FC3 that I plan to upgrade to FC4.

Thanks for the feedback, Nathan! -I used another machine and moved the entire imap-config and 
structure over and tried what's written in the URL you sent. I moved the /var/lib/imap back to my server 
and restarted it. Everything seems ok, exept some trouble with some CA sertificates. Again thank you for 
the input!

You are welcome. Thank you for sharing the results.

A very quick and very dirty way to make things work again looks like this:

rm -rvf /var/lib/imap/db/* /var/lib/imap/db.backup?

This simply removes the BDB logs and backups which have a modified file format
for 4.3. If something goes wrong, the last backup is your friend. It has always
worked for me, YMMW.

*** Bug 160748 has been marked as a duplicate of this bug. ***

My workaround was brute force:

Delete the database files, move the message hierarchy out
of the way, remove cyrus-imapd, reinstall it fresh,
move the messages back, and reconstruct.

That worked, since I'm my only mail user, but not acceptable
for anything where you've got real users.

You can also fix the database using tools in db-compat:

# service cyrus-imapd stop
# db42_checkpoint -v -1 -h /var/lib/imap/db
# db42_recover -v -h /var/lib/imap/db
# rm -vf /var/lib/imap/db/log.*
# service cyrus-imapd start

I believe the steps outlined in comment #7 represent the optimal series of steps
for fixing this problem. However there are a couple of caveats:

The package name is compat-db.

You need to run the version of db_checkpoint and db_recover that the previous
version of cyrus-imapd was built with. The version is encoded in the name, e.g.
db42_checkpoint is Berkely DB 4.2. But how do you know what version the previous
cyrus-imap was built with? It's in the file
/usr/share/doc/cyrus-imapd-*/README.buildoptions, but the doc directory will
have been replaced by the new rpm install, so that's telling you about the
current version. You would have needed to have read this file before upgrading
and since you're reading this bugzilla chances are you already lost that
opportunity. The %pre script should proably be checking this and doing the
checkpoint/recover. I'm looking into that option.

The good news is since we haven't shipped a lot of versions of cyrus-imap the
mapping from cyrus-imap rpm to Berkely DB is trival. If its FC4 its linked
against Berkely 4.3, otherwise for all other Red Hat supplied RPMS its Berkley
DB 4.2. Thus if you're upgrading from any previous release to FC4 then use the
db42_checkpoint, db42_recover commands in the compat-db rpm.

If anyone experiences trouble with the method in comment #7 I would like to know
about it as soon as possible because it will proably be folded into the spec
file for upgrades and a new release made so others are not bitten by this.

I get:

db42_checkpoint -v -1 -h /var/lib/imap/db
db_checkpoint: Program version 4.2 doesn't match environment version
db_checkpoint: open: Invalid argument

This is after an FC3->FC4 upgrade, using cyrus-imapd frome extras.

John, my real question is why do we have to checkpoint/recover any db files
after cyrus-imapd has been shutdown cleanly? Is there anything kept somewhere
else than in the .db files? I can understand there is something left after a
crash but why after a clean shutdown? But then, I'm not an expert in BDB.

Can some BDB guru answer my question?

I'm quite sure I'll do something else in the upcoming Invoca rpms because
checkpoint/recover does simply not work for those using my rpms. I'll very
likely create backups of all db related files and then remove the logs as
mentioned in my comment #4 so if something goes wrong, one still has the old db
files around to fix things by hand.

With respect to comment #10 from Simon:

I'm not a BDB guru either :-) But over the last two days have tried to educate
myself by reading all the BDB doc and the source code in cyrus-imap that uses
BDB (lib/cyrusdb_berkley.c). Here is my current understanding, abeit imperfect,
if other's see errors please correct me.

BDB has upgrade procedures which depend what changed between releases. It is not
an automatic process, BDB does not detect the need to perform the upgrade
procedures, it is up to the installation to do this. The first step in this
process is knowing what changed between the version of the BDB library the
application previously linked with and the version of the BDB library it is
currently linked with. We went from 4.2 to 4.3.

Here is the doc on what changed in 4.3:
http://www.sleepycat.com/docs/ref/upgrade.4.3/disk.html

The only thing that changed in the environment was the log file format (there
were API changes too, but that's not an issue in this instance).

Here is the doc on how perform an upgrade once you know what changed:
http://www.sleepycat.com/docs/ref/upgrade/process.html

You will note it is necessary to know what features of BDB are used by the
application, in particular if its using transactions. Cyrus-imap is using
transactions and logging (see cyrusdb_berkley.c).

BDB uses write ahead logging to implement its transactions. The operations
necessary to commit a transaction is first written to the log file. To assure
the transaction is atomic the state of the database can be queried to see what
operations in the log file were performed and if necessary roll back or roll
forward operations to make sure the database is consistent. This operation is
called recovery. Recovery cannot be performed without the log file.

Checkpointing assures transaction components are flushed to the log file in a
consistent manner. If the application shutdown cleanly checkpointing is not
necessary, however it does not hurt and is considered a safe step.

Just because the application shut down cleanly does not mean outstanding
transactions are cleanly commited to the database. Remember a transaction is a
series of database operations that are meant to appear atomic. Thus it is
possible for a clean shutdown to occur but leave partial transactions in the log
file.

Recovery is what makes the database consistent. The recovery process includes
review of log files and databases to ensure that the changes from each committed
transaction appear in the database, and that no changes from an unfinished (or
aborted) transaction do.

Log files may be specific to a BDB version (in our case they are). Therefore
recovery has to be performed using the version of BDB the log file was written
with. Once recovery is complete the log file can be removed so that a new log
file can be generated by the new version of the BDB library when the new version
of the application starts.

My concern is that if you do just remove the log files without recovery you may
leave the database in an inconsistent state. If log file removal has worked for
you in the past you may have been fortunate or it may just indicate that
cyrus-imap's transactions are not complex with interdepenent transaction components.

After reading a lot of the BDB doc its clear to me that transitioning BDB
versions is not simple nor automatic. A lot of onus is placed on admins, this
does not seem very friendly to me. In the past we've discussed whether BDB
should be packaged with cyrus-imap. We both felt this was a liability and added
needless complexity. However, as I've investigated this more I've learned many
of the major applications which use BDB do in fact package their own version of
BDB and I'm beginning to understand why. BDB makes very few concessions to
version upgrades leaving most of the issues to installations :-(

When performing an upgrade installation it can be seen from above it is
necessary to know the BDB version previously in use. Unfortunately I could not
find anyway to query this information. One cannot use the BDB utilities because
BDB may have been upgraded independently because cyrus-imap and BDB are separate
packages. I can think of only two ways to get the version information.

1) Query the existing log files or database files for version information, in
other words, tell me what version of BDB wrote you? It's a pretty obvious
utility but I couldn't find any way to do it (to the best of my knowledge the -V
option on many of the BDB utilities only tell you about the library version that
utility was linked with). Data files do not seem to have version information
embedded in them (true??).

2) Query the cyrus-imap that is being replaced during the upgrade. There are
several ways to get the BDB version BEFORE the upgrade takes place. Once the
upgrade occurs the information is lost. This is why I'm of the mind at the
moment doing something in %pre is the optimal solution.

This has been a challenging problem to unravel, at least in terms of finding a
solution is robust. I'm leary of operating on assumptions and prefer a solution
that follows the guidelines. It is a shame none of this showed up during the FC4
testing period.

With respect to comment # from Rob:

Did you run cyrus-imapd for any length of time after doing the FC4 upgrade? I'm
wondering if DBB 4.3 files got written into the environment by the new version
of cyrus-imapd.

FYI, I will be out of the office till 7/4/2005, I am not ignoring this issue.
I've searched info-cyrus archives. I did not find any new information that is
not otherwise present in the bug report.

Viz comment #12: Yes, I'm afraid I did.

As per comment #7, I tried the commands during an upgrade from FC3 to FC4. They
seem to have worked perfectly for me.

I performed the steps listed in #7 and still have a problem. I restored the
database files from my most recent level 0 backup and re-ran the steps in #7. I
still have problems. What do I try next?

Jul  3 19:35:51 k2 master[4646]: process started
Jul  3 19:35:51 k2 master[4648]: about to exec /usr/lib/cyrus-imapd/ctl_cyrusdb
Jul  3 19:35:52 k2 ctl_cyrusdb[4648]: DBERROR: db4
Jul  3 19:35:52 k2 ctl_cyrusdb[4648]: recovering cyrus databases
Jul  3 19:35:52 k2 ctl_cyrusdb[4648]: skiplist: recovered /var/imap/mailboxes.db
(80 records, 12588 bytes) in 0 seconds
Jul  3 19:35:52 k2 ctl_cyrusdb[4648]: skiplist: recovered
/var/imap/annotations.db (0 records, 144 bytes) in 0 seconds
Jul  3 19:35:52 k2 ctl_cyrusdb[4648]: DBERROR: db4
Jul  3 19:35:52 k2 ctl_cyrusdb[4648]: DBERROR: db4
Jul  3 19:35:52 k2 ctl_cyrusdb[4648]: DBERROR: critical database situation
Jul  3 19:35:52 k2 master[4646]: process 4648 exited, status 75
Jul  3 19:35:52 k2 master[4649]: about to exec /usr/lib/cyrus-imapd/idled
Jul  3 19:35:52 k2 idled[4649]: DBERROR: db4
Jul  3 19:35:52 k2 idled[4649]: DBERROR: critical database situation
Jul  3 19:35:52 k2 master[4646]: process 4649 exited, status 75
Jul  3 19:35:52 k2 master[4646]: ready for work
Jul  3 19:35:52 k2 master[4650]: about to exec /usr/lib/cyrus-imapd/tls_prune
Jul  3 19:35:52 k2 master[4651]: about to exec /usr/lib/cyrus-imapd/cyr_expire
Jul  3 19:35:52 k2 master[4652]: about to exec /usr/lib/cyrus-imapd/ctl_cyrusdb
Jul  3 19:35:52 k2 ctl_cyrusdb[4652]: DBERROR: db4
Jul  3 19:35:52 k2 ctl_cyrusdb[4652]: DBERROR: critical database situation
Jul  3 19:35:52 k2 master[4646]: process 4652 exited, status 75
Jul  3 19:35:52 k2 tls_prune[4650]: DBERROR: db4
Jul  3 19:35:52 k2 tls_prune[4650]: DBERROR: critical database situation
Jul  3 19:35:52 k2 master[4646]: process 4650 exited, status 75
Jul  3 19:35:52 k2 cyr_expire[4651]: DBERROR: db4
Jul  3 19:35:52 k2 cyr_expire[4651]: DBERROR: critical database situation
Jul  3 19:35:52 k2 master[4646]: process 4651 exited, status 75
Jul  3 19:36:13 k2 master[4646]: exiting on SIGTERM/SIGINT

How about including commands, similar to those outlined
in comment #7, to the init script, in the "stop" section?

This may not solve this particular upgrade problem, but
may prevent similar problems in the future.

That idea seems a good one. I found my problems from #16. Sometime over the past
upgrades I had created both /var/imap and /var/lib/imap. I kept trying to
recover /var/lib/imap when the most recent stuff is in /var/imap. I was able to
recover /var/imap and am now up and working again.

Thanks for the commands in #7!

As promised incomment #10, I have implemented a solution in our invoca rpms.
Whenever cyrus-imapd is stopped via the sysv init script, all Berkeley databases
are checkpointed, recovered and then exported to skiplist. On startup, databases
which are configured as Berkeley DB are imported from skiplist. This is all done
using the cvt_cyrusdb_all script which is called from the init script. I've put
much work and testing into the whole process and it seems to work fine. For
example I tested the migration from RedHat 7.2 based server to Fedora Core 4. I
simply stopped cyrus-imapd on the RedHat 7.2 box, rsynced /var/lib/imap and
/var/spool/imap over to the new Fedora Core 4 box, and started cyrus-imapd
again. Everything worked as expected.

Thank you Simon!!

In our company, we have migrated a production server from and old server  
(Fedora Core 3) to a new one (Fedora Core 4). We had the same problem with BDB  
version due to distro release upgrade.  
  
We have applied SUCCESSFULLY the procedure suggested in this post into the  
recover process.  
  
After a few tries in our test enviroment, this is our very basic "script". The  
files, obtained from the "backup" are in "/MYDIR/mail".  
  
[root@mymachine mail]# pwd  
/MYDIR/mail  
[root@mymachine mail]# ls  
200602011358_mail.tar  etc  restaura-cyrus-FC3toFC4.sh  var  
  
[root@ mail]# cat restaura-cyrus-FC3toFC4.sh  
  
#!/bin/bash  
# Script to recover cyrus-imapd server from FC 3 to FC 4.  
# The backup files must be recovered in /MYDIR/mail  
# Cyrus-imapd must be stopped.  
  
# Delete default mailbox DB installed by RPMs  
  
rm /var/lib/imap/deliver.db  
rm /var/lib/imap/annotations.db  
rm /var/lib/imap/mailboxes.db  
rm /var/lib/imap/backup/*  
rm /var/lib/imap/db.backup*/*  
rm /var/lib/imap/db/*  
rm /var/lib/imap/log/*  
rm /var/lib/imap/msg/*  
rm /var/lib/imap/proc/*  
rm /var/lib/imap/quota/*  
rm /var/lib/imap/sieve/*  
rm -r /var/lib/imap/user/*  
rm -r /var/spool/imap/*  
  
# Restore from backup  
  
cp -p /MYDIR/mail/var/lib/imap/deliver.db /var/lib/imap/  
cp -p /MYDIR/mail/var/lib/imap/annotations.db /var/lib/imap/  
cp -p /MYDIR/mail/var/lib/imap/mailboxes.db /var/lib/imap/  
cp -p /MYDIR/mail/var/lib/imap/backup/* /var/lib/imap/backup/  
cp -p /MYDIR/mail/var/lib/imap/db/* /var/lib/imap/db/  
cp -p /MYDIR/mail/var/lib/imap/log/* /var/lib/imap/log/  
cp -p /MYDIR/mail/var/lib/imap/msg/* /var/lib/imap/msg/  
cp -p /MYDIR/mail/var/lib/imap/quota/* /var/lib/imap/quota/  
cp -p /MYDIR/mail/var/lib/imap/sieve/* /var/lib/imap/sieve/  
cp -p -R /MYDIR/mail/var/lib/imap/user/* /var/lib/imap/user/  
cp -p -R /MYDIR/mail/var/spool/imap/* /var/spool/imap/  
  
# DB status recover  
  
db42_checkpoint -v -1 -h /var/lib/imap/db  
db42_recover -v -h /var/lib/imap/db  
rm -vf /var/lib/imap/db/log.*  
  
The script output:  
  
[root@mymachine mail]# ./restaura-cyrus-FC3toFC4.sh  
rm: no se puede borrar «/var/lib/imap/db.backup*/*»: No existe el fichero o el  
directorio  
rm: no se puede borrar «/var/lib/imap/log/*»: No existe el fichero o el  
directorio  
rm: no se puede borrar «/var/lib/imap/msg/*»: No existe el fichero o el  
directorio  
rm: no se puede borrar «/var/lib/imap/proc/*»: No existe el fichero o el  
directorio  
rm: no se puede borrar «/var/lib/imap/quota/*»: No existe el fichero o el  
directorio  
rm: no se puede borrar «/var/lib/imap/sieve/*»: No existe el fichero o el  
directorio  
cp: no se puede efectuar `stat' sobre «/MYDIR/mail/var/lib/imap/log/*»: No  
existe el fichero o el directorio  
cp: no se puede efectuar `stat' sobre «/MYDIR/mail/var/lib/imap/msg/*»: No  
existe el fichero o el directorio  
cp: no se puede efectuar `stat' sobre «/MYDIR/mail/var/lib/imap/quota/*»: No  
existe el fichero o el directorio  
cp: no se puede efectuar `stat' sobre «/MYDIR/mail/var/lib/imap/sieve/*»: No  
existe el fichero o el directorio  
db_checkpoint: checkpoint: Fri Jan 27 09:56:05 2006  
  
db_recover: Finding last valid log LSN: file: 1 offset 776504  
db_recover: Recovery starting from [1][776299]  
db_recover: Recovery complete at Fri Jan 27 09:56:13 2006  
db_recover: Maximum transaction ID 80000011 Recovery checkpoint [1][776605]  
«/var/lib/imap/db/log.0000000001» borrado  
  
A warning should be included into distro release notes about this issue. For  
example, the same will happen when someone upgrade from FC3 to FC5.  
  
Thank you. 

Fedora Core 4 is no longer maintained.

Setting status to "INSUFFICIENT_DATA". If you can reproduce this bug in the
current Fedora release, please reopen this bug and assign it to the
corresponding Fedora version.

*** Bug 322451 has been marked as a duplicate of this bug. ***