Bug 470084
| Summary: | Problems migrating from libdb-4.4 to libdb-4.7 | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Retired] 389 | Reporter: | Vitaly Kuznetsov <vitty> | ||||||||
| Component: | Database - General | Assignee: | Deon Ballard <dlackey> | ||||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Chandrasekar Kannan <ckannan> | ||||||||
| Severity: | high | Docs Contact: | |||||||||
| Priority: | high | ||||||||||
| Version: | 1.1.3 | CC: | benl, nhosoi, nkinder, rmeggins, rom | ||||||||
| Target Milestone: | --- | ||||||||||
| Target Release: | --- | ||||||||||
| Hardware: | x86_64 | ||||||||||
| OS: | Linux | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2009-05-03 23:23:33 UTC | Type: | --- | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Bug Depends On: | |||||||||||
| Bug Blocks: | 249650, 493682 | ||||||||||
| Attachments: |
|
||||||||||
Created attachment 324089 [details]
cvs diff ldap/servers/slapd/back-ldbm/dblayer.c
File: ldap/servers/slapd/back-ldbm/dblayer.c
Description: when starting the db with the mode DBLAYER_CLEAN_RECOVER_MODE (db upgrade case), transaction logs were removed, which should not have.
Followings are the supported upgrading scenario (including the multiple hops, e.g., 4.2 -> 4.6): BDB 4.2 -> BDB 4.3 The log file format changed in the Berkeley DB 4.3 release. No database formats changed in the Berkeley DB 4.3 release. BDB 4.3 -> BDB 4.4 The log file format changed in the Berkeley DB 4.4 release. No database formats changed in the Berkeley DB 4.4 release. BDB 4.4 -> BDB 4.5 The log file format changed in the Berkeley DB 4.5 release. No database formats changed in the Berkeley DB 4.5 release. BDB 4.5 -> BDB 4.6 The log file format changed in the Berkeley DB 4.6 release. No database except hash format changed in the Berkeley DB 4.6 release. BDB 4.6 -> BDB 4.7 The log file format changed in the Berkeley DB 4.7 release. No database formats changed in the Berkeley DB 4.7 release. All the above supported upgrade falls into the following case on the page "Upgrading Berkeley DB installations" http://www.oracle.com/technology/documentation/berkeley-db/db/ref/upgrade/process.html If the application has a Berkeley DB transactional environment, and the log files need upgrading but the databases do not, the application may be installed in the field using the following steps: 1. Shut down the old version of the application. 2. Still using the old version of Berkeley DB, run recovery on the database environment using the DB_ENV->open method or the db_recover utility. 3. If you used the DB_ENV->open method to run recovery, make sure that the Berkeley DB environment is removed using the DB_ENV->remove method or an appropriate system utility. 4. Archive the database environment for catastrophic recovery. See Archival procedures for more information. 5. Recompile and install the new version of the application. 6. Force a checkpoint using the DB_ENV->txn_checkpoint method or the db_checkpoint utility. If you use the db_checkpoint utility, make sure to use the new version of the utility; that is, the version that came with the release of Berkeley DB to which you are upgrading. 7. Restart the application. We should put this info somewhere such as the release notes since this is the recommended steps to upgrade the db version. I tested BDB 4.6.21 -> 4.7.25 on Fedora-9 and BDB 4.2.52 -> 4.7.25 on RHEL4 without running the old version of db_recover, but relying on the new recover start, which is called internally. Before calling the recoveriy start, BDB api dbp->remove is called to remove the BDB environment (__db.* files), which issues this message in the errors log: [..] - libdb: Program version 4.7 doesn't match environment version 4.6 Regardless of the version mismatch message, upgrade is successfully done. # ./dbverify -V [..] DB verify - /var/lib/dirsrv/slapd-kiki1/db/userRoot/uid.db4: ok [19/Nov/2008:12:18:31 -0800] DB verify - /var/lib/dirsrv/slapd-kiki1/db/userRoot/cn.db4: ok [..] DB verify - /var/lib/dirsrv/slapd-kiki1/db/userRoot/parentid.db4: ok [...] [..] DB verify - /var/lib/dirsrv/slapd-kiki1/db/userRoot/numsubordinates.db4: ok DB verify: Passed Note: When starting, if the Directory Server finds out the BDB version that the server is liked to is newer than the version the db files have, the server automatically starts the db with DBLAYER_CLEAN_RECOVER_MODE, which is similar to running the BDB utility db_recover. Thanks, I've tested this patch with db4.4->db4.7, db4.6 -> 4.7 migrations - seems working. (In reply to comment #3) > Thanks, > > I've tested this patch with db4.4->db4.7, db4.6 -> 4.7 migrations - seems > working. Wow, it was very quick! Thank you so much for testing these cases. --noriko Created attachment 324212 [details]
cvs commt message
Reviewed by Rich and Nathan (Thank you, both of you!!)
Checked in into CVS HEAD.
Deon, could you add the info in the comment #2 to the release notes? The comment explains the recommended way to upgrade the BDB version. Thanks! --noriko *** Bug 480550 has been marked as a duplicate of this bug. *** I assume above comment refers to FDS release notes... Since this breaks on upgrade from Fedora 9 to Fedora 10, it may need added to Fedora 10's release notes as well. I tried the steps listed above in comment #2 (as closely as I could anyways, since FDS linked against db4-4.7 had already attempted to upgrade it), same result. dirsrv won't start, same error in log file. I installed compat-db46 in order to have access to 4.6 version of db4 utilities. I restored a backup of db made before attempting above, and rebuilt fedora-ds-base package with patch file from above, same result. dirsrv won't start, same error in log file. So, my question is this, is my database unrepairable since I don't have a copy of it from before my upgrade to Fedora 10, and FDS linked against db4-4.7 tried to upgrade it. Or is there something else entirely going on? My original steps taken, and error log file are attached to Bug 480550, which has been marked as a duplicate of this one. (In reply to comment #9) > I tried the steps listed above in comment #2 (as closely as I could anyways, > since FDS linked against db4-4.7 had already attempted to upgrade it), same > result. When you did the steps, was your database healthy or corrupted? If corrupted, do you happen to have a healthy backup to try the migration? > dirsrv won't start, same error in log file. I installed compat-db46 > in order to have access to 4.6 version of db4 utilities. > > I restored a backup of db made before attempting above, and rebuilt > fedora-ds-base package with patch file from above, same result. dirsrv won't > start, same error in log file. > > So, my question is this, is my database unrepairable since I don't have a copy > of it from before my upgrade to Fedora 10, and FDS linked against db4-4.7 tried > to upgrade it. Or is there something else entirely going on? > > My original steps taken, and error log file are attached to Bug 480550, which > has been marked as a duplicate of this one. (In reply to comment #10) > (In reply to comment #9) > > I tried the steps listed above in comment #2 (as closely as I could anyways, > > since FDS linked against db4-4.7 had already attempted to upgrade it), same > > result. > > When you did the steps, was your database healthy or corrupted? If corrupted, > do you happen to have a healthy backup to try the migration? I believe it became corrupted when dirsrv tried to run for the first time under Fedora 10. It was healthy pre-upgrade. I unfortunately do not have a pre-upgrade healthy copy. (In reply to comment #11) > (In reply to comment #10) > > (In reply to comment #9) > > > I tried the steps listed above in comment #2 (as closely as I could anyways, > > > since FDS linked against db4-4.7 had already attempted to upgrade it), same > > > result. > > > > When you did the steps, was your database healthy or corrupted? If corrupted, > > do you happen to have a healthy backup to try the migration? > > I believe it became corrupted when dirsrv tried to run for the first time under > Fedora 10. It was healthy pre-upgrade. I unfortunately do not have a > pre-upgrade healthy copy. Could it be possible to export (db2ldif) from the corrupted db? If it does, could you import (ldif2db) the ldif file to the new FDS on F10? (In reply to comment #12) > > Could it be possible to export (db2ldif) from the corrupted db? If it does, > could you import (ldif2db) the ldif file to the new FDS on F10? db2ldif attempt failed. I'll attach a log file. Created attachment 329897 [details]
db2ldif log
That's bad news. This is not a recommended method at all, but if dbscan utility could dump your db, you may be able to recover the db. $ dbscan -f /path/to/your/main/db (normally, /var/lib/dirsrv/slapd-<your_id>/userRoot/id2entry.db4 > id2entry.out Edit id2entry.out to convert it to a correct ldif file that can be imported to the server. 1) remove lines for system attributes . nsUniqueId: ... . creatorsName: ... . modifiersName: ... . createTimestamp: ... . modifyTimestamp: ... . parentid: ... . entryid: ... . entrydn: ... . numSubordinates: ... 2) remove lines of "id ##" 3) remove the first tab in each line. E.g., " objectClass: top" ==> "objectClass: top" Hopefully, you could recover your db contents with these steps. Well, I think that got it closer. # /usr/lib/dirsrv/slapd-DRAGON/ldif2db -n userRoot -i /root/db/id2entry/id2entry.ldif importing data ... [26/Jan/2009:13:24:24 -0500] - dblayer_instance_start: pagesize: 4096, pages: 124155, procpages: 8279 [26/Jan/2009:13:24:24 -0500] - cache autosizing: import cache: 198648k [26/Jan/2009:13:24:24 -0500] - li_import_cache_autosize: 50, import_pages: 49662, pagesize: 4096 [26/Jan/2009:13:24:24 -0500] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [26/Jan/2009:13:24:24 -0500] - dblayer_instance_start: pagesize: 4096, pages: 124155, procpages: 8279 [26/Jan/2009:13:24:24 -0500] - cache autosizing: import cache: 198648k [26/Jan/2009:13:24:24 -0500] - li_import_cache_autosize: 50, import_pages: 49662, pagesize: 4096 [26/Jan/2009:13:24:25 -0500] - import userRoot: Beginning import job... [26/Jan/2009:13:24:25 -0500] - import userRoot: Index buffering enabled with bucket size 60 [26/Jan/2009:13:24:25 -0500] - import userRoot: Processing file "/root/db/id2entry/id2entry.ldif" [26/Jan/2009:13:24:25 -0500] - Entry "krbprincipalname=kadmin/changepw@DRAGON,cn=DRAGON,cn=kerberos,dc=dragon" has unknown object class "topon" [26/Jan/2009:13:24:25 -0500] - import userRoot: WARNING: skipping entry "krbprincipalname=kadmin/changepw@DRAGON,cn=DRAGON,cn=kerberos,dc=dragon" which violates schema, ending line 429 of file "/root/db/id2entry/id2entry.ldif" [26/Jan/2009:13:24:25 -0500] - Entry "krbprincipalname=kadmin/history@DRAGON,cn=DRAGON,cn=kerberos,dc=dragon" has unknown object class "topn" [26/Jan/2009:13:24:25 -0500] - import userRoot: WARNING: skipping entry "krbprincipalname=kadmin/history@DRAGON,cn=DRAGON,cn=kerberos,dc=dragon" which violates schema, ending line 446 of file "/root/db/id2entry/id2entry.ldif" [26/Jan/2009:13:24:25 -0500] - Entry "krbprincipalname=kadmin/twister.dragon@DRAGON,cn=DRAGON,cn=kerberos,dc=dragon" has unknown object class "topc=dragon" [26/Jan/2009:13:24:25 -0500] - import userRoot: WARNING: skipping entry "krbprincipalname=kadmin/twister.dragon@DRAGON,cn=DRAGON,cn=kerberos,dc=dragon" which violates schema, ending line 471 of file "/root/db/id2entry/id2entry.ldif" [26/Jan/2009:13:24:25 -0500] - Entry "krbprincipalname=ldap/twister.dragon@DRAGON,cn=DRAGON,cn=kerberos,dc=dragon" has unknown object class "topdragon" [26/Jan/2009:13:24:25 -0500] - import userRoot: WARNING: skipping entry "krbprincipalname=ldap/twister.dragon@DRAGON,cn=DRAGON,cn=kerberos,dc=dragon" which violates schema, ending line 495 of file "/root/db/id2entry/id2entry.ldif" [26/Jan/2009:13:24:25 -0500] - Entry "krbprincipalname=host/twister.dragon@DRAGON,cn=DRAGON,cn=kerberos,dc=dragon" has unknown object class "topdragon" [26/Jan/2009:13:24:25 -0500] - import userRoot: WARNING: skipping entry "krbprincipalname=host/twister.dragon@DRAGON,cn=DRAGON,cn=kerberos,dc=dragon" which violates schema, ending line 519 of file "/root/db/id2entry/id2entry.ldif" [26/Jan/2009:13:24:25 -0500] - Entry "krbprincipalname=HTTP/twister.dragon@DRAGON,cn=DRAGON,cn=kerberos,dc=dragon" has unknown object class "topdragon" [26/Jan/2009:13:24:25 -0500] - import userRoot: WARNING: skipping entry "krbprincipalname=HTTP/twister.dragon@DRAGON,cn=DRAGON,cn=kerberos,dc=dragon" which violates schema, ending line 543 of file "/root/db/id2entry/id2entry.ldif" [26/Jan/2009:13:24:25 -0500] - import userRoot: Finished scanning file "/root/db/id2entry/id2entry.ldif" (51 entries) [26/Jan/2009:13:24:25 -0500] - import userRoot: Workers finished; cleaning up... [26/Jan/2009:13:24:26 -0500] - import userRoot: Workers cleaned up. [26/Jan/2009:13:24:26 -0500] - import userRoot: Cleaning up producer thread... [26/Jan/2009:13:24:26 -0500] - import userRoot: Indexing complete. Post-processing... [26/Jan/2009:13:24:26 -0500] - import userRoot: Flushing caches... [26/Jan/2009:13:24:26 -0500] - import userRoot: Closing files... [26/Jan/2009:13:24:26 -0500] - All database threads now stopped [26/Jan/2009:13:24:26 -0500] - import userRoot: Import complete. Processed 51 entries (6 were skipped) in 1 seconds. (51.00 entries/sec) # /usr/lib/dirsrv/slapd-DRAGON/dbverify [26/Jan/2009:13:24:44 -0500] - libdb: Page 1: out-of-order key at entry 2 [26/Jan/2009:13:24:44 -0500] - libdb: /var/lib/dirsrv/slapd-DRAGON/db/userRoot/uidnumber.db4: DB_VERIFY_BAD: Database verification failed [26/Jan/2009:13:24:44 -0500] DB verify - verify failed(-30972): /var/lib/dirsrv/slapd-DRAGON/db/userRoot/uidnumber.db4 DB verify: Failed Any luck since then? I've noticed you received these error messages: unknown object class "topn" unknown object class "topdragon" Do you happen to have had custom schema in the previous deploy? If so, could you migrate it to the new server, as well? Regarding uidnumber.db4, is the attribute from posixAccount? If you dump the index file with dbscan what do you see? dbscan -r -n -f /var/lib/dirsrv/slapd-DRAGON/db/userRoot/uidnumber.db4 Also, at this point, does your server start? How concerned should I be about the dbverify errors?(In reply to comment #17) > Any luck since then? Not quite sure I understand the dbverify error, or what would be the best way to fix it. Kinda learning on the fly. > I've noticed you received these error messages: > unknown object class "topn" > unknown object class "topdragon" dragon is the base dn > > Do you happen to have had custom schema in the previous deploy? If so, could > you migrate it to the new server, as well? The schema is just as FreeIPA installed it. I've not customized it beyond that. > > Regarding uidnumber.db4, is the attribute from posixAccount? If you dump the > index file with dbscan what do you see? > dbscan -r -n -f /var/lib/dirsrv/slapd-DRAGON/db/userRoot/uidnumber.db4 I can send you the file privately if that would be helpful. > Also, at this point, does your server start? The dirsrv does start, but how concerned should I be about the dbverify errors? On a somewhat unrelated note, have a suggestion for a feature. Trigger a backup of the database before automaticly upgrading it, or attempting to repair it. I am concerned with how repeatable this might be for other people upgrading from Fedora 9 to 10 that have Fedora-DS installed. Cleaned up ldif file per emailed recommendation from Noriko, restored backup of database (taken after initial db corruption), and re-imported. [root@twister db]# /usr/lib/dirsrv/slapd-DRAGON/ldif2db -n userRoot -i /root/db/id2entry/id2entry.ldif importing data ... [28/Jan/2009:21:49:17 -0500] - dblayer_instance_start: pagesize: 4096, pages: 124155, procpages: 8280 [28/Jan/2009:21:49:17 -0500] - cache autosizing: import cache: 198648k [28/Jan/2009:21:49:17 -0500] - li_import_cache_autosize: 50, import_pages: 49662, pagesize: 4096 [28/Jan/2009:21:49:17 -0500] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [28/Jan/2009:21:49:17 -0500] - dblayer_instance_start: pagesize: 4096, pages: 124155, procpages: 8280 [28/Jan/2009:21:49:17 -0500] - cache autosizing: import cache: 198648k [28/Jan/2009:21:49:17 -0500] - li_import_cache_autosize: 50, import_pages: 49662, pagesize: 4096 [28/Jan/2009:21:49:17 -0500] - import userRoot: Beginning import job... [28/Jan/2009:21:49:17 -0500] - import userRoot: Index buffering enabled with bucket size 60 [28/Jan/2009:21:49:17 -0500] - import userRoot: Processing file "/root/db/id2entry/id2entry.ldif" [28/Jan/2009:21:49:17 -0500] - import userRoot: Finished scanning file "/root/db/id2entry/id2entry.ldif" (57 entries) [28/Jan/2009:21:49:17 -0500] - import userRoot: Workers finished; cleaning up... [28/Jan/2009:21:49:18 -0500] - import userRoot: Workers cleaned up. [28/Jan/2009:21:49:18 -0500] - import userRoot: Cleaning up producer thread... [28/Jan/2009:21:49:18 -0500] - import userRoot: Indexing complete. Post-processing... [28/Jan/2009:21:49:18 -0500] - import userRoot: Flushing caches... [28/Jan/2009:21:49:18 -0500] - import userRoot: Closing files... [28/Jan/2009:21:49:18 -0500] - All database threads now stopped [28/Jan/2009:21:49:18 -0500] - import userRoot: Import complete. Processed 57 entries in 1 seconds. (57.00 entries/sec) [root@twister db]# /usr/lib/dirsrv/slapd-DRAGON/dbverify [28/Jan/2009:21:50:50 -0500] - libdb: Page 1: out-of-order key at entry 2 [28/Jan/2009:21:50:50 -0500] - libdb: /var/lib/dirsrv/slapd-DRAGON/db/userRoot/uidnumber.db4: DB_VERIFY_BAD: Database verification failed [28/Jan/2009:21:50:50 -0500] DB verify - verify failed(-30972): /var/lib/dirsrv/slapd-DRAGON/db/userRoot/uidnumber.db4 DB verify: Failed [root@twister db]# dbscan -r -n -f /var/lib/dirsrv/slapd-DRAGON/db/userRoot/uidnumber.db4 =999 1 11 =1100 1 39 =1101 1 40 =1102 1 53 =1103 1 54 =1105 1 57 Sorry missed request for dbscan of uidnumber.db4 in earlier note. dirsrv will start now. I added the information in comment #2 to the RHDS 8.1 release notes. These are live at http://www.redhat.com/docs/manuals/dir-server/8.1/rel-notes/Release_Notes-Bugs_Addressed-Known_Issues.html. Closing the bug. |
Hello! I'm trying to build Fedora-DS for ALT Linux project with new libdb-4.7. Old instances use libdb-4.4. I have some problems with data migration. FDS with db4.7 does not want to read old (db4.4) data with errors. I tried to delete logs and do db_upgrade - without success. New instances work fine. Fedora DS versions (old and new) are both 1.1.3. Errors: [05/Nov/2008:13:56:37 +0300] - Fedora-Directory/1.1.3 B2008.310.1047 starting up [05/Nov/2008:13:56:37 +0300] - Clean up db environment and start from archive. [05/Nov/2008:13:56:37 +0300] - libdb: Program version 4.7 doesn't match environment version 4.4 [05/Nov/2008:13:56:37 +0300] - libdb: Program version 4.7 doesn't match environment version 4.4 [05/Nov/2008:13:56:37 +0300] - Deleting log file: (/var/lib/fedora-ds/slapd-ldap1/db/log.0000000001) [05/Nov/2008:13:56:37 +0300] - Deleting log file: (/var/lib/fedora-ds/slapd-ldap1/db/log.0000000147) [05/Nov/2008:13:56:38 +0300] - attrcrypt_unwrap_key: failed to unwrap key for cipher AES [05/Nov/2008:13:56:38 +0300] - Failed to retrieve key for cipher AES in attrcrypt_cipher_init [05/Nov/2008:13:56:38 +0300] - Failed to initialize cipher AES in attrcrypt_init [05/Nov/2008:13:56:38 +0300] - libdb: file userRoot/id2entry.db4 has LSN 134/2584752, past end of log at 1/140 [05/Nov/2008:13:56:38 +0300] - libdb: Commonly caused by moving a database from one database environment [05/Nov/2008:13:56:38 +0300] - libdb: to another without clearing the database LSNs, or by removing all of [05/Nov/2008:13:56:38 +0300] - libdb: the log files from a database environment [05/Nov/2008:13:56:38 +0300] - libdb: /var/lib/fedora-ds/slapd-ldap1/db/userRoot/id2entry.db4: unexpected file type or format [05/Nov/2008:13:56:38 +0300] - dbp->open("userRoot/id2entry.db4") failed: Invalid argument (22) [05/Nov/2008:13:56:38 +0300] - dblayer_instance_start fail: Invalid argument (22) [05/Nov/2008:13:56:38 +0300] - attrcrypt_unwrap_key: failed to unwrap key for cipher AES [05/Nov/2008:13:56:38 +0300] - Failed to retrieve key for cipher AES in attrcrypt_cipher_init [05/Nov/2008:13:56:38 +0300] - Failed to initialize cipher AES in attrcrypt_init [05/Nov/2008:13:56:38 +0300] - libdb: file NetscapeRoot/id2entry.db4 has LSN 1/3096357, past end of log at 1/140 [05/Nov/2008:13:56:38 +0300] - libdb: Commonly caused by moving a database from one database environment [05/Nov/2008:13:56:38 +0300] - libdb: to another without clearing the database LSNs, or by removing all of [05/Nov/2008:13:56:38 +0300] - libdb: the log files from a database environment [05/Nov/2008:13:56:38 +0300] - libdb: /var/lib/fedora-ds/slapd-ldap1/db/NetscapeRoot/id2entry.db4: unexpected file type or format [05/Nov/2008:13:56:38 +0300] - dbp->open("NetscapeRoot/id2entry.db4") failed: Invalid argument (22) [05/Nov/2008:13:56:38 +0300] - dblayer_instance_start fail: Invalid argument (22) [05/Nov/2008:13:56:38 +0300] - start: Failed to start databases, err=22 Invalid argument [05/Nov/2008:13:56:38 +0300] - Failed to allocate 10000000 byte dbcache. Please reduce nsslapd-cache-autosize and Restart the server. [05/Nov/2008:13:56:38 +0300] - Failed to start database plugin ldbm database [05/Nov/2008:13:56:38 +0300] - WARNING: ldbm instance userRoot already exists [05/Nov/2008:13:56:38 +0300] - WARNING: ldbm instance NetscapeRoot already exists [05/Nov/2008:13:56:38 +0300] binder-based resource limits - nsLookThroughLimit: parameter error (slapi_reslimit_register() already registered) [05/Nov/2008:13:56:38 +0300] - start: Resource limit registration failed [05/Nov/2008:13:56:38 +0300] - Failed to start database plugin ldbm database [05/Nov/2008:13:56:38 +0300] - Error: Failed to resolve plugin dependencies [05/Nov/2008:13:56:38 +0300] - Error: preoperation plugin 7-bit check is not started [05/Nov/2008:13:56:38 +0300] - Error: accesscontrol plugin ACL Plugin is not started [05/Nov/2008:13:56:38 +0300] - Error: preoperation plugin ACL preoperation is not started