The DBVERSION and guardian files used by the database backends should convey some information about the database implementation used (e.g. berkeley db, sqlite), the version, the directory server database backend used, and any other information pertinent to the server being able to use or upgrade the files. The following format is proposed: implementation/version/server backend plugin name[/other tag][/other tag].... For example: bdb/4.2/libback-ldbm/newidl This indicates that the files use Berkeley DB version 4.2, they are used by the server libback-ldbm database plugin, and the index files use the newidl format. The other tags are optional and may be implementation specific. At a minimum, the file must contain the implementation/version/pluginname. The reasons for changing the format are as follows: 1) Make it easy for humans and code to identify what the files are, how they are used, and how to upgrade or migrate them 2) Separate the database file version and formats from the directory server version, vendor, and brand 3) Allow new database implementations, plugins, and features e.g. sqlite/3.0/libback-sqlite 4) A one line description is easy to read in and easy to parse When the server reads in a database with an old format DBVERSION or guardian file, it will change the file to use the new format and upgrade the data as necessary. Other related bugs: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=231905 https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=231652
Created attachment 150009 [details] cvs diffs Files: back-ldbm.h dblayer.c dbversion.c ldbm_config.c proto-back-ldbm.h upgrade.c First attempt... Changes: 1) introduced new strings for DBVERSION 2) added the logic to compare the new DBVERSION strings note: we don't store the current db version string in the backend source code any more. Instead, we get it from Berkeley DB header file db.h. 3) eliminated potential buffer overflow code in dbversion_read. This is the DBVERSION file changes: [before] $ head DBVERSION */DBVERSION ==> DBVERSION <== Netscape-ldbm/7.0 ==> NetscapeRoot/DBVERSION <== Netscape-ldbm/7.0 ==> userRoot/DBVERSION <== Netscape-ldbm/7.0 [after installing the new DS] $ head DBVERSION */DBVERSION ==> DBVERSION <== bdb/4.2/libback-ldbm/newidl ==> NetscapeRoot/DBVERSION <== bdb/4.2/libback-ldbm/newidl ==> userRoot/DBVERSION <== bdb/4.2/libback-ldbm/newidl This is the guardian file change: [before] $ cat db/guardian cachesize:8388608 ncache:0 version:3 [after] $ cat db/guardian cachesize:8388608 ncache:0 version:4 Actually, I lied in the IRC conversation, Rich. The cachesize and ncache values are used to learn the backend needs to change the dbcache configuration or not. So, I'm just setting the right db version number, which is not used anyway... The diff of dblayer.c contains the diff for "[231093] db2bak: crash bug".
If newidl is the default, can we just remove /newidl from the version string, and just assume newidl in the code? I don't understand this code: https://bugzilla.redhat.com/bugzilla/attachment.cgi?id=150009&action=diff#dbversion.c_sec2 if (NULL == *ldbmversion || NULL == *dataversion) { return rc; It seems that when this code is called, *ldbmversion and *dataversion will be NULL? char *ldbmversion = NULL; char *dataversion = NULL; if (dbversion_read(li, home_dir, &ldbmversion, &dataversion) != 0)
(In reply to comment #2) > If newidl is the default, can we just remove /newidl from the version string, > and just assume newidl in the code? Sure. > I don't understand this code: Yeah, me, too... :p > https://bugzilla.redhat.com/bugzilla/attachment.cgi?id=150009&action=diff#dbversion.c_sec2 > > if (NULL == *ldbmversion || NULL == *dataversion) { > return rc; I meant this... (and I'm moving the check for dataversion lower...) if (NULL == ldbmversion || NULL == dataversion) { return rc; > > It seems that when this code is called, *ldbmversion and *dataversion will be NULL? > char *ldbmversion = NULL; > char *dataversion = NULL; > if (dbversion_read(li, home_dir, &ldbmversion, &dataversion) != 0) I'm attaching the new diffs next.
Created attachment 150100 [details] cvs diffs Files: back-ldbm.h dblayer.c dbversion.c ldbm_config.c proto-back-ldbm.h start.c upgrade.c Changes: In addition to the changes described in Comment #1, fixed the bugs Rich pointed out in the Comment #2 as well as eliminated the "newidl" keyword. I also had to make the macro DBVERSION_UPGRADE_4_4 separated from DBVERSION_UPGRADE_3_4. First, I though they could be shared since most operations are identical. But there was one exception: db4->db4 upgrade does not have to change the db extension.
I keep working on the changelog upgrade...
Created attachment 150172 [details] cvs diffs Files: repl_shared.h cl5_api.h cl5_api.c Changes: 1) change the format of changelogdb/{DBVERSION,guardian} files [before] Changelog5/NSMMReplicationPlugin/4.0 [after] bdb/4.4/libreplication-plugin Note: changelog guardian file stores the same string as DBVERSION does, thus changed the format to follow the DBVERSION style. As being done for the backend, use DB_VERSION_MAJOR and _MINOR to get the current version of the Berkeley DB. 2) Added the 3-rd arg buflen to _cl5ReadDBVersion. 3) Added transaction removing code to _cl5Upgrade3_4 4) Added _cl5Upgrade4_4 which is a subset of _cl5Upgrade3_4.
Created attachment 150177 [details] cvs commit message Commit message for Comment #4. Checked in into HEAD. Reviewed by Rich (Thank you!)
Created attachment 150178 [details] cvs commit message Commit message for Comment #6. Chacked in into HEAD. Reviewed by Rich (Thank you!!)
*** Bug 231652 has been marked as a duplicate of this bug. ***
mrclone test case mrc_8 fails due to the server crashed caused by db2bak.pl. Observed on Solaris. (dbx) bt current thread: t@44 =>[1] dbversion_read(li = 0x100310d50, directory = 0x1006a2520 "/home/nhosoi/wor k/tet/tetframework/testcases/DS/6.0/tet_tmp_dir/totoro/mrclone/data/Abak", ldbmv ersion = 0xffffffff71a3fd2c, dataversion = 0xffffffff71a3fcec), line 181 in "dbv ersion.c" [2] ldbm_back_archive2ldbm(pb = 0x1006a22c0), line 78 in "archive.c" [3] task_restore_thread(arg = 0x1006a22c0), line 1111 in "task.c" [4] _pt_root(0x1006a2590, 0xffffffffffffffff, 0x1, 0xffffffff7db41438, 0xfffff fff7f225198, 0x3), at 0xffffffff7da34c88 (dbx) p t t = 0xffffffff71a3f3b0 "bdb/4.2/libback-ldbm" (dbx) p ldbmversion ldbmversion = 0xffffffff71a3fd2c (dbx) p *ldbmversion *ldbmversion = (nil) (dbx) up Current function is ldbm_back_archive2ldbm 78 if (dbversion_read(li, directory, dbversion, dataversion) != 0) The cause of the crash is in the fix I made in the comment #7. I missed one place to adjust the arguments' type change. The following change fixes the crash in bak2db. Index: archive.c =================================================================== RCS file: /cvs/dirsec/ldapserver/ldap/servers/slapd/back-ldbm/archive.c,v retrieving revision 1.11 diff -t -w -U4 -r1.11 archive.c --- archive.c 10 Nov 2006 23:45:39 -0000 1.11 +++ archive.c 26 Mar 2007 17:37:44 -0000 @@ -70,13 +70,13 @@ /* check the current idl format vs backup DB version */ if (idl_get_idl_new()) { - char dbversion[LDBM_VERSION_MAXBUF]; - char dataversion[LDBM_VERSION_MAXBUF]; + char *dbversion = NULL; + char *dataversion = NULL; int value = 0; - if (dbversion_read(li, directory, dbversion, dataversion) != 0) + if (dbversion_read(li, directory, &dbversion, &dataversion) != 0) { LDAPDebug(LDAP_DEBUG_ANY, "Warning: Unable to read dbversion " "file in %s\n", directory, 0, 0); } @@ -84,8 +84,10 @@ if (value & DBVERSION_OLD_IDL) { is_old_to_new = 1; } + slapi_ch_free_string(&dbversion); + slapi_ch_free_string(&dataversion); } /* No ldbm be's exist until we process the config information. */ if (run_from_cmdline) {
Created attachment 150974 [details] cvs commit message Reviewed by Rich (Thank you!) Checked in into HEAD.
Verified on Rhel5_32bit machine. cd /var/lib/dirsrv/slapd-babylon1/db [root@babylon1 db]# cat DBVERSION userRoot/DBVERSION bdb/4.3/libback-ldbm bdb/4.3/libback-ldbm [root@babylon1 db]#