Bug 232050

Summary: Change format of DBVERSION and guardian files
Product: [Retired] 389 Reporter: Rich Megginson <rmeggins>
Component: Database - GeneralAssignee: Noriko Hosoi <nhosoi>
Status: CLOSED CURRENTRELEASE QA Contact: Viktor Ashirov <vashirov>
Severity: medium Docs Contact:
Priority: medium    
Version: 1.0.4CC: rmeggins
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-12-07 16:56:01 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 152373, 240316, 427409    
Attachments:
Description Flags
cvs diffs
none
cvs diffs
none
cvs diffs
none
cvs commit message
none
cvs commit message
none
cvs commit message none

Description Rich Megginson 2007-03-13 18:42:37 UTC
The DBVERSION and guardian files used by the database backends should convey
some information about the database implementation used (e.g. berkeley db,
sqlite), the version, the directory server database backend used, and any other
information pertinent to the server being able to use or upgrade the files.  The
following format is proposed:
implementation/version/server backend plugin name[/other tag][/other tag]....
For example:
bdb/4.2/libback-ldbm/newidl
This indicates that the files use Berkeley DB version 4.2, they are used by the
server libback-ldbm database plugin, and the index files use the newidl format.
 The other tags are optional and may be implementation specific.  At a minimum,
the file must contain the implementation/version/pluginname.

The reasons for changing the format are as follows:
1) Make it easy for humans and code to identify what the files are, how they are
used, and how to upgrade or migrate them
2) Separate the database file version and formats from the directory server
version, vendor, and brand
3) Allow new database implementations, plugins, and features e.g.
sqlite/3.0/libback-sqlite
4) A one line description is easy to read in and easy to parse

When the server reads in a database with an old format DBVERSION or guardian
file, it will change the file to use the new format and upgrade the data as
necessary.

Other related bugs:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=231905
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=231652

Comment 1 Noriko Hosoi 2007-03-14 00:39:26 UTC
Created attachment 150009 [details]
cvs diffs

Files:
 back-ldbm.h
 dblayer.c
 dbversion.c
 ldbm_config.c
 proto-back-ldbm.h
 upgrade.c

First attempt...
Changes:
1) introduced new strings for DBVERSION
2) added the logic to compare the new DBVERSION strings
   note: we don't store the current db version string in the backend source
code
   any more.  Instead, we get it from Berkeley DB header file db.h.
3) eliminated potential buffer overflow code in dbversion_read.

This is the DBVERSION file changes:
[before]
$ head DBVERSION */DBVERSION
==> DBVERSION <==
Netscape-ldbm/7.0

==> NetscapeRoot/DBVERSION <==
Netscape-ldbm/7.0

==> userRoot/DBVERSION <==
Netscape-ldbm/7.0

[after installing the new DS]
$ head DBVERSION */DBVERSION
==> DBVERSION <==
bdb/4.2/libback-ldbm/newidl

==> NetscapeRoot/DBVERSION <==
bdb/4.2/libback-ldbm/newidl

==> userRoot/DBVERSION <==
bdb/4.2/libback-ldbm/newidl

This is the guardian file change:
[before]
$ cat db/guardian
cachesize:8388608
ncache:0
version:3

[after]
$ cat db/guardian
cachesize:8388608
ncache:0
version:4

Actually, I lied in the IRC conversation, Rich.  The cachesize and ncache
values are used to learn the backend needs to change the dbcache configuration
or not.  So, I'm just setting the right db version number, which is not used
anyway...

The diff of dblayer.c contains the diff for "[231093] db2bak: crash bug".

Comment 2 Rich Megginson 2007-03-14 02:55:39 UTC
If newidl is the default, can we just remove /newidl from the version string,
and just assume newidl in the code?

I don't understand this code:
https://bugzilla.redhat.com/bugzilla/attachment.cgi?id=150009&action=diff#dbversion.c_sec2

    if (NULL == *ldbmversion || NULL == *dataversion) {
        return rc;

It seems that when this code is called, *ldbmversion and *dataversion will be NULL?
        char *ldbmversion = NULL;
        char *dataversion = NULL;
        if (dbversion_read(li, home_dir, &ldbmversion, &dataversion) != 0)




Comment 3 Noriko Hosoi 2007-03-14 17:03:45 UTC
(In reply to comment #2)
> If newidl is the default, can we just remove /newidl from the version string,
> and just assume newidl in the code?

Sure.  

> I don't understand this code:

Yeah, me, too... :p
>
https://bugzilla.redhat.com/bugzilla/attachment.cgi?id=150009&action=diff#dbversion.c_sec2
> 
>     if (NULL == *ldbmversion || NULL == *dataversion) {
>         return rc;

I meant this...  (and I'm moving the check for dataversion lower...)
     if (NULL == ldbmversion || NULL == dataversion) {
         return rc;

> 
> It seems that when this code is called, *ldbmversion and *dataversion will be
NULL?
>         char *ldbmversion = NULL;
>         char *dataversion = NULL;
>         if (dbversion_read(li, home_dir, &ldbmversion, &dataversion) != 0)

I'm attaching the new diffs next.



Comment 4 Noriko Hosoi 2007-03-14 23:03:19 UTC
Created attachment 150100 [details]
cvs diffs

Files:
 back-ldbm.h
 dblayer.c
 dbversion.c
 ldbm_config.c
 proto-back-ldbm.h
 start.c
 upgrade.c

Changes:
In addition to the changes described in Comment #1, fixed the bugs Rich pointed
out in the Comment #2 as well as eliminated the "newidl" keyword.  I also had
to make the macro DBVERSION_UPGRADE_4_4 separated from DBVERSION_UPGRADE_3_4. 
First, I though they could be shared since most operations are identical.  But
there was one exception: db4->db4 upgrade does not have to change the db
extension.

Comment 5 Noriko Hosoi 2007-03-14 23:09:25 UTC
I keep working on the changelog upgrade...

Comment 6 Noriko Hosoi 2007-03-15 20:39:18 UTC
Created attachment 150172 [details]
cvs diffs

Files:
 repl_shared.h
 cl5_api.h
 cl5_api.c

Changes:
1) change the format of changelogdb/{DBVERSION,guardian} files
   [before]
   Changelog5/NSMMReplicationPlugin/4.0
   [after]
   bdb/4.4/libreplication-plugin
   Note: changelog guardian file stores the same string as DBVERSION does,
   thus changed the format to follow the DBVERSION style.
   As being done for the backend, use DB_VERSION_MAJOR and _MINOR to get
   the current version of the Berkeley DB.
2) Added the 3-rd arg buflen to _cl5ReadDBVersion.
3) Added transaction removing code to _cl5Upgrade3_4
4) Added _cl5Upgrade4_4 which is a subset of _cl5Upgrade3_4.

Comment 7 Noriko Hosoi 2007-03-15 21:38:32 UTC
Created attachment 150177 [details]
cvs commit message

Commit message for Comment #4.	Checked in into HEAD.

Reviewed by Rich (Thank you!)

Comment 8 Noriko Hosoi 2007-03-15 21:42:12 UTC
Created attachment 150178 [details]
cvs commit message

Commit message for Comment #6.	Chacked in into HEAD.

Reviewed by Rich (Thank you!!)

Comment 9 Noriko Hosoi 2007-03-20 20:30:33 UTC
*** Bug 231652 has been marked as a duplicate of this bug. ***

Comment 10 Noriko Hosoi 2007-03-26 21:48:20 UTC
mrclone test case mrc_8 fails due to the server crashed caused by db2bak.pl. 
Observed on Solaris.

(dbx) bt
current thread: t@44
=>[1] dbversion_read(li = 0x100310d50, directory = 0x1006a2520 "/home/nhosoi/wor
k/tet/tetframework/testcases/DS/6.0/tet_tmp_dir/totoro/mrclone/data/Abak", ldbmv
ersion = 0xffffffff71a3fd2c, dataversion = 0xffffffff71a3fcec), line 181 in "dbv
ersion.c"
 [2] ldbm_back_archive2ldbm(pb = 0x1006a22c0), line 78 in "archive.c"
 [3] task_restore_thread(arg = 0x1006a22c0), line 1111 in "task.c"
 [4] _pt_root(0x1006a2590, 0xffffffffffffffff, 0x1, 0xffffffff7db41438, 0xfffff
fff7f225198, 0x3), at 0xffffffff7da34c88
(dbx) p t
t = 0xffffffff71a3f3b0 "bdb/4.2/libback-ldbm"
(dbx) p ldbmversion
ldbmversion = 0xffffffff71a3fd2c
(dbx) p *ldbmversion
*ldbmversion = (nil)
(dbx) up
Current function is ldbm_back_archive2ldbm
  78           if (dbversion_read(li, directory, dbversion, dataversion) != 0)

The cause of the crash is in the fix I made in the comment #7.  I missed one
place to adjust the arguments' type change.  The following change fixes the
crash in bak2db.

Index: archive.c
===================================================================
RCS file: /cvs/dirsec/ldapserver/ldap/servers/slapd/back-ldbm/archive.c,v
retrieving revision 1.11
diff -t -w -U4 -r1.11 archive.c
--- archive.c   10 Nov 2006 23:45:39 -0000      1.11
+++ archive.c   26 Mar 2007 17:37:44 -0000
@@ -70,13 +70,13 @@

     /* check the current idl format vs backup DB version */
     if (idl_get_idl_new())
     {
-        char dbversion[LDBM_VERSION_MAXBUF];
-        char dataversion[LDBM_VERSION_MAXBUF];
+        char *dbversion = NULL;
+        char *dataversion = NULL;
         int value = 0;

-        if (dbversion_read(li, directory, dbversion, dataversion) != 0)
+        if (dbversion_read(li, directory, &dbversion, &dataversion) != 0)
         {
             LDAPDebug(LDAP_DEBUG_ANY, "Warning: Unable to read dbversion "
                       "file in %s\n", directory, 0, 0);
         }
@@ -84,8 +84,10 @@
         if (value & DBVERSION_OLD_IDL)
         {
             is_old_to_new = 1;
         }
+        slapi_ch_free_string(&dbversion);
+        slapi_ch_free_string(&dataversion);
     }

     /* No ldbm be's exist until we process the config information. */
     if (run_from_cmdline) {

Comment 11 Noriko Hosoi 2007-03-26 23:05:16 UTC
Created attachment 150974 [details]
cvs commit message

Reviewed by Rich (Thank you!)

Checked in into HEAD.

Comment 12 Anh Nguyen 2007-12-04 18:53:53 UTC
Verified on Rhel5_32bit machine.
cd /var/lib/dirsrv/slapd-babylon1/db
[root@babylon1 db]# cat DBVERSION userRoot/DBVERSION 
bdb/4.3/libback-ldbm
bdb/4.3/libback-ldbm
[root@babylon1 db]#