Bug 1173273
Summary: | [RFE] BDB backend - clear free page files to reduce main db and changelog db size | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Noriko Hosoi <nhosoi> |
Component: | 389-ds-base | Assignee: | Noriko Hosoi <nhosoi> |
Status: | CLOSED ERRATA | QA Contact: | Viktor Ashirov <vashirov> |
Severity: | medium | Docs Contact: | |
Priority: | medium | ||
Version: | 7.0 | CC: | maci, nhosoi, nkinder, rmeggins, vashirov |
Target Milestone: | rc | Keywords: | FutureFeature |
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | 389-ds-base-1.3.3.1-10.el7 | Doc Type: | Enhancement |
Doc Text: |
Database file size is now redueced by releasing the deleted pages.
New configuration parameters are introduced to specify the interval of the compacting timing.
nsslapd-db-compactdb-interval is for the primary database; nsslapd-changelogcompactdb-interval is for the changelog.
|
Story Points: | --- |
Clone Of: | Environment: | ||
Last Closed: | 2015-03-05 09:40:16 UTC | Type: | --- |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 1187830 |
Description
Noriko Hosoi
2014-12-11 19:25:40 UTC
Steps to verify: Set up MMR. Set following config parameters some reasonable values, e.g., 240 (seconds) on one master. Please note that unless you set nsslapd-changelogmaxage in cn=changelog5 as well as nsds5ReplicaPurgeDelay and nsds5ReplicaTombstonePurgeInterval in the replica entry, the changelogs as well as the deleted entries won't be really removed from the dbs. Primary DBs (id2entry): dn: cn=config,cn=ldbm database,cn=plugins,cn=config nsslapd-db-compactdb-interval: 240 Changelog DBs: dn: cn=changelog5,cn=config nsslapd-changelogcompactdb-interval: 240 nsslapd-changelogmaxage: 200 dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config nsds5ReplicaPurgeDelay: 10 nsds5ReplicaTombstonePurgeInterval: 240 Run series of adds and deletes monitoring the size of id2entry and changelog db. If the size is reduced at the interval timing, the fix is verified. # ls -l /var/lib/dirsrv/slapd-M0/changelogdb/b9fd337e-092711e3-a8ce8123-6a055cd2_5212ab4f000000010000.db /var/lib/dirsrv/slapd-M0/db/userRoot/id2entry.db -rw-------. 1 nobody nobody 4186112 8/21 15:35 /var/lib/dirsrv/slapd-M0/changelogdb/b9fd337e-092711e3-a8ce8123-6a055cd2_5212ab4f000000010000.db -rw-------. 1 nobody nobody 15302656 8/21 15:35 /var/lib/dirsrv/slapd-M0/db/userRoot/id2entry.db # ls -l /var/lib/dirsrv/slapd-M0/changelogdb/b9fd337e-092711e3-a8ce8123-6a055cd2_5212ab4f000000010000.db /var/lib/dirsrv/slapd-M0/db/userRoot/id2entry.db -rw-------. 1 nobody nobody 49152 8/21 15:42 /var/lib/dirsrv/slapd-M0/changelogdb/b9fd337e-092711e3-a8ce8123-6a055cd2_5212ab4f000000010000.db -rw-------. 1 nobody nobody 1400832 8/21 15:44 /var/lib/dirsrv/slapd-M0/db/userRoot/id2entry.db Note: nsslapd-db-compactdb-interval & nsslapd-changelogcompactdb-interval need to be doc'ed in "Configuration Command and File Reference". I have few questions: What minimum values can be set for these attributes? nsslapd-db-compactdb-interval: 20 nsslapd-changelogcompactdb-interval: 20 nsslapd-changelogmaxage: 20 Does it look reasonable to test it? If yes, then it should clean the db every 20 secs. Are these values purgeDelay and Purge interval need to be set according to what values we give for compactdb-interval and changelogcompactdb-interval? This feature is somewhat performance related. So, how much of non-functional testing is required? or, may be testing 2 to 3 possible combinations of values would suffice? (In reply to Sankar Ramalingam from comment #3) > I have few questions: > > What minimum values can be set for these attributes? > > nsslapd-db-compactdb-interval: 20 > nsslapd-changelogcompactdb-interval: 20 > nsslapd-changelogmaxage: 20 > > Does it look reasonable to test it? I guess so. > If yes, then it should clean the db every 20 secs. "Cleaning the db" sounds a bit misleading. It releases the pages which have been occupied by the deleted data. That is, if you have no deleted data, nothing happens. Plus, if you have no DB page that contains only deleted data, nothing happens, either. So, to test these new config params: nsslapd-db-compactdb-interval: 20 nsslapd-changelogcompactdb-interval: 20 you have to add lots of entries and delete lots of them. Also plsease note that if you test it with the MMR, deleted entries are turned to be tombstone entries which are NOT really deleted from the database stand point. To delete them, you have to purge the tombstone entries. > Are these values purgeDelay and Purge interval need to be set according to > what values we give for compactdb-interval and changelogcompactdb-interval? As explained above, we want to "delete" the tombstone entries to test the compact db. So, these params have to be set to small enough values to free the tombstone entries from the database. nsds5ReplicaPurgeDelay: 10 nsds5ReplicaTombstonePurgeInterval: 240 > This feature is somewhat performance related. So, how much of non-functional > testing is required? or, may be testing 2 to 3 possible combinations of > values would suffice? Not really a performance. It's more of the disk usage issue. Probably, adding the compactdb + tombstone purge to some MMR stress, it'd be nice. We'd like to make sure it does not do any harm (crash or hang) to the server. will this make it into the el6 release ? The patch applied almost cleanly. It would be nice to have that in the RHDS9 tree. This would also solve an outstanding issue our customer has with RHDS Is this a blocking operation ? How do you think it would perform with a database approx. 400 GB in size ? Are there ways to do it externally ? like having a c program running db_compact on id2entry.db4 (i guess that would be the correct file) I tried with the tcl db4 bindings (stopped dirsrv) but didnt have any success. maybe i did it wrong tho. #! /usr/bin/tclsh load libdb_tcl-4.7.so berkdb open -btree id2entry.db4 db0 compact $ rpm -qa | grep 389 389-ds-base-libs-1.3.3.1-13.el7.x86_64 389-ds-base-1.3.3.1-13.el7.x86_64 389-ds-base-debuginfo-1.3.3.1-13.el7.x86_64 [0] Set up MMR. [1] Set following config parameters some reasonable values, e.g., 240 (seconds) on one master. $ ldapmodify -D "cn=Directory Manager" -w Secret123 -H ldap://localhost:1189 << EOF dn: cn=changelog5,cn=config changetype: modify add: nsslapd-changelogcompactdb-interval nsslapd-changelogcompactdb-interval: 240 - add: nsslapd-changelogmaxage nsslapd-changelogmaxage: 200 - dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config changetype: modify replace: nsds5ReplicaPurgeDelay nsds5ReplicaPurgeDelay: 10 - add: nsds5ReplicaTombstonePurgeInterval nsds5ReplicaTombstonePurgeInterval: 240 - dn: cn=config,cn=ldbm database,cn=plugins,cn=config changetype: modify replace: nsslapd-db-compactdb-interval nsslapd-db-compactdb-interval: 240 - EOF modifying entry "cn=changelog5,cn=config" modifying entry "cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config" modifying entry "cn=config,cn=ldbm database,cn=plugins,cn=config" [2] Run series of adds and deletes monitoring the size of id2entry and changelog db. Added and deleted 500 entries: $ ldclt -h localhost -p 1189 -D "cn=Directory Manager" -w Secret123 -f cn=MrXXXXX -b "ou=people,dc=example,dc=com" -e add,person,incr,noloop,commoncounter -r0 -R499 -n 1 -v <snip> $ ldclt -h localhost -p 1189 -D "cn=Directory Manager" -w Secret123 -f cn=MrXXXXX -b "ou=people,dc=example,dc=com" -e delete,person,incr,noloop,commoncounter -r0 -R499 -n 1 -v <snip> Size of databases before modifications: -rw-------. 1 vashirov vashirov 16384 Jan 30 20:07 /var/lib/dirsrv/slapd-M1/changelogdb/2c10d0b9-a8af11e4-a42be4e3-299c61fe_54cbcfa4000004cf0000.db -rw-------. 1 vashirov vashirov 16384 Jan 30 19:40 /var/lib/dirsrv/slapd-M1/db/userRoot/id2entry.db After modifications: -rw-------. 1 vashirov vashirov 401408 Jan 30 20:08 /var/lib/dirsrv/slapd-M1/changelogdb/2c10d0b9-a8af11e4-a42be4e3-299c61fe_54cbcfa4000004cf0000.db -rw-------. 1 vashirov vashirov 655360 Jan 30 20:08 /var/lib/dirsrv/slapd-M1/db/userRoot/id2entry.db After purging: -rw-------. 1 vashirov vashirov 16384 Jan 30 20:12 /var/lib/dirsrv/slapd-M1/changelogdb/2c10d0b9-a8af11e4-a42be4e3-299c61fe_54cbcfa4000004cf0000.db -rw-------. 1 vashirov vashirov 368640 Jan 30 20:16 /var/lib/dirsrv/slapd-M1/db/userRoot/id2entry.db Data was reduced, hence marking as VERIFIED. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-0416.html |