Bug 1109339

Summary:	Nested tombstones become orphaned after purge
Product:	Red Hat Enterprise Linux 7	Reporter:	Noriko Hosoi <nhosoi>
Component:	389-ds-base	Assignee:	Noriko Hosoi <nhosoi>
Status:	CLOSED ERRATA	QA Contact:	Viktor Ashirov <vashirov>
Severity:	unspecified	Docs Contact:
Priority:	medium
Version:	7.1	CC:	mreynolds, nkinder, rmeggins, vashirov
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	389-ds-base-1.3.3.1-1.el7	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1109337	Environment:
Last Closed:	2015-03-05 09:35:21 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1109337
Bug Blocks:

Description Noriko Hosoi 2014-06-13 17:20:52 UTC

+++ This bug was initially created as a clone of Bug #1109337 +++

This bug is created as a clone of upstream ticket:
https://fedorahosted.org/389/ticket/47767

When tombstone purging occurs it will remove a parent tombstone, which leaves any child tombstone entries orphaned.

Comment 2 Viktor Ashirov 2015-01-26 18:05:52 UTC

Using verification steps from https://bugzilla.redhat.com/show_bug.cgi?id=1109337#c0

[0] Set up MMR
[1] Add a tree to the master, e.g.,
  ou=A,<suffix>
  ou=subA,ou=A,<suffix>
  ou=subsubA,ou=subA,ou=A,<suffix>
  uid=user1,ou=subsubA,ou=subA,ou=A,<suffix>

$ ldapmodify -D "cn=Directory Manager" -w Secret123  -H ldap://localhost:1189 -a << EOF
dn: ou=A,dc=example,dc=com
ou: People
ou: A
objectClass: top
objectClass: organizationalunit

dn: ou=subA,ou=A,dc=example,dc=com
ou: People
ou: subA
objectClass: top
objectClass: organizationalunit

dn: ou=subsubA,ou=subA,ou=A,dc=example,dc=com
ou: People
ou: subsubA
objectClass: top
objectClass: organizationalunit

dn: uid=user,ou=subsubA,ou=subA,ou=A,dc=example,dc=com
uid: user
givenName: user
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetorgperson
sn: user
cn: user
EOF
adding new entry "ou=A,dc=example,dc=com"

adding new entry "ou=subA,ou=A,dc=example,dc=com"

adding new entry "ou=subsubA,ou=subA,ou=A,dc=example,dc=com"

adding new entry "uid=user,ou=subsubA,ou=subA,ou=A,dc=example,dc=com"

[2] Set small number (e.g., 30 sec) to these config params on all the servers.
nsds5ReplicaPurgeDelay
nsds5ReplicaTombstonePurgeInterval

$ ldapmodify -D "cn=Directory Manager" -w Secret123  -H ldap://localhost:1189 -a << EOF
dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
changetype: modify
replace: nsds5ReplicaPurgeDelay
nsds5ReplicaPurgeDelay: 10
-
replace: nsds5ReplicaTombstonePurgeInterval
nsds5ReplicaTombstonePurgeInterval: 10
-
EOF
modifying entry "cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config"

$ ldapmodify -D "cn=Directory Manager" -w Secret123  -H ldap://localhost:1289 -a << EOF
dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
changetype: modify
replace: nsds5ReplicaPurgeDelay
nsds5ReplicaPurgeDelay: 10
-
replace: nsds5ReplicaTombstonePurgeInterval
nsds5ReplicaTombstonePurgeInterval: 10
-
EOF
modifying entry "cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config"

[3] Restart the servers.
$ sudo systemctl restart dirsrv.target

[4] Delete the tree.
$ ldapdelete -D "cn=Directory Manager"  -w Secret123  -H ldap://localhost:1189 -r "ou=A,dc=example,dc=com"

[5] Check all the entries in the tree are turned to be a tombstone entry.
$ ldapsearch -LLL -o ldif-wrap=no -D 'cn=Directory Manager' -w Secret123 -H ldap://localhost:1189 -b dc=example,dc=com '(objectclass=nsTombstone)'
dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
objectClass: nsDS5Replica
objectClass: top
nsDS5ReplicaRoot: dc=example,dc=com
nsDS5ReplicaType: 3
nsDS5Flags: 1
nsds5ReplicaPurgeDelay: 10
nsDS5ReplicaId: 1231
nsDS5ReplicaBindDN: cn=SyncManager,cn=config
cn: replica
nsState:: zwQAAAAAAAAjd8ZUAAAAAAAAAAAAAAAAAAAAAAAAAAACAAAAAAAAAA==
nsDS5ReplicaName: 5f85ef39-a57f11e4-ba92a3c2-663fcbbb
nsds5ReplicaTombstonePurgeInterval: 10
nsds50ruv: {replicageneration} 54c676f5000004cf0000
nsds50ruv: {replica 1231 ldap://rhel7.brq.redhat.com:1189} 54c67740000004cf0000 54c67740000804cf0000
nsds50ruv: {replica 1232 ldap://rhel7.brq.redhat.com:1289}
nsds5agmtmaxcsn: dc=example,dc=com;1189_to_1626_on_rhel7.brq.redhat.com;rhel7.brq.redhat.com;1626;1232;54c67740000804cf0000
nsruvReplicaLastModified: {replica 1231 ldap://rhel7.brq.redhat.com:1189} 54c67740
nsruvReplicaLastModified: {replica 1232 ldap://rhel7.brq.redhat.com:1289} 00000000
nsds5ReplicaChangeCount: 8
nsds5replicareapactive: 0

dn: nsuniqueid=7d535426-a57f11e4-ba92a3c2-663fcbbb,ou=A,dc=example,dc=com
ou: People
ou: A
objectClass: top
objectClass: organizationalunit
objectClass: nsTombstone
nsParentUniqueId: b2f73d00-a57d11e4-b70d837b-f5154b0c
nstombstonecsn: 54c67740000804cf0000

dn: nsuniqueid=7d535427-a57f11e4-ba92a3c2-663fcbbb,ou=subA,ou=A,dc=example,dc=com
ou: People
ou: subA
objectClass: top
objectClass: organizationalunit
objectClass: nsTombstone
nsParentUniqueId: 7d535426-a57f11e4-ba92a3c2-663fcbbb
nstombstonecsn: 54c67740000704cf0000

dn: nsuniqueid=7d535428-a57f11e4-ba92a3c2-663fcbbb,ou=subsubA,ou=subA,ou=A,dc=example,dc=com
ou: People
ou: subsubA
objectClass: top
objectClass: organizationalunit
objectClass: nsTombstone
nsParentUniqueId: 7d535427-a57f11e4-ba92a3c2-663fcbbb
nstombstonecsn: 54c67740000604cf0000

dn: nsuniqueid=7d535429-a57f11e4-ba92a3c2-663fcbbb,uid=user,ou=subsubA,ou=subA,ou=A,dc=example,dc=com
uid: user
givenName: user
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetorgperson
objectClass: nsTombstone
sn: user
cn: user
nsParentUniqueId: 7d535428-a57f11e4-ba92a3c2-663fcbbb
nstombstonecsn: 54c67740000504cf0000

[6] Wait for the second set to the purge interval.
[7] Do some modify operation such as add another dummy entry.
$ ldapmodify -D "cn=Directory Manager" -w Secret123  -H ldap://localhost:1189 -a << EOF
dn: ou=People,dc=example,dc=com
changetype: modify
replace: description
description: dummy change
EOF
modifying entry "ou=People,dc=example,dc=com"

[8] Check the tombstone entries which are supposed to have disappeared.
If you see no tombstone entries (especially any orphaned tombstone entry), the fix was verified.

Tombstone entries are still in place after purge delay, unless I perform *two* modify operations in [7]. Off-by-one error somewhere?

After second modify all tombstone entries were purged after configured delay:

$ ldapsearch -LLL -o ldif-wrap=no -D 'cn=Directory Manager' -w Secret123 -H ldap://localhost:1189 -b dc=example,dc=com '(objectclass=nsTombstone)'
dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
objectClass: nsDS5Replica
objectClass: top
nsDS5ReplicaRoot: dc=example,dc=com
nsDS5ReplicaType: 3
nsDS5Flags: 1
nsds5ReplicaPurgeDelay: 10
nsDS5ReplicaId: 1231
nsDS5ReplicaBindDN: cn=SyncManager,cn=config
cn: replica
nsState:: zwQAAAAAAABDd8ZUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==
nsDS5ReplicaName: 5f85ef39-a57f11e4-ba92a3c2-663fcbbb
nsds5ReplicaTombstonePurgeInterval: 10
nsds50ruv: {replicageneration} 54c676f5000004cf0000
nsds50ruv: {replica 1231 ldap://rhel7.brq.redhat.com:1189} 54c67740000004cf0000 54c67755000004cf0000
nsds50ruv: {replica 1232 ldap://rhel7.brq.redhat.com:1289}
nsds5agmtmaxcsn: dc=example,dc=com;1189_to_1626_on_rhel7.brq.redhat.com;rhel7.brq.redhat.com;1626;1232;54c67755000004cf0000
nsruvReplicaLastModified: {replica 1231 ldap://rhel7.brq.redhat.com:1189} 54c67755
nsruvReplicaLastModified: {replica 1232 ldap://rhel7.brq.redhat.com:1289} 00000000
nsds5ReplicaChangeCount: 10
nsds5replicareapactive: 0

Comment 3 Viktor Ashirov 2015-01-26 18:06:32 UTC

Build tested: 
$ rpm -qa | grep 389
389-ds-base-1.3.3.1-12.el7.x86_64
389-ds-base-libs-1.3.3.1-12.el7.x86_64
389-ds-base-debuginfo-1.3.3.1-12.el7.x86_64

Comment 4 Viktor Ashirov 2015-01-27 11:07:57 UTC

Hello Mark, 

tombstone entries are still in place after purge delay, unless I perform *two* modify operations in [7]. Is this OK?

Comment 5 mreynolds 2015-01-27 14:12:36 UTC

(In reply to Viktor Ashirov from comment #4)
> Hello Mark, 
> 
> tombstone entries are still in place after purge delay, unless I perform
> *two* modify operations in [7]. Is this OK?

This is fine, it was probably a timing issue.  The original bug would keep those tombstones around indefinitely - they would never be removed.

Comment 6 Viktor Ashirov 2015-01-27 14:18:39 UTC

Thank you, Mark!

Marking this bug as VERIFIED then.

Comment 8 errata-xmlrpc 2015-03-05 09:35:21 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0416.html