Bug 1079574

Summary: delete on leaf fails with error 66 not allowed on non-leaf
Product: Red Hat Enterprise Linux 6 Reporter: Marc Sauton <msauton>
Component: 389-ds-baseAssignee: Noriko Hosoi <nhosoi>
Status: CLOSED DUPLICATE QA Contact: Sankar Ramalingam <sramling>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 6.5CC: akaiser, jgalipea, mreynolds, msauton, nkinder, rmeggins
Target Milestone: pre-dev-freeze   
Target Release: 6.6   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-04-21 14:56:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
LDID test exmple, was exported in 2xMMR env. after some delete op. none

Description Marc Sauton 2014-03-21 21:07:56 UTC 
Description of problem:

This problem looks like
bz 947583 - ldapdelete returns non-leaf entry error while trying to remove a leaf entry
which is supposed to be fixed in 389-ds-base-1.2.11.15-22.el6

using 389-ds-base-1.2.11.15-31.el6_5.x86_64 we seem to see the very same symptoms, impossible to delete a leaf entry, err=66, when in a replication environment.

dbscan -f /var/lib/dirsrv/slapd-m1/db/test1/numsubordinates.db4
+

ldapdelete -x -D "cn=directory manager" -w password ou=people,dc=test1,dc=example,dc=com -v -r
ldap_initialize( ldap://11.11.11.11 )
deleting entry "ou=people,dc=test1,dc=example,dc=com"
deleting children of: ou=people,dc=test1,dc=example,dc=com
ldap_delete: Operation not allowed on non-leaf (66)



Version-Release number of selected component (if applicable):

389-ds-base-1.2.11.15-31.el6_5.x86_64
db4-4.7.25-18.el6_4.x86_64
redhat-release-server-6Server-6.5.0.1.el6.x86_64


How reproducible:
seem inconsistent, sometimes I can delete an empty ou=people, sometimes not (with err=6), it seem to mostly happen after deleting entrie, add back, del

Steps to Reproduce:
1. 2xMMR
2. have suffix, test data
3. del entries from suffix, leave suffix empty, example:
ldapsearch -LLLx -D "cn=directory manager" -w password -b "ou=people,dc=test1,dc=example,dc=com" numsubordinates tombstonenumsubordinates hassubordinates
dn: ou=people,dc=test1,dc=example,dc=com
numsubordinates: 100000
tombstonenumsubordinates: 0
hassubordinates: TRUE

manage to reap tombstone by tuning down nsDS5ReplicaPurgeDelay and nsDS5ReplicaTombstonePurgeInterval
so no more tombstones:

ldapsearch -LLLx -D "cn=directory manager" -w password -b dc=test1,dc=example,dc=com objectclass=nstombstone dn
# ffffffff-ffffffff-ffffffff-ffffffff, dc=example,dc=com
dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=test1,dc=example,dc=com

the ou=people has no entries, it is a leaf:

ldapsearch -LLLx -D "cn=directory manager" -w password -b "ou=people,dc=test1,dc=example,dc=com" dn
dn: ou=people,dc=test1,dc=example,dc=com


Actual results:

ldapdelete -xh 10.14.5.27 -D "cn=directory manager" -w password ou=people,dc=test1,dc=example,dc=com -v
ldap_initialize( ldap://10.14.5.27 )
deleting entry "ou=people,dc=test1,dc=example,dc=com"
ldap_delete: Operation not allowed on non-leaf (66)



Expected results:

able to delete a leaf, able to delete recursively a container


Additional info:

re-indexing numsubordinates does not help, a search till returns numsubordinates: 100000 and delete err=66

/usr/lib64/dirsrv/slapd-m1/db2index.pl -D 'cn=directory manager' -w password -n test1 -t numsubordinates
adding new entry "cn=db2index_2014_3_21_12_9_28, cn=index, cn=tasks, cn=config"

less /var/log/dirsrv/slapd-m1/errors
[21/Mar/2014:12:09:28 -0700] - test1: Indexing attribute: numsubordinates
[21/Mar/2014:12:09:28 -0700] - test1: Finished indexing.
(END)

ls -l /var/lib/dirsrv/slapd-m1/db/test1/numsubordinates.db4
-rw-------. 1 nobody nobody 16384 Mar 21 12:10 /var/lib/dirsrv/slapd-m1/db/test1/numsubordinates.db4

this output looks ok, just dc=test1,dc=example,dc=com and ou=people,dc=test1,dc=example,dc=com

dbscan -f /var/lib/dirsrv/slapd-m1/db/test1/numsubordinates.db4 -rk +
+                                       
        1 2
Comment 2 Marc Sauton 2014-03-24 20:08:50 UTC 
some testing notes about indexes sanity check in small test:

Indexes looks correct:

dbscan -f /var/lib/dirsrv/slapd-m1/db/test1/id2entry.db4 |less
id 1
        rdn: dc=test1,dc=example,dc=com
        nsUniqueId: b004a701-a25d11e3-9836db48-7f346edb
...
id 2
        rdn: ou=people
        nsUniqueId: b004a702-a25d11e3-9836db48-7f346edb
...
id 641  
        rdn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff


dbscan -f /var/lib/dirsrv/slapd-m1/db/test1/parentid.db4 -rk =1
=1
        2 641

dbscan -f /var/lib/dirsrv/slapd-m1/db/test1/nsuniqueid.db4
=b004a701-a25d11e3-9836db48-7f346edb
=b004a702-a25d11e3-9836db48-7f346edb
=ffffffff-ffffffff-ffffffff-ffffffff

dbscan -f /var/lib/dirsrv/slapd-m1/db/test1/nsuniqueid.db4  -rk =b004a702-a25d11e3-9836db48-7f346edb
=b004a702-a25d11e3-9836db48-7f346edb
        2

dbscan -f /var/lib/dirsrv/slapd-m1/db/test1/objectclass.db4 -rk =nstombstone |less
=nstombstone
        641

dbscan -f /var/lib/dirsrv/slapd-m1/db/test1/objectclass.db4 -rk =organizationalunit |less
=organizationalunit
        2
Comment 3 Noriko Hosoi 2014-03-24 20:39:45 UTC 
So, Marc.  Were you able to reproduce the problem in house?  If yes, could you do these?
Could your run these?
dbscan -f /var/lib/dirsrv/slapd-m1/db/test1/id2entry.db4 -K 2 # -K is upper case
dbscan -f /var/lib/dirsrv/slapd-m1/db/test1/entryrdn.db -k 8  # -k is lower case

Thanks!
--noriko

(In reply to Marc Sauton from comment #2)
> some testing notes about indexes sanity check in small test:
> 
> Indexes looks correct:
> 
> dbscan -f /var/lib/dirsrv/slapd-m1/db/test1/id2entry.db4 |less
> id 1
>         rdn: dc=test1,dc=example,dc=com
>         nsUniqueId: b004a701-a25d11e3-9836db48-7f346edb
> ...
> id 2
>         rdn: ou=people
>         nsUniqueId: b004a702-a25d11e3-9836db48-7f346edb
> ...
> id 641  
>         rdn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff
> 
> 
> dbscan -f /var/lib/dirsrv/slapd-m1/db/test1/parentid.db4 -rk =1
> =1
>         2 641
> 
> dbscan -f /var/lib/dirsrv/slapd-m1/db/test1/nsuniqueid.db4
> =b004a701-a25d11e3-9836db48-7f346edb
> =b004a702-a25d11e3-9836db48-7f346edb
> =ffffffff-ffffffff-ffffffff-ffffffff
> 
> dbscan -f /var/lib/dirsrv/slapd-m1/db/test1/nsuniqueid.db4  -rk
> =b004a702-a25d11e3-9836db48-7f346edb
> =b004a702-a25d11e3-9836db48-7f346edb
>         2
> 
> dbscan -f /var/lib/dirsrv/slapd-m1/db/test1/objectclass.db4 -rk =nstombstone
> |less
> =nstombstone
>         641
> 
> dbscan -f /var/lib/dirsrv/slapd-m1/db/test1/objectclass.db4 -rk
> =organizationalunit |less
> =organizationalunit
>         2
Comment 4 Marc Sauton 2014-03-25 05:30:33 UTC 
I modify the data in the test since last comments, but it is similar, and still "strange":

ldapsearch -LLLxh 10.14.5.27 -p 389 -D "cn=Directory Manager" -w password -b ou=people,dc=test1,dc=example,dc=com dn
dn: ou=people,dc=test1,dc=example,dc=com


ldapsearch -LLLxh 10.14.5.27 -p 389 -D "cn=Directory Manager" -w password -b dc=test1,dc=example,dc=com objectClass=nstombstone dn
dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=test1,dc=example,dc=com


ldapsearch -LLLxh 10.14.5.27 -p 389 -D "cn=Directory Manager" -w password -b ou=people,dc=test1,dc=example,dc=com -s base objectClass=* hassubordinates numSubordinates tombstonenumsubordinates
dn: ou=people,dc=test1,dc=example,dc=com
hassubordinates: TRUE
numSubordinates: 68
tombstonenumsubordinates: 0



ldapdelete -xh 10.14.5.27 -D "cn=directory manager" -w password ou=people,dc=test1,dc=example,dc=com -v -r
ldap_initialize( ldap://10.14.5.27 )
deleting entry "ou=people,dc=test1,dc=example,dc=com"
deleting children of: ou=people,dc=test1,dc=example,dc=com
ldap_delete: Operation not allowed on non-leaf (66)


dbscan -f /var/lib/dirsrv/slapd-m1/db/test1/id2entry.db4 |less
id 1
        rdn: dc=test1,dc=example,dc=com
        nsUniqueId: b004a701-a25d11e3-9836db48-7f346edb
        dc: test1
        objectClass: dcObject
        objectClass: top
        creatorsName:
        modifiersName:
        createTimestamp: 20140321235843Z
        modifyTimestamp: 20140321235843Z
        entryid: 1
        numSubordinates: 2
        
id 2
        rdn: ou=people
        modifyTimestamp;adcsn-53311e760000000a0000;vucsn-53311e760000000a0000: 2014032
         5051306Z
        modifiersName;adcsn-53311e760000000a0000;vucsn-53311e760000000a0000: cn=direct
         ory manager
        description;adcsn-53311e760000000a0000;vucsn-53311e760000000a0000: test
        nsUniqueId: b004a702-a25d11e3-9836db48-7f346edb
        ou: people
        objectClass: organizationalUnit
        objectClass: top
        creatorsName:
        createTimestamp: 20140321235843Z
        parentid: 1
        entryid: 2
        numSubordinates: 68
        tombstoneNumSubordinates: 0
        
id 641
        rdn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff
        nsUniqueId: ffffffff-ffffffff-ffffffff-ffffffff
        objectClass: top
        objectClass: nsTombstone
        objectClass: extensibleobject
        nsds50ruv: {replicageneration} 5313b6dd0000000a0000
        nsds50ruv: {replica 10 ldap://m1.example.com:389} 5313c7290000000a0000 53311e7
         60000000a0000
        nsds50ruv: {replica 11 ldap://m2.example.com:389} 5313c97d0000000b0000 5313c98
         90001000b0000
        dc: test1
        nsruvReplicaLastModified: {replica 10 ldap://m1.example.com:389} 53311062
        nsruvReplicaLastModified: {replica 11 ldap://m2.example.com:389} 00000000
        creatorsName:
        modifiersName:
        createTimestamp: 20140321235843Z
        modifyTimestamp: 20140321235843Z
        parentid: 1
        entryid: 641
        
(END) 


and if I do a
/usr/lib64/dirsrv/slapd-m1/db2ldif.pl -D "cn=directory manager" -w password -r -a /var/tmp/m1.test3.dc.test1.dc.example.some.del.db2ldif.online.with.r.option.ldif -s dc=test1,dc=example,dc=com
I get the same 3 entries with id 1, 2, 641, not sure why the numSubordinates: 68 on entry id 2 (ou=people)


dbscan -f /var/lib/dirsrv/slapd-m1/db/test1/id2entry.db4 -K 2
id 2
        rdn: ou=people
        modifyTimestamp;adcsn-53311e760000000a0000;vucsn-53311e760000000a0000: 2014032
         5051306Z
        modifiersName;adcsn-53311e760000000a0000;vucsn-53311e760000000a0000: cn=direct
         ory manager
        description;adcsn-53311e760000000a0000;vucsn-53311e760000000a0000: test
        nsUniqueId: b004a702-a25d11e3-9836db48-7f346edb
        ou: people
        objectClass: organizationalUnit
        objectClass: top
        creatorsName:
        createTimestamp: 20140321235843Z
        parentid: 1
        entryid: 2
        numSubordinates: 68
        tombstoneNumSubordinates: 0



what key did you want to see in the entryrdn? 2?

dbscan -f /var/lib/dirsrv/slapd-m1/db/test1/entryrdn.db4 |less
2
  ID: 2; RDN: "ou=people"; NRDN: "ou=people"
641
  ID: 641; RDN: "nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff"; NRDN: "nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff"
C1
  ID: 641; RDN: "nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff"; NRDN: "nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff"
C1
  ID: 2; RDN: "ou=people"; NRDN: "ou=people"
P2
  ID: 1; RDN: "dc=test1,dc=example,dc=com"; NRDN: "dc=test1,dc=example,dc=com"
P641
  ID: 1; RDN: "dc=test1,dc=example,dc=com"; NRDN: "dc=test1,dc=example,dc=com"
dc=test1,dc=example,dc=com
  ID: 1; RDN: "dc=test1,dc=example,dc=com"; NRDN: "dc=test1,dc=example,dc=com"
(END)
Comment 24 Marc Sauton 2014-04-14 15:56:33 UTC 
Created attachment 886175 [details]
LDID test exmple, was exported in 2xMMR env. after some delete op.
Comment 25 Marc Sauton 2014-04-14 16:37:07 UTC 
ldapsearch -xh 10.14.5.27 -D "cn=directory manager" -w password -b "ou=people,dc=test1,dc=example,dc=com" -s one objectclass=* dn | grep "dn: " | wc -l
0

ldapsearch -xh 10.14.5.27 -D "cn=directory manager" -w password -b "ou=people,dc=test1,dc=example,dc=com" -s base objectclass=*  numsubordinates
# extended LDIF
#
# LDAPv3
# base <ou=people,dc=test1,dc=example,dc=com> with scope baseObject
# filter: objectclass=*
# requesting: numsubordinates 
#

# people, test1.example.com
dn: ou=people,dc=test1,dc=example,dc=com
numsubordinates: 1

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1
Comment 27 Marc Sauton 2014-04-14 16:57:33 UTC 
test notes withOUT (yet) the new patch from comment 26


dn: cn=replica,cn=dc\3Dtest1\2Cdc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
changetype: modify
replace: nsds5ReplicaPurgeDelay
nsds5ReplicaPurgeDelay: 40
-
replace: nsDS5ReplicaTombstonePurgeInterval
nsDS5ReplicaTombstonePurgeInterval: 40


ldapmodify -xh 10.14.5.27 -D "cn=directory manager" -w password -f modify.purge.delay.ldif


ldapsearch -LLLxh 10.14.5.27 -D "cn=directory manager" -w password -b "cn=replica,cn=dc\3Dtest1\2Cdc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config"  nsds5ReplicaPurgeDelay nsDS5ReplicaTombstonePurgeInterval
dn: cn=replica,cn=dc\3Dtest1\2Cdc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=conf
 ig
nsds5ReplicaPurgeDelay: 40
nsDS5ReplicaTombstonePurgeInterval: 40

dn: cn=test1m1tom2,cn=replica,cn=dc\3Dtest1\2Cdc\3Dexample\2Cdc\3Dcom,cn=mappi
 ng tree,cn=config




/usr/lib64/dirsrv/slapd-m1/ldif2db.pl -D "cn=directory manager" -w password -s dc=test1,dc=example,dc=com -i /root/m1.dc.test1.dc.example.guest1.del.db2ldif.online.with.r.option.ldif

ldapsearch -LLLxh 10.14.5.27 -D "cn=directory manager" -w password -b "ou=people,dc=test1,dc=example,dc=com" -s one objectclass=* dn | grep "dn: " |  wc -l
9

ldapsearch -LLLxh 10.14.5.27 -D "cn=directory manager" -w password -b "ou=people,dc=test1,dc=example,dc=com" -s base objectclass=*  numsubordinates
dn: ou=people,dc=test1,dc=example,dc=com
numsubordinates: 10



wait more than 1 minute for the purge delay to expire


ldapsearch -LLLxh 10.14.5.27 -D "cn=directory manager" -w password -b "ou=people,dc=test1,dc=example,dc=com" -s one objectclass=* dn | grep "dn: " |  wc -l
9

ldapsearch -LLLxh 10.14.5.27 -D "cn=directory manager" -w password -b "ou=people,dc=test1,dc=example,dc=com" -s base objectclass=*  numsubordinates
dn: ou=people,dc=test1,dc=example,dc=com
numsubordinates: 10



ldapdelete -xh 10.14.5.27 -D "cn=directory manager" -w password ou=people,dc=test1,dc=example,dc=com -v -r
ldap_initialize( ldap://10.14.5.27 )
deleting entry "ou=people,dc=test1,dc=example,dc=com"
deleting children of: ou=people,dc=test1,dc=example,dc=com
deleting children of: uid=guest2,ou=people,dc=test1,dc=example,dc=com
	removing uid=guest2,ou=people,dc=test1,dc=example,dc=com
	uid=guest2,ou=people,dc=test1,dc=example,dc=com removed
deleting children of: uid=guest3,ou=people,dc=test1,dc=example,dc=com
	removing uid=guest3,ou=people,dc=test1,dc=example,dc=com
	uid=guest3,ou=people,dc=test1,dc=example,dc=com removed
deleting children of: uid=guest4,ou=people,dc=test1,dc=example,dc=com
	removing uid=guest4,ou=people,dc=test1,dc=example,dc=com
	uid=guest4,ou=people,dc=test1,dc=example,dc=com removed
deleting children of: uid=guest5,ou=people,dc=test1,dc=example,dc=com
	removing uid=guest5,ou=people,dc=test1,dc=example,dc=com
	uid=guest5,ou=people,dc=test1,dc=example,dc=com removed
deleting children of: uid=guest6,ou=people,dc=test1,dc=example,dc=com
	removing uid=guest6,ou=people,dc=test1,dc=example,dc=com
	uid=guest6,ou=people,dc=test1,dc=example,dc=com removed
deleting children of: uid=guest7,ou=people,dc=test1,dc=example,dc=com
	removing uid=guest7,ou=people,dc=test1,dc=example,dc=com
	uid=guest7,ou=people,dc=test1,dc=example,dc=com removed
deleting children of: uid=guest8,ou=people,dc=test1,dc=example,dc=com
	removing uid=guest8,ou=people,dc=test1,dc=example,dc=com
	uid=guest8,ou=people,dc=test1,dc=example,dc=com removed
deleting children of: uid=guest9,ou=people,dc=test1,dc=example,dc=com
	removing uid=guest9,ou=people,dc=test1,dc=example,dc=com
	uid=guest9,ou=people,dc=test1,dc=example,dc=com removed
deleting children of: uid=guest10,ou=people,dc=test1,dc=example,dc=com
	removing uid=guest10,ou=people,dc=test1,dc=example,dc=com
	uid=guest10,ou=people,dc=test1,dc=example,dc=com removed
ldap_delete: Operation not allowed on non-leaf (66)




ldapsearch -LLLxh 10.14.5.27 -D "cn=directory manager" -w password -b "ou=people,dc=test1,dc=example,dc=com" -s one objectclass=* dn | grep "dn: " |  wc -l
0

ldapsearch -LLLxh 10.14.5.27 -D "cn=directory manager" -w password -b "ou=people,dc=test1,dc=example,dc=com" -s base objectclass=*  numsubordinates
dn: ou=people,dc=test1,dc=example,dc=com
numsubordinates: 1
Comment 28 mreynolds 2014-04-14 17:47:11 UTC 
I've reproduced the issue.

The problem is that during an import, tombstone entries are counted towards numsubordinates.  In Marc's LDIF file there is one tombstone, and we can see after the import that numsubordinates is 1 higher than what it should be:  it's 10 instead of 9.

Fixing this doesn't appear to be trivial, still investigating...
Comment 30 Marc Sauton 2014-04-14 21:23:51 UTC 
patch work against my small test, behavior is different and much better when deleting a container with a tombstone:

ldapdelete -xh 10.14.5.27 -D "cn=directory manager" -w password ou=people,dc=test1,dc=example,dc=com -v -r
ldap_initialize( ldap://10.14.5.27 )
deleting entry "ou=people,dc=test1,dc=example,dc=com"
deleting children of: ou=people,dc=test1,dc=example,dc=com
deleting children of: uid=guest2,ou=people,dc=test1,dc=example,dc=com
        removing uid=guest2,ou=people,dc=test1,dc=example,dc=com
        uid=guest2,ou=people,dc=test1,dc=example,dc=com removed
...
deleting children of: uid=guest10,ou=people,dc=test1,dc=example,dc=com
        removing uid=guest10,ou=people,dc=test1,dc=example,dc=com
        uid=guest10,ou=people,dc=test1,dc=example,dc=com removed
[msauton@testms0 ~]$


before patch:
...
deleting children of: uid=guest10,ou=people,dc=test1,dc=example,dc=com
        removing uid=guest10,ou=people,dc=test1,dc=example,dc=com
        uid=guest10,ou=people,dc=test1,dc=example,dc=com removed
ldap_delete: Operation not allowed on non-leaf (66)



ldapsearch -LLLxh 10.14.5.27 -D "cn=directory manager" -w password -b "ou=people,dc=test1,dc=example,dc=com" -s one objectclass=* dn | grep "dn: " |  wc -l
No such object (32)
Matched DN: dc=test1,dc=example,dc=com
0

before patch:
0


ldapsearch -LLLxh 10.14.5.27 -D "cn=directory manager" -w password -b "ou=people,dc=test1,dc=example,dc=com" -s base objectclass=*  numsubordinates
No such object (32)
Matched DN: dc=test1,dc=example,dc=com

!!! works, was before patch !!!
dn: ou=people,dc=test1,dc=example,dc=com
numsubordinates: 1
Comment 31 Marc Sauton 2014-04-15 22:05:33 UTC 
re-indexing id2entry and the last hotfix did not help the customer, should we suggest to reindex nusubordinates ? other idx?
Comment 32 Marc Sauton 2014-04-15 22:20:16 UTC 
or should we suggest to re-import after the fix is installed, before the customer redo the test