Bug 848925 - Duplicate objects in directory with identical DNs
Duplicate objects in directory with identical DNs
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: 389-ds-base (Show other bugs)
7.0
Unspecified Unspecified
high Severity unspecified
: rc
: ---
Assigned To: Rich Megginson
Sankar Ramalingam
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-08-16 16:27 EDT by Jenny Galipeau
Modified: 2014-06-17 22:54 EDT (History)
9 users (show)

See Also:
Fixed In Version: 389-ds-base-1.3.1.2-1.el7
Doc Type: Bug Fix
Doc Text:
Cause: The directory server runs some operations as triggered by update operations. For example, updating memberof attributes in user entries is triggered by adding a member to a group entry. This updating is done after the original operation. In older versions of directory server, this updating was done outside the original transaction and potentially after sending the result to the client. Consequence: The client would assume all of the triggered operations were complete after the server returned the response. The client would then expect to see data that wasn't actually there. Fix: The directory server has been modified so that all of the triggered operations are done inside the same transaction as the original request. All of the triggered operations are complete before the response is sent to the client. Result: The client expects and sees a consistent view of the data in the directory server.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-06-13 06:42:57 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
stack trace from send_ldap_search_entry_ext (119.31 KB, text/plain)
2012-08-16 17:22 EDT, Rich Megginson
no flags Details

  None (edit)
Description Jenny Galipeau 2012-08-16 16:27:59 EDT
Description of problem:

This has happened twice now, but is not readily reproducible.  It is possible to end up with duplicate objects in the directory with identical DNs when adding a sudo rule within IPA.  It was produced once with the CLIs and then again with the  webUI.

Duplicate entries ::

# ldapsearch -x -D "cn=Directory Manager" -w mysecret -b "dc=testrelm,dc=com" | grep AlowRule
# AlowRule, sudoers, testrelm.com
dn: cn=AlowRule,ou=sudoers,dc=testrelm,dc=com
cn: AlowRule
# AlowRule, sudoers, testrelm.com
dn: cn=AlowRule,ou=sudoers,dc=testrelm,dc=com
cn: AlowRule

Version-Release number of selected component (if applicable):

ipa-server-2.2.0-16.el6.x86_64
389-ds-base-1.2.10.2-19.el6_3.x86_64


How reproducible:
intermittent

Steps to Reproduce:
1. aquire admin credentials
2. ipa sudorule-add AlowRule
3. ipa sudorule-add AlowRule
  
Actual results:


Expected results:


Additional info:

ldapsearch -LLL -x -D "cn=Directory Manager" -w mysecret -b
"dc=testrelm,dc=com" cn=AlowRule entryid nsuniqueid

dn: cn=AlowRule,ou=sudoers,dc=testrelm,dc=com
cn: AlowRule

dn: cn=AlowRule,ou=sudoers,dc=testrelm,dc=com
cn: AlowRule

dn:
ipaUniqueID=d4de6ea8-e7d3-11e1-ad68-782bcb785283,cn=sudorules,cn=sudo,dc=t
  estrelm,dc=com
entryid: 20269
nsuniqueid: c8c07f81-e7d311e1-ada6e619-e3bd77c0
cn: AlowRule

dn:
ipaUniqueID=d5db1356-e7d3-11e1-9d92-782bcb785283,cn=sudorules,cn=sudo,dc=t
  estrelm,dc=com
entryid: 20270
nsuniqueid: c8c07f82-e7d311e1-ada6e619-e3bd77c0
cn: AlowRule
Comment 1 Rich Megginson 2012-08-16 16:40:05 EDT
I've dumped the id2entry and cn indexes - no duplicate there - the only two real entries are the ipauniqueid DNs.  There is no AlowRule in the entryrdn index either.

I've inspected the running process in gdb and dumped the dncache and the idcache - I don't see the duplicate DNs there

Next step is to trace the search request in gdb.
Comment 2 Rich Megginson 2012-08-16 16:54:34 EDT
Tracing search output - id candidate list contains only the two ipauniqueid DN entries . . .
Comment 3 Rich Megginson 2012-08-16 17:22:00 EDT
Created attachment 605026 [details]
stack trace from send_ldap_search_entry_ext

steps
debuginfo-install 389-ds-base slapi-nis
this doesn't always find the right packages - so go to 
http://download.devel.redhat.com/brewroot/packages/389-ds-base/
and
http://download.devel.redhat.com/brewroot/packages/slapi-nis

to manually install the debuginfo packages e.g.
yum install http://download.devel.redhat.com/brewroot/packages/389-ds-base/.....389-ds-base-debuginfo.... http://download.devel.redhat.com/brewroot/packages/slapi-nis/.../slapi-nis-debuginfo....

then in gdb - set a break point at do_search line 224
in another window, do
ldapsearch -LLL -x -D "cn=Directory Manager" -w Secret123 -b "dc=testrelm,dc=com" cn=AlowRule

in the debugger, print fstr whenever the breakpoint is hit - wait until it is "cn=AlowRule"

next, find the current thread id
p $_thread

next, set a breakpoint in send_ldap_search_entry_ext

b send_ldap_search_entry_ext if $_thread == thethreadid from above
Comment 4 Nalin Dahyabhai 2012-08-16 17:41:21 EDT
Hmm, it looks like we're using a non-unique attribute to name the compat entry that corresponds to the IPA sudo rule.  The compat plugin doesn't key off of the DNs internally, so it isn't thrown by of the input data, as a result we get two entries in the search results which have the same DN, though other attributes could well be different.

A. Is this something that IPA allows intentionally?
B. Is it desired that the compat plugin drop one or the other entry, or attempt to merge them somehow, or check if that happens, log it, and carry on?
Comment 5 Rob Crittenden 2012-08-16 17:42:47 EDT
IPA should not allow this to happen.
Comment 6 Rich Megginson 2012-08-16 18:00:22 EDT
(03:40:41 PM) richm: so add ipauniqueid=1 which has cn: AlowRule - then search for cn=AlowRule
(03:40:57 PM) richm: Then add ipauniqueid=2 also with cn: AlowRule - then search for cn=AlowRule
(03:42:52 PM) richm: is there anything in ipa to prevent the second add ipauniqueid=2 with cn: AlowRule?
(03:43:30 PM) richm: I mean, does sudorule_add first search to see if a rule with that name is already there, then try to add it?
(03:43:57 PM) richm: If not, then it seems that the schemacompat is responsible for ensuring the uniqueness of cn's 
(03:44:27 PM) richm: or, does sudorule_add expect to receive an err=68 AlreadyExists if a rule with that name already exists?
(03:45:31 PM) richm: if so, then you'll either have to enable the attribute-uniqueness plugin for 'cn' under cn=sudorules,cn=sudo,DOMAIN, or schemacompat will have to do that
Comment 7 Rob Crittenden 2012-08-17 08:41:50 EDT
I'd like to understand how these were added. Were the adds being done on a single machine or multiple and relying on replication to keep things in sync?

I wonder if it is possible that our check failed because DS was so busy with memberof the search timed out and we took that as "nope, entry isn't there".

I think that uniqueness is the way to go in any case.
Comment 8 baiesi 2012-08-17 09:48:32 EDT
The first time i saw this issue, I had run 2 python ipa cli threads on one Ipa client, adding the same sudorule name via command line by accident....

The second time i saw this issue, I added 2 sudorules with the same name within 1 minute of each other via the UI on purpose.

In both cases the setup consits of;
2 IPA Servers
2 IPA Clients
Comment 9 Jenny Galipeau 2012-08-17 10:30:20 EDT
(In reply to comment #8)
> The first time i saw this issue, I had run 2 python ipa cli threads on one
> Ipa client, adding the same sudorule name via command line by accident....
> 
> The second time i saw this issue, I added 2 sudorules with the same name
> within 1 minute of each other via the UI on purpose.
> 
> In both cases the setup consits of;
> 2 IPA Servers
> 2 IPA Clients

also there are 10,000 users, not sure how many groups and group members in each ... so entirely possible that the directory server was busy with post operations and that thought had crossed my mind too.
Comment 10 baiesi 2012-08-20 15:03:04 EDT
There were 10 User groups, 1K members assigned in each group, for the total of 10K ipa users...
Comment 11 Dmitri Pal 2012-08-21 10:35:37 EDT
Upstream ticket:
https://fedorahosted.org/freeipa/ticket/3017
Comment 12 baiesi 2012-08-27 11:28:03 EDT
Again it reproduced itself once more adding sudo rule "test" via the UI by accident in the same test env listed above while sudo runtime load was being applied.
Comment 13 Rob Crittenden 2012-10-16 09:41:02 EDT
Fixed upstream.

master: 96decfea26a1ca977aa81456e46e0590dde1f861

ipa-3-0: b89367a23000592915bd0ffc68b24c0331f45694
Comment 15 Jenny Galipeau 2012-11-26 08:57:15 EST
Please add steps to verify. Thanks
Comment 16 Rob Crittenden 2012-11-26 10:00:31 EST
I have no steps other than are in the bug report itself. This was caused by a race condition of two adds at the same time on a loaded server. 

The majority entries in IPA rely on unique attributes in the DN to prevent duplication (e.g. uid for user). In the case of SUDO we use a UUID in the DN which will allow the entry itself to be unique but doesn't guarantee unique CN values (CN being the name of the rule itself). So for LDAP it is perfectly legal to have two unique entries with the same CN, for IPA not so much.

Our fix is to add a uniqueness rule for the sudo CN attribute for sudo rules covering entries in cn=sudorules,cn=sudo,$SUFFIX.

I guess you could verify that the uniqueness rule is added.
Comment 17 baiesi 2012-11-26 13:36:06 EST
While trying to verify this defect, seems  I was able to reproduce this issue. I now have 2 sudorules with the same name. 2 different ipaUniqueIds with the same cn "dup_sudo_rule_test"

Steps
1. I created and ran a simple single threaded python script to cycle to create a sudo rule named "dup_sudo_rule_test"
2. I ran the script again as a second process concurrent to the first process in step 1 above.
3. All seemed ok for a while with no additional admin load.
4. Until I asked the system to perform some additional admin load such as deleting 2 sudo rules in the UI while steps 1 and 2 continued.

Result:
I now have 2 sudorules with the same name. 2 different ipaUniqueIds with the same cn "dup_sudo_rule_test"

CLI error indicates:
98 Time 12:33:30.872778
Add Rule dup_sudo_rule_test
Command:ipa sudorule-add dup_sudo_rule_test
ipa: ERROR: The search criteria was not specific enough. Expected 1 and found 2.

UI error indicates:
An error has occurred (IPA Error 4027)
The search criteria was not specific enough. Expected 1 and found 2.
Please try the following options:
    Refresh the page.
    Return to the main page and retry the operation
    Reload the browser.
If the problem persists please contact the system administrator.

Setup:
Build Installed 11/16/2012
Red Hat Enterprise Linux Server release 6.4 Beta (Santiago)
IPA server version 3.0.1. API version 2.46
2 Ipa Servers
2 Ipa Clients

Pkgs installed
libipa_hbac-1.9.90-0.20121114T2317zgit3a97c85.el6.x86_64
ipa-client-3.0.0-107.20121114T2335zgit863e15c.el6.x86_64
ipa-python-3.0.0-107.20121114T2335zgit863e15c.el6.x86_64
ipa-server-selinux-3.0.0-107.20121114T2335zgit863e15c.el6.x86_64
ipa-pki-ca-theme-9.0.3-7.20121114T2248z.el6.noarch
libipa_hbac-python-1.9.90-0.20121114T2317zgit3a97c85.el6.x86_64
ipa-server-3.0.0-107.20121114T2335zgit863e15c.el6.x86_64
ipa-admintools-3.0.0-107.20121114T2335zgit863e15c.el6.x86_64
ipa-pki-common-theme-9.0.3-7.20121114T2248z.el6.noarch
sssd-1.9.90-0.20121114T2317zgit3a97c85.el6.x86_64
sssd-client-1.9.90-0.20121114T2317zgit3a97c85.el6.x86_64
Comment 18 Rob Crittenden 2012-11-26 14:00:45 EST
That version *may* have contained the fix for this, probably best to try to reproduce again using 6.4 candidate build.

Can you see if the uniqueness entry exists:

$ ldapsearch -x -D 'cn=directory manager' -W -b 'cn=sudorule name uniqueness,cn=plugins,cn=config'
Comment 19 baiesi 2012-11-26 16:18:57 EST
Sure;
[root@sti-high-1 ~]# ldapsearch -x -D 'cn=directory manager' -W -b 'cn=sudorule name uniqueness,cn=plugins,cn=config'
Enter LDAP Password: 
# extended LDIF
#
# LDAPv3
# base <cn=sudorule name uniqueness,cn=plugins,cn=config> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# sudorule name uniqueness, plugins, config
dn: cn=sudorule name uniqueness,cn=plugins,cn=config
objectClass: top
objectClass: nsSlapdPlugin
objectClass: extensibleObject
cn: sudorule name uniqueness
nsslapd-pluginDescription: Enforce unique attribute values
nsslapd-pluginPath: libattr-unique-plugin
nsslapd-pluginInitfunc: NSUniqueAttr_Init
nsslapd-pluginType: preoperation
nsslapd-pluginEnabled: on
nsslapd-pluginarg0: cn
nsslapd-pluginarg1: cn=sudorules,cn=sudo,dc=testrelm,dc=com
nsslapd-plugin-depends-on-type: database
nsslapd-pluginId: NSUniqueAttr
nsslapd-pluginVersion: 1.2.11.15
nsslapd-pluginVendor: 389 Project

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

Does this mean it has the fix?
Comment 20 Rob Crittenden 2012-11-26 16:41:22 EST
Yes, this was the proposed fix, so no need to re-test.

Can you also show the two entries that were created: 

# kinit admin
# ldapsearch -Y GSSAPI -b cn=sudorules,cn=sudo,dc=testrelm,dc=com cn=dup_sudo_rule_test

I'd have thought the uniqueness plugin would prevent this.
Comment 21 baiesi 2012-11-26 17:26:54 EST
Sure:
[root@sti-high-1 ~]# ldapsearch -Y GSSAPI -b cn=sudorules,cn=sudo,dc=testrelm,dc=com cn=dup_sudo_rule_test
SASL/GSSAPI authentication started
SASL username: admin@TESTRELM.COM
SASL SSF: 56
SASL data security layer installed.
# extended LDIF
#
# LDAPv3
# base <cn=sudorules,cn=sudo,dc=testrelm,dc=com> with scope subtree
# filter: cn=dup_sudo_rule_test
# requesting: ALL
#

# 192717ca-37ef-11e2-9aa4-782bcb785283, sudorules, sudo, testrelm.com
dn: ipaUniqueID=192717ca-37ef-11e2-9aa4-782bcb785283,cn=sudorules,cn=sudo,dc=t
 estrelm,dc=com
objectClass: ipaassociation
objectClass: ipasudorule
cn: dup_sudo_rule_test
ipaEnabledFlag: TRUE
ipaUniqueID: 192717ca-37ef-11e2-9aa4-782bcb785283

# 1926b2b2-37ef-11e2-9482-782bcb785283, sudorules, sudo, testrelm.com
dn: ipaUniqueID=1926b2b2-37ef-11e2-9482-782bcb785283,cn=sudorules,cn=sudo,dc=t
 estrelm,dc=com
objectClass: ipaassociation
objectClass: ipasudorule
cn: dup_sudo_rule_test
ipaEnabledFlag: TRUE
ipaUniqueID: 1926b2b2-37ef-11e2-9482-782bcb785283

# search result
search: 4
result: 0 Success

# numResponses: 3
# numEntries: 2
Comment 22 Rob Crittenden 2012-11-27 13:41:36 EST
Rich, is adding the uniqueness configuration not adequate? Do we perhaps need to do something else?
Comment 23 Rich Megginson 2012-11-27 13:59:17 EST
(In reply to comment #22)
> Rich, is adding the uniqueness configuration not adequate? Do we perhaps
> need to do something else?

It should be adequate.  Is this with the version of 389-ds-base with all of the plugins enabled to be betxn plugins?  If not, then there is a race condition - if you time the add operations just right, you will be able to add two entries that have the same value.
Comment 24 Rob Crittenden 2012-11-27 14:20:24 EST
This is 1.2.x in 6.4. I can't be sure of the version, maybe Bruce can comment, but I'd imagine that transactions are not enabled.
Comment 25 Rich Megginson 2012-11-27 14:31:28 EST
(In reply to comment #24)
> This is 1.2.x in 6.4. I can't be sure of the version, maybe Bruce can
> comment, but I'd imagine that transactions are not enabled.

If this is 1.2.x in 6.4 then betxn are not being used.  Therefore, there are certain conditions under which it is possible to defeat the attribute uniqueness checking.  This will be fixed in 389-ds-base 1.3.0
Comment 26 Jenny Galipeau 2012-11-27 14:36:38 EST
I suggest we change the component to 389-ds-base and target RHEL 7.0.  It will also need to be removed from the RHEL 6.4 ipa errata.
Comment 27 Rob Crittenden 2012-11-29 11:19:04 EST
Re-assigning to 389-ds-base.
Comment 28 Nathan Kinder 2013-03-04 18:02:07 EST
This issue should be solved by the addition of betxn plug-in support in 389-ds-base.  We will want to explicitly verify this issue, so we should leave this bug separate from the main betxn feature bug.
Comment 29 Rich Megginson 2013-10-01 19:26:36 EDT
moving all ON_QA bugs to MODIFIED in order to add them to the errata (can't add bugs in the ON_QA state to an errata).  When the errata is created, the bugs should be automatically moved back to ON_QA.
Comment 30 Rich Megginson 2013-11-07 08:58:28 EST
It is possible that you could reproduce this issue with only 389-ds-base.  But you would have to configure it exactly how it is configured when using ipa, so it's probably easier to verify using ipa.
Comment 31 Steeve Goveas 2013-11-08 06:15:22 EST
Verified in version

[root@hp-z600-01 ~]# rpm -q 389-ds-base ipa-server
389-ds-base-1.3.1.6-8.el7.x86_64
ipa-server-3.3.3-3.el7.x86_64

[root@hp-z600-01 ~]# kinit admin
Password for admin@TESTRELM.COM:

[root@hp-z600-01 ~]# ipa sudorule-add AlowRule
--------------------------
Added Sudo Rule "AlowRule"
--------------------------
  Rule name: AlowRule
  Enabled: TRUE

>>[root@hp-z600-01 ~]# ipa sudorule-add AlowRule
>>ipa: ERROR: sudo rule with name "AlowRule" already exists
[root@hp-z600-01 ~]# ipa sudorule-add AlowRule
ipa: ERROR: sudo rule with name "AlowRule" already exists
[root@hp-z600-01 ~]# ipa sudorule-add AlowRule
ipa: ERROR: sudo rule with name "AlowRule" already exists

[root@hp-z600-01 ~]# ldapsearch -x -D "cn=Directory Manager" -w Secret123 -b "dc=testrelm,dc=com" | grep -B5 AlowRule
# sudoers, testrelm.com
dn: ou=sudoers,dc=testrelm,dc=com
objectClass: extensibleObject
ou: sudoers

# AlowRule, sudoers, testrelm.com
dn: cn=AlowRule,ou=sudoers,dc=testrelm,dc=com
objectClass: sudoRole
cn: AlowRule
--
# bc04b196-485b-11e3-bebb-0024810e5d52, sudorules, sudo, testrelm.com
dn: ipaUniqueID=bc04b196-485b-11e3-bebb-0024810e5d52,cn=sudorules,cn=sudo,dc=t
 estrelm,dc=com
objectClass: ipaassociation
objectClass: ipasudorule
cn: AlowRule

[root@hp-z600-01 ~]# ldapsearch -LLL -x -D "cn=Directory Manager" -w Secret123 -b "dc=testrelm,dc=com" cn=AlowRule entryid nsuniqueid
dn: cn=AlowRule,ou=sudoers,dc=testrelm,dc=com

dn: ipaUniqueID=bc04b196-485b-11e3-bebb-0024810e5d52,cn=sudorules,cn=sudo,dc=t
 estrelm,dc=com
entryid: 285
nsuniqueid: 9a591508-485b11e3-b8a0a85b-4d2cb512


[root@hp-z600-01 ~]# ldapsearch -x -D 'cn=directory manager' -w Secret123 -b 'cn=sudorule name uniqueness,cn=plugins,cn=config'
# extended LDIF
#
# LDAPv3
# base <cn=sudorule name uniqueness,cn=plugins,cn=config> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# sudorule name uniqueness, plugins, config
dn: cn=sudorule name uniqueness,cn=plugins,cn=config
objectClass: top
objectClass: nsSlapdPlugin
objectClass: extensibleObject
cn: sudorule name uniqueness
nsslapd-pluginDescription: Enforce unique attribute values
nsslapd-pluginPath: libattr-unique-plugin
nsslapd-pluginInitfunc: NSUniqueAttr_Init
nsslapd-pluginType: preoperation
nsslapd-pluginEnabled: on
nsslapd-pluginarg0: cn
nsslapd-pluginarg1: cn=sudorules,cn=sudo,dc=testrelm,dc=com
nsslapd-plugin-depends-on-type: database
nsslapd-pluginId: NSUniqueAttr
nsslapd-pluginVersion: 1.3.1.6
nsslapd-pluginVendor: 389 Project

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

[root@hp-z600-01 ~]# ldapsearch -Y GSSAPI -b cn=sudorules,cn=sudo,dc=testrelm,dc=com cn=AlowRule
SASL/GSSAPI authentication started
SASL username: admin@TESTRELM.COM
SASL SSF: 56
SASL data security layer installed.
# extended LDIF
#
# LDAPv3
# base <cn=sudorules,cn=sudo,dc=testrelm,dc=com> with scope subtree
# filter: cn=AlowRule
# requesting: ALL
#

# bc04b196-485b-11e3-bebb-0024810e5d52, sudorules, sudo, testrelm.com
dn: ipaUniqueID=bc04b196-485b-11e3-bebb-0024810e5d52,cn=sudorules,cn=sudo,dc=t
 estrelm,dc=com
objectClass: ipaassociation
objectClass: ipasudorule
cn: AlowRule
ipaEnabledFlag: TRUE
ipaUniqueID: bc04b196-485b-11e3-bebb-0024810e5d52

# search result
search: 4
result: 0 Success

# numResponses: 2
# numEntries: 1
Comment 32 Ludek Smid 2014-06-13 06:42:57 EDT
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.

Note You need to log in before you can comment on or make changes to this bug.