I've dumped the id2entry and cn indexes - no duplicate there - the only two real entries are the ipauniqueid DNs.  There is no AlowRule in the entryrdn index either.

I've inspected the running process in gdb and dumped the dncache and the idcache - I don't see the duplicate DNs there

Next step is to trace the search request in gdb.

Tracing search output - id candidate list contains only the two ipauniqueid DN entries . . .

Created attachment 605026 [details]
stack trace from send_ldap_search_entry_ext

steps
debuginfo-install 389-ds-base slapi-nis
this doesn't always find the right packages - so go to 
http://download.devel.redhat.com/brewroot/packages/389-ds-base/
and
http://download.devel.redhat.com/brewroot/packages/slapi-nis

to manually install the debuginfo packages e.g.
yum install http://download.devel.redhat.com/brewroot/packages/389-ds-base/.....389-ds-base-debuginfo.... http://download.devel.redhat.com/brewroot/packages/slapi-nis/.../slapi-nis-debuginfo....

then in gdb - set a break point at do_search line 224
in another window, do
ldapsearch -LLL -x -D "cn=Directory Manager" -w Secret123 -b "dc=testrelm,dc=com" cn=AlowRule

in the debugger, print fstr whenever the breakpoint is hit - wait until it is "cn=AlowRule"

next, find the current thread id
p $_thread

next, set a breakpoint in send_ldap_search_entry_ext

b send_ldap_search_entry_ext if $_thread == thethreadid from above

Hmm, it looks like we're using a non-unique attribute to name the compat entry that corresponds to the IPA sudo rule.  The compat plugin doesn't key off of the DNs internally, so it isn't thrown by of the input data, as a result we get two entries in the search results which have the same DN, though other attributes could well be different.

A. Is this something that IPA allows intentionally?
B. Is it desired that the compat plugin drop one or the other entry, or attempt to merge them somehow, or check if that happens, log it, and carry on?

IPA should not allow this to happen.

(03:40:41 PM) richm: so add ipauniqueid=1 which has cn: AlowRule - then search for cn=AlowRule
(03:40:57 PM) richm: Then add ipauniqueid=2 also with cn: AlowRule - then search for cn=AlowRule
(03:42:52 PM) richm: is there anything in ipa to prevent the second add ipauniqueid=2 with cn: AlowRule?
(03:43:30 PM) richm: I mean, does sudorule_add first search to see if a rule with that name is already there, then try to add it?
(03:43:57 PM) richm: If not, then it seems that the schemacompat is responsible for ensuring the uniqueness of cn's 
(03:44:27 PM) richm: or, does sudorule_add expect to receive an err=68 AlreadyExists if a rule with that name already exists?
(03:45:31 PM) richm: if so, then you'll either have to enable the attribute-uniqueness plugin for 'cn' under cn=sudorules,cn=sudo,DOMAIN, or schemacompat will have to do that

I'd like to understand how these were added. Were the adds being done on a single machine or multiple and relying on replication to keep things in sync?

I wonder if it is possible that our check failed because DS was so busy with memberof the search timed out and we took that as "nope, entry isn't there".

I think that uniqueness is the way to go in any case.

The first time i saw this issue, I had run 2 python ipa cli threads on one Ipa client, adding the same sudorule name via command line by accident....

The second time i saw this issue, I added 2 sudorules with the same name within 1 minute of each other via the UI on purpose.

In both cases the setup consits of;
2 IPA Servers
2 IPA Clients

(In reply to comment #8)
> The first time i saw this issue, I had run 2 python ipa cli threads on one
> Ipa client, adding the same sudorule name via command line by accident....
> 
> The second time i saw this issue, I added 2 sudorules with the same name
> within 1 minute of each other via the UI on purpose.
> 
> In both cases the setup consits of;
> 2 IPA Servers
> 2 IPA Clients

also there are 10,000 users, not sure how many groups and group members in each ... so entirely possible that the directory server was busy with post operations and that thought had crossed my mind too.

There were 10 User groups, 1K members assigned in each group, for the total of 10K ipa users...

Upstream ticket:
https://fedorahosted.org/freeipa/ticket/3017

Again it reproduced itself once more adding sudo rule "test" via the UI by accident in the same test env listed above while sudo runtime load was being applied.

Fixed upstream.

master: 96decfea26a1ca977aa81456e46e0590dde1f861

ipa-3-0: b89367a23000592915bd0ffc68b24c0331f45694

Please add steps to verify. Thanks

I have no steps other than are in the bug report itself. This was caused by a race condition of two adds at the same time on a loaded server. 

The majority entries in IPA rely on unique attributes in the DN to prevent duplication (e.g. uid for user). In the case of SUDO we use a UUID in the DN which will allow the entry itself to be unique but doesn't guarantee unique CN values (CN being the name of the rule itself). So for LDAP it is perfectly legal to have two unique entries with the same CN, for IPA not so much.

Our fix is to add a uniqueness rule for the sudo CN attribute for sudo rules covering entries in cn=sudorules,cn=sudo,$SUFFIX.

I guess you could verify that the uniqueness rule is added.

While trying to verify this defect, seems  I was able to reproduce this issue. I now have 2 sudorules with the same name. 2 different ipaUniqueIds with the same cn "dup_sudo_rule_test"

Steps
1. I created and ran a simple single threaded python script to cycle to create a sudo rule named "dup_sudo_rule_test"
2. I ran the script again as a second process concurrent to the first process in step 1 above.
3. All seemed ok for a while with no additional admin load.
4. Until I asked the system to perform some additional admin load such as deleting 2 sudo rules in the UI while steps 1 and 2 continued.

Result:
I now have 2 sudorules with the same name. 2 different ipaUniqueIds with the same cn "dup_sudo_rule_test"

CLI error indicates:
98 Time 12:33:30.872778
Add Rule dup_sudo_rule_test
Command:ipa sudorule-add dup_sudo_rule_test
ipa: ERROR: The search criteria was not specific enough. Expected 1 and found 2.

UI error indicates:
An error has occurred (IPA Error 4027)
The search criteria was not specific enough. Expected 1 and found 2.
Please try the following options:
    Refresh the page.
    Return to the main page and retry the operation
    Reload the browser.
If the problem persists please contact the system administrator.

Setup:
Build Installed 11/16/2012
Red Hat Enterprise Linux Server release 6.4 Beta (Santiago)
IPA server version 3.0.1. API version 2.46
2 Ipa Servers
2 Ipa Clients

Pkgs installed
libipa_hbac-1.9.90-0.20121114T2317zgit3a97c85.el6.x86_64
ipa-client-3.0.0-107.20121114T2335zgit863e15c.el6.x86_64
ipa-python-3.0.0-107.20121114T2335zgit863e15c.el6.x86_64
ipa-server-selinux-3.0.0-107.20121114T2335zgit863e15c.el6.x86_64
ipa-pki-ca-theme-9.0.3-7.20121114T2248z.el6.noarch
libipa_hbac-python-1.9.90-0.20121114T2317zgit3a97c85.el6.x86_64
ipa-server-3.0.0-107.20121114T2335zgit863e15c.el6.x86_64
ipa-admintools-3.0.0-107.20121114T2335zgit863e15c.el6.x86_64
ipa-pki-common-theme-9.0.3-7.20121114T2248z.el6.noarch
sssd-1.9.90-0.20121114T2317zgit3a97c85.el6.x86_64
sssd-client-1.9.90-0.20121114T2317zgit3a97c85.el6.x86_64

That version *may* have contained the fix for this, probably best to try to reproduce again using 6.4 candidate build.

Can you see if the uniqueness entry exists:

$ ldapsearch -x -D 'cn=directory manager' -W -b 'cn=sudorule name uniqueness,cn=plugins,cn=config'

Sure;
[root@sti-high-1 ~]# ldapsearch -x -D 'cn=directory manager' -W -b 'cn=sudorule name uniqueness,cn=plugins,cn=config'
Enter LDAP Password: 
# extended LDIF
#
# LDAPv3
# base <cn=sudorule name uniqueness,cn=plugins,cn=config> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# sudorule name uniqueness, plugins, config
dn: cn=sudorule name uniqueness,cn=plugins,cn=config
objectClass: top
objectClass: nsSlapdPlugin
objectClass: extensibleObject
cn: sudorule name uniqueness
nsslapd-pluginDescription: Enforce unique attribute values
nsslapd-pluginPath: libattr-unique-plugin
nsslapd-pluginInitfunc: NSUniqueAttr_Init
nsslapd-pluginType: preoperation
nsslapd-pluginEnabled: on
nsslapd-pluginarg0: cn
nsslapd-pluginarg1: cn=sudorules,cn=sudo,dc=testrelm,dc=com
nsslapd-plugin-depends-on-type: database
nsslapd-pluginId: NSUniqueAttr
nsslapd-pluginVersion: 1.2.11.15
nsslapd-pluginVendor: 389 Project

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

Does this mean it has the fix?

Yes, this was the proposed fix, so no need to re-test.

Can you also show the two entries that were created: 

# kinit admin
# ldapsearch -Y GSSAPI -b cn=sudorules,cn=sudo,dc=testrelm,dc=com cn=dup_sudo_rule_test

I'd have thought the uniqueness plugin would prevent this.

Sure:
[root@sti-high-1 ~]# ldapsearch -Y GSSAPI -b cn=sudorules,cn=sudo,dc=testrelm,dc=com cn=dup_sudo_rule_test
SASL/GSSAPI authentication started
SASL username: admin@TESTRELM.COM
SASL SSF: 56
SASL data security layer installed.
# extended LDIF
#
# LDAPv3
# base <cn=sudorules,cn=sudo,dc=testrelm,dc=com> with scope subtree
# filter: cn=dup_sudo_rule_test
# requesting: ALL
#

# 192717ca-37ef-11e2-9aa4-782bcb785283, sudorules, sudo, testrelm.com
dn: ipaUniqueID=192717ca-37ef-11e2-9aa4-782bcb785283,cn=sudorules,cn=sudo,dc=t
 estrelm,dc=com
objectClass: ipaassociation
objectClass: ipasudorule
cn: dup_sudo_rule_test
ipaEnabledFlag: TRUE
ipaUniqueID: 192717ca-37ef-11e2-9aa4-782bcb785283

# 1926b2b2-37ef-11e2-9482-782bcb785283, sudorules, sudo, testrelm.com
dn: ipaUniqueID=1926b2b2-37ef-11e2-9482-782bcb785283,cn=sudorules,cn=sudo,dc=t
 estrelm,dc=com
objectClass: ipaassociation
objectClass: ipasudorule
cn: dup_sudo_rule_test
ipaEnabledFlag: TRUE
ipaUniqueID: 1926b2b2-37ef-11e2-9482-782bcb785283

# search result
search: 4
result: 0 Success

# numResponses: 3
# numEntries: 2

Rich, is adding the uniqueness configuration not adequate? Do we perhaps need to do something else?

(In reply to comment #22)
> Rich, is adding the uniqueness configuration not adequate? Do we perhaps
> need to do something else?

It should be adequate.  Is this with the version of 389-ds-base with all of the plugins enabled to be betxn plugins?  If not, then there is a race condition - if you time the add operations just right, you will be able to add two entries that have the same value.

This is 1.2.x in 6.4. I can't be sure of the version, maybe Bruce can comment, but I'd imagine that transactions are not enabled.

(In reply to comment #24)
> This is 1.2.x in 6.4. I can't be sure of the version, maybe Bruce can
> comment, but I'd imagine that transactions are not enabled.

If this is 1.2.x in 6.4 then betxn are not being used.  Therefore, there are certain conditions under which it is possible to defeat the attribute uniqueness checking.  This will be fixed in 389-ds-base 1.3.0

I suggest we change the component to 389-ds-base and target RHEL 7.0.  It will also need to be removed from the RHEL 6.4 ipa errata.

Re-assigning to 389-ds-base.

This issue should be solved by the addition of betxn plug-in support in 389-ds-base.  We will want to explicitly verify this issue, so we should leave this bug separate from the main betxn feature bug.

moving all ON_QA bugs to MODIFIED in order to add them to the errata (can't add bugs in the ON_QA state to an errata).  When the errata is created, the bugs should be automatically moved back to ON_QA.

It is possible that you could reproduce this issue with only 389-ds-base.  But you would have to configure it exactly how it is configured when using ipa, so it's probably easier to verify using ipa.

Verified in version

[root@hp-z600-01 ~]# rpm -q 389-ds-base ipa-server
389-ds-base-1.3.1.6-8.el7.x86_64
ipa-server-3.3.3-3.el7.x86_64

[root@hp-z600-01 ~]# kinit admin
Password for admin@TESTRELM.COM:

[root@hp-z600-01 ~]# ipa sudorule-add AlowRule
--------------------------
Added Sudo Rule "AlowRule"
--------------------------
  Rule name: AlowRule
  Enabled: TRUE

>>[root@hp-z600-01 ~]# ipa sudorule-add AlowRule
>>ipa: ERROR: sudo rule with name "AlowRule" already exists
[root@hp-z600-01 ~]# ipa sudorule-add AlowRule
ipa: ERROR: sudo rule with name "AlowRule" already exists
[root@hp-z600-01 ~]# ipa sudorule-add AlowRule
ipa: ERROR: sudo rule with name "AlowRule" already exists

[root@hp-z600-01 ~]# ldapsearch -x -D "cn=Directory Manager" -w Secret123 -b "dc=testrelm,dc=com" | grep -B5 AlowRule
# sudoers, testrelm.com
dn: ou=sudoers,dc=testrelm,dc=com
objectClass: extensibleObject
ou: sudoers

# AlowRule, sudoers, testrelm.com
dn: cn=AlowRule,ou=sudoers,dc=testrelm,dc=com
objectClass: sudoRole
cn: AlowRule
--
# bc04b196-485b-11e3-bebb-0024810e5d52, sudorules, sudo, testrelm.com
dn: ipaUniqueID=bc04b196-485b-11e3-bebb-0024810e5d52,cn=sudorules,cn=sudo,dc=t
 estrelm,dc=com
objectClass: ipaassociation
objectClass: ipasudorule
cn: AlowRule

[root@hp-z600-01 ~]# ldapsearch -LLL -x -D "cn=Directory Manager" -w Secret123 -b "dc=testrelm,dc=com" cn=AlowRule entryid nsuniqueid
dn: cn=AlowRule,ou=sudoers,dc=testrelm,dc=com

dn: ipaUniqueID=bc04b196-485b-11e3-bebb-0024810e5d52,cn=sudorules,cn=sudo,dc=t
 estrelm,dc=com
entryid: 285
nsuniqueid: 9a591508-485b11e3-b8a0a85b-4d2cb512


[root@hp-z600-01 ~]# ldapsearch -x -D 'cn=directory manager' -w Secret123 -b 'cn=sudorule name uniqueness,cn=plugins,cn=config'
# extended LDIF
#
# LDAPv3
# base <cn=sudorule name uniqueness,cn=plugins,cn=config> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# sudorule name uniqueness, plugins, config
dn: cn=sudorule name uniqueness,cn=plugins,cn=config
objectClass: top
objectClass: nsSlapdPlugin
objectClass: extensibleObject
cn: sudorule name uniqueness
nsslapd-pluginDescription: Enforce unique attribute values
nsslapd-pluginPath: libattr-unique-plugin
nsslapd-pluginInitfunc: NSUniqueAttr_Init
nsslapd-pluginType: preoperation
nsslapd-pluginEnabled: on
nsslapd-pluginarg0: cn
nsslapd-pluginarg1: cn=sudorules,cn=sudo,dc=testrelm,dc=com
nsslapd-plugin-depends-on-type: database
nsslapd-pluginId: NSUniqueAttr
nsslapd-pluginVersion: 1.3.1.6
nsslapd-pluginVendor: 389 Project

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

[root@hp-z600-01 ~]# ldapsearch -Y GSSAPI -b cn=sudorules,cn=sudo,dc=testrelm,dc=com cn=AlowRule
SASL/GSSAPI authentication started
SASL username: admin@TESTRELM.COM
SASL SSF: 56
SASL data security layer installed.
# extended LDIF
#
# LDAPv3
# base <cn=sudorules,cn=sudo,dc=testrelm,dc=com> with scope subtree
# filter: cn=AlowRule
# requesting: ALL
#

# bc04b196-485b-11e3-bebb-0024810e5d52, sudorules, sudo, testrelm.com
dn: ipaUniqueID=bc04b196-485b-11e3-bebb-0024810e5d52,cn=sudorules,cn=sudo,dc=t
 estrelm,dc=com
objectClass: ipaassociation
objectClass: ipasudorule
cn: AlowRule
ipaEnabledFlag: TRUE
ipaUniqueID: bc04b196-485b-11e3-bebb-0024810e5d52

# search result
search: 4
result: 0 Success

# numResponses: 2
# numEntries: 1

This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.