Bug 971111 - DNA plugin failed to fetch replication agreement
DNA plugin failed to fetch replication agreement
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: 389-ds-base (Show other bugs)
7.0
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Rich Megginson
Sankar Ramalingam
:
Depends On: 970225
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-05 13:50 EDT by Nathan Kinder
Modified: 2014-06-17 22:59 EDT (History)
10 users (show)

See Also:
Fixed In Version: 389-ds-base-1.3.1.6-8.el7
Doc Type: Bug Fix
Doc Text:
Cause: Misconfiguration of the DNA plugin replication settings(for getting new ranges), where a replication server is specified, but there is no agreement present for that replication server. Consequence: The range from that remote replication server can not be obtained because the plugin tries to use the replication bind credentials from an agreement. In this case there is no agreement to that server, so it fails to get the next range. Fix: In the DNA plugin configuration, you can now specify the replication credentials. Result: If properly configured, there is no requirement to have a replication agreement in order to get a range from a remote replication server.
Story Points: ---
Clone Of: 970225
Environment:
Last Closed: 2014-06-13 08:39:36 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Nathan Kinder 2013-06-05 13:50:24 EDT
+++ This bug was initially created as a clone of Bug #970225 +++

Description of problem:

In an IPA environment, I'm seeing the DNA plugin fail to fetch a replication agreement.  The DNA plugin is trying a replica where there is no replication agreement.  This is causing ipa user-add to fail.

[root@ipaqa64vmd tmp.izaYf564ZD]# ipa user-add test --first=f --last=l
ipa: ERROR: Operations error: Allocation of a new value for range cn=posix ids,cn=distributed numeric assignment plugin,cn=plugins,cn=config failed! Unable to proceed.

[root@ipaqa64vmd tmp.izaYf564ZD]# ldapsearch -xLLL -D "$ROOTDN" -w "$ROOTDNPWD" -b "cn=posix ids,cn=distributed numeric assignment plugin,cn=plugins,cn=config"
dn: cn=Posix IDs,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config
objectClass: top
objectClass: extensibleObject
cn: Posix IDs
dnaType: uidNumber
dnaType: gidNumber
dnaNextValue: 1101
dnaMaxValue: 1100
dnaMagicRegen: -1
dnaFilter: (|(objectClass=posixAccount)(objectClass=posixGroup)(objectClass=ip
 aIDobject))
dnaScope: dc=testrelm,dc=com
dnaThreshold: 500
dnaSharedCfgDN: cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=testrelm,dc=com

So, looking in the logs at the time of the failure:

[29/May/2013:10:03:14 -0400] dna-plugin - dna_get_replica_bind_creds: Failed to fetch replication agreement for range cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=testrelm,dc=com, server ipaqa64vmf.testrelm.com, port 389
[29/May/2013:10:03:14 -0400] dna-plugin - dna_request_range: Unable to retrieve replica bind credentials.
...
[29/May/2013:10:03:14 -0400] dna-plugin - dna_get_replica_bind_creds: Failed to fetch replication agreement for range cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=testrelm,dc=com, server cloud-qe-15.testrelm.com, port 389
[29/May/2013:10:03:14 -0400] dna-plugin - dna_request_range: Unable to retrieve replica bind credentials.
[29/May/2013:10:03:14 -0400] dna-plugin - dna_pre_op: no more values available!!

After some help from Dev, it was pointed out that my IPA replica is running the dna-plugin.  The plugin fails to get the range from the master because it doesn't actually have a replication agreement with that master.

Topology is:

R1 - M - R2 - R3 - R4

Failure is occurring on R3.  dna-plugin on R3 is attempting to contact M but, there is not replication agreement.  M="master" and was the first IPA server setup in the environment.  


Version-Release number of selected component (if applicable):
389-ds-base-1.3.0.6-1.fc18.x86_64

How reproducible:
very

Steps to Reproduce:
1.  Setup IPA environment with similar topology.  
2.  On R3 or R4, ipa user-add

Actual results:
failure like above.

Expected results:
dna-plugin accurately looks up the range.  

Additional info:

--- Additional comment from Nathan Kinder on 2013-06-03 15:01:54 EDT ---

Upstream ticket:
https://fedorahosted.org/389/ticket/47379
Comment 1 Nathan Kinder 2013-06-05 13:51:45 EDT
If DNA does not find a replication agreement for the first range it selected for a transfer operation, it should cycle through the rest of the available ranges in the shared config (in descending order of available range size). This should continue until we perform a successful range transfer or run out of servers who have available range values.
Comment 2 mreynolds 2013-06-17 11:21:42 EDT
This has been fixed and pushed upstream for 1.3.2
Comment 3 Nathan Kinder 2013-07-07 22:41:02 EDT
This has been fixed upstream in 389-ds-base-1.3.1.3.
Comment 4 Rich Megginson 2013-10-01 19:24:12 EDT
moving all ON_QA bugs to MODIFIED in order to add them to the errata (can't add bugs in the ON_QA state to an errata).  When the errata is created, the bugs should be automatically moved back to ON_QA.
Comment 6 Scott Poore 2013-11-05 10:20:46 EST
Moving this one back to assigned as I'm still seeing this bug in RHEL7.  

Version ::

389-ds-base-1.3.1.6-5.el7.x86_64

Test Failure ::

:: [   PASS   ] :: Running 'ssh ibm-x3650m4-02-vm-01.testrelm.com 'echo Secret123| kinit admin'' (Expected 0, got 0)
:: [   FAIL   ] :: Running 'ssh ibm-x3650m4-02-vm-01.testrelm.com 'ipa user-add testuser012256 --first=test --last=user' > /tmp/output.irm_useradd 2>&1' (Expected 0, got 1)
ipa: ERROR: Operations error: Allocation of a new value for range cn=posix ids,cn=distributed numeric assignment plugin,cn=plugins,cn=config failed! Unable to proceed.
:: [   PASS   ] :: Running 'cat /tmp/output.irm_useradd' (Expected 0, got 0)
:: [   FAIL   ] :: File '/tmp/output.irm_useradd' should not contain 'Allocation of a new value for range.*failed' 
[05/Nov/2013:01:23:03 -0500] dna-plugin - dna_get_replica_bind_creds: Failed to fetch replication agreement for range cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=testrelm,dc=com, server mgmt9.testrelm.com, port 389
[05/Nov/2013:01:23:03 -0500] dna-plugin - dna_get_replica_bind_creds: Failed to fetch replication agreement for range cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=testrelm,dc=com, server cloud-qe-5.testrelm.com, port 389
:: [   PASS   ] :: Running 'ssh ibm-x3650m4-02-vm-01.testrelm.com "grep 'dna-plugin.*Failed to fetch replication agreement for range' /var/log/dirsrv/slapd-TESTRELM-COM/errors"|tail -2' (Expected 0, got 0)
:: [   FAIL   ] :: BZ 970225 found...DNA plugin failed to fetch replication agreement 

Note this is a clone of bug #970225.
Comment 7 mreynolds 2013-11-05 10:35:01 EST
(In reply to Scott Poore from comment #6)
> Moving this one back to assigned as I'm still seeing this bug in RHEL7.  
> 
> Version ::
> 
> 389-ds-base-1.3.1.6-5.el7.x86_64
> 
> Test Failure ::
> 
> :: [   PASS   ] :: Running 'ssh ibm-x3650m4-02-vm-01.testrelm.com 'echo
> Secret123| kinit admin'' (Expected 0, got 0)
> :: [   FAIL   ] :: Running 'ssh ibm-x3650m4-02-vm-01.testrelm.com 'ipa
> user-add testuser012256 --first=test --last=user' > /tmp/output.irm_useradd
> 2>&1' (Expected 0, got 1)
> ipa: ERROR: Operations error: Allocation of a new value for range cn=posix
> ids,cn=distributed numeric assignment plugin,cn=plugins,cn=config failed!
> Unable to proceed.
> :: [   PASS   ] :: Running 'cat /tmp/output.irm_useradd' (Expected 0, got 0)
> :: [   FAIL   ] :: File '/tmp/output.irm_useradd' should not contain
> 'Allocation of a new value for range.*failed' 
> [05/Nov/2013:01:23:03 -0500] dna-plugin - dna_get_replica_bind_creds: Failed
> to fetch replication agreement for range
> cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=testrelm,dc=com, server
> mgmt9.testrelm.com, port 389
> [05/Nov/2013:01:23:03 -0500] dna-plugin - dna_get_replica_bind_creds: Failed
> to fetch replication agreement for range
> cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=testrelm,dc=com, server
> cloud-qe-5.testrelm.com, port 389
> :: [   PASS   ] :: Running 'ssh ibm-x3650m4-02-vm-01.testrelm.com "grep
> 'dna-plugin.*Failed to fetch replication agreement for range'
> /var/log/dirsrv/slapd-TESTRELM-COM/errors"|tail -2' (Expected 0, got 0)
> :: [   FAIL   ] :: BZ 970225 found...DNA plugin failed to fetch replication
> agreement 
> 
> Note this is a clone of bug #970225.

Scott, if there is no replication agreement to the replica for the next range then the bind DN, password, protocol, etc need to be added to the plugin's configuration/shared config for the remote replica that has the next range:

objectClass: dnaPluginConfig
dnaRemoteBindDN 
dnaRemoteBindCred

objectClass: dnaSharedConfig
dnaRemoteBindMethod  (SIMPLE, SSL, SASL/DIGEST-MD5, or SASL/GSSAPI)
dnaRemoteConnProtocol:  (LDAP, TLS, or SSL)


Can you confirm if there is an agreement for mgmt9.testrelm.com:389?  DNA uses the authentication credentials and protocol settings from the agreement to get the next range.  If there is no agreement it can not make contact with the remote replica - hence this bug.  

So, in this case you need to update the DNA plugin configuration and shared server config with the attributes listed above so it knows how to contact the remote replica.
Comment 9 Scott Poore 2014-01-28 11:35:22 EST
What was changed here?  

Considering the other discussions around bug #1029640, and necessary work there will cover support for replicas seeing the DNA ranges, what do I need to test here to verify this fix?

Just check the log for if I'm still seeing this error:

dna-plugin.*Failed to fetch replication agreement for range

Thanks,
Scott
Comment 10 Nathan Kinder 2014-03-03 10:50:06 EST
(In reply to Scott Poore from comment #9)
> What was changed here?  

The ability to add dnaRemoteBindDN and dnaRemoteBindCred attributes is what was implemented here, which allows DNA ranges to be transferred between masters that don't have direct replication agreements.

> 
> Considering the other discussions around bug #1029640, and necessary work
> there will cover support for replicas seeing the DNA ranges, what do I need
> to test here to verify this fix?

Well, this should be verified as a RHDS issue.  IPA doesn't yet use this new functionality, so the RHDS QE team should be setting up 3 masters in a chain where 2 of the masters don't have a direct replication agreement between them:

    M1 <---> M2 <---> M3

DNA needs to be configured such that M3 has lots of range available, but M1 and M2 are exhausted.  With the new attributes configured, adding an entry to M1 that would trigger DNA should result in range being transferred from M3.  Prior to the fix for this issue, M1 would not know how to connect and bind to M3 since it did not have a replication agreement to M3 that it could extract the information from.
Comment 11 Milan Kubík 2014-03-11 05:24:07 EDT
By using the topology suggested by Nathan:

# ldapsearch -xLLL -o ldif-wrap=no -h localhost -p 2389 -D "cn=directory manager" -w Secret123 -b "cn=Account UIDs,ou=ranges,dc=example,dc=com" dnaportnum dnaremainingvalues dnaremoteconnprotocol dnaremotebindmethod
dn: cn=Account UIDs,ou=Ranges,dc=example,dc=com

dn: dnaHostname=example.com+dnaPortNum=2389,cn=Account UIDs,ou=Ranges,dc=example,dc=com
dnaportnum: 2389
dnaremainingvalues: 9
dnaremoteconnprotocol: LDAP
dnaremotebindmethod: SIMPLE

dn: dnaHostname=example.com+dnaPortNum=3389,cn=Account UIDs,ou=Ranges,dc=example,dc=com
dnaportnum: 3389
dnaremainingvalues: 196
dnaremoteconnprotocol: LDAP
dnaremotebindmethod: SIMPLE

dn: dnaHostname=example.com+dnaPortNum=1389,cn=Account UIDs,ou=Ranges,dc=example,dc=com
dnaportnum: 1389
dnaremainingvalues: 10
dnaremoteconnprotocol: LDAP
dnaremotebindmethod: SIMPLE


# ldapsearch -xLLL -o ldif-wrap=no -h localhost -p 1389 -D "cn=directory manager" -w Secret123 -s sub -b "cn=distributed numeric assignment plugin,cn=plugins,cn=config" "cn=Account UIDs" dnanextvalue dnamaxvalue dnathreshold dnanextrange
dn: cn=Account UIDs,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config
dnanextvalue: 91
dnamaxvalue: 100
dnathreshold: 10

# ldapsearch -xLLL -o ldif-wrap=no -h localhost -p 3389 -D "cn=directory manager" -w Secret123 -s sub -b "cn=distributed numeric assignment plugin,cn=plugins,cn=config" "cn=Account UIDs" dnanextvalue dnamaxvalue dnathreshold dnanextrange
dn: cn=Account UIDs,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config
dnanextvalue: 305
dnamaxvalue: 500
dnathreshold: 10


adding an user
# cat posix_users_to_m1.ldif 
dn: uid=range_user11,ou=people,dc=example,dc=com
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetorgperson
objectclass: posixAccount
telephoneNumber: 98989819
mail: range_user11@redhat.com
uid: range_user11
givenName: range_user11
sn: range_user11
cn: range_user11
homeDirectory: /home/range_user11
loginShell: /bin/bash
userPassword: Secret123
uidNumber: 0
gidNumber: 0

# ldapmodify -x -h localhost -p 1389 -D "cn=directory manager" -w Secret123 -avf posix_users_to_m1.ldif 

checking the configuration
# ldapsearch -xLLL -o ldif-wrap=no -h localhost -p 2389 -D "cn=directory manager" -w Secret123 -b "cn=Account UIDs,ou=ranges,dc=example,dc=com" dnaportnum dnaremainingvalues dnaremoteconnprotocol dnaremotebindmethod
dn: cn=Account UIDs,ou=Ranges,dc=example,dc=com

dn: dnaHostname=example.com+dnaPortNum=2389,cn=Account UIDs,ou=Ranges,dc=example,dc=com
dnaportnum: 2389
dnaremainingvalues: 9
dnaremoteconnprotocol: LDAP
dnaremotebindmethod: SIMPLE

dn: dnaHostname=example.com+dnaPortNum=3389,cn=Account UIDs,ou=Ranges,dc=example,dc=com
dnaportnum: 3389
dnaremainingvalues: 106
dnaremoteconnprotocol: LDAP
dnaremotebindmethod: SIMPLE

dn: dnaHostname=example.com+dnaPortNum=1389,cn=Account UIDs,ou=Ranges,dc=example,dc=com
dnaportnum: 1389
dnaremainingvalues: 93
dnaremoteconnprotocol: LDAP
dnaremotebindmethod: SIMPLE

# ldapsearch -xLLL -o ldif-wrap=no -h localhost -p 1389 -D "cn=directory manager" -w Secret123 -s sub -b "cn=distributed numeric assignment plugin,cn=plugins,cn=config" "cn=Account UIDs" dnanextvalue dnamaxvalue dnathreshold dnanextrange
dn: cn=Account UIDs,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config
dnanextvalue: 98
dnamaxvalue: 100
dnathreshold: 10
dnanextrange: 411-500

# ldapsearch -xLLL -o ldif-wrap=no -h localhost -p 3389 -D "cn=directory manager" -w Secret123 -s sub -b "cn=distributed numeric assignment plugin,cn=plugins,cn=config" "cn=Account UIDs" dnanextvalue dnamaxvalue dnathreshold dnanextrange
dn: cn=Account UIDs,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config
dnanextvalue: 305
dnamaxvalue: 410
dnathreshold: 10

Using 389-ds-base-1.3.1.6-22.el7
Marking as verified.
Comment 12 Ludek Smid 2014-06-13 08:39:36 EDT
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.

Note You need to log in before you can comment on or make changes to this bug.