Bug 1010396

Summary: PTR record synchronization can deadlock if connection count <= 2 (only plugin versions < 3.0)
Product: Red Hat Enterprise Linux 6 Reporter: Dmitri Pal <dpal>
Component: bind-dyndb-ldapAssignee: Petr Spacek <pspacek>
Status: CLOSED ERRATA QA Contact: Namita Soman <nsoman>
Severity: unspecified Docs Contact:
Priority: medium    
Version: 6.4CC: mkosek, mrhodes, parsonsa, pspacek, tlavigne
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: bind-dyndb-ldap-2.3-5.el6 Doc Type: Bug Fix
Doc Text:
Cause: Bind-dyndb-ldap plugin with default configuration didn't establish enough connections to LDAP server for 'PTR record synchronization' feature. Consequence: PTR record synchronization failed. Fix: Default number of connections was raised to 4. Result: PTR record synchronization works.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2013-11-21 12:11:22 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
0001-Prevent-deadlock-in-PTR-record-synchronization.patch none

Description Dmitri Pal 2013-09-20 17:02:35 UTC 
This bug is created as a clone of upstream ticket:
https://fedorahosted.org/bind-dyndb-ldap/ticket/113

PTR record synchronization doesn't work in certain scenarios.

'''Steps to reproduce'''
1. Configure IPA in Fedora 18.
2. Enable DNS dynamic updates for forward and reverse zone.
3. Enable PTR record synchronization for forward zone.
4. Install IPA client with `ipa-client-install --domain=testrelm.com --realm=TESTRELM.COM -p admin -w Secret123 --unattended --server=f18-ipa-master.testrelm.com --enable-dns-updates`

'''Symptoms'''

DNS update will fail
{{{
Failed to update DNS records.
}}}


`/var/named/data/named.run`
{{{
timeout in ldap_pool_getconnection(): try to raise 'connections' parameter; potential deadlock?
Can not synchronize PTR record, ldapdb_rdatalist_get = 2
update_record (psearch) failed, dn 'idnsname=173,idnsname=201.65.10.in-addr.arpa.,cn=dns,dc=testrelm,dc=com' change type 0x4. Records can be outdated, run `rndc reload`: not found
}}}


'''Investigation'''

`/etc/named.conf`
{{{
dynamic-db "ipa" {
        library "ldap.so";
        arg "uri ldapi://%2fvar%2frun%2fslapd-TESTRELM-COM.socket";
        arg "base cn=dns, dc=testrelm,dc=com";
        arg "fake_mname f18-ipa-master.testrelm.com.";
        arg "auth_method sasl";
        arg "sasl_mech GSSAPI";
        arg "sasl_user DNS/f18-ipa-master.testrelm.com";
        arg "zone_refresh 0";
        arg "psearch yes";
        arg "serial_autoincrement yes";
};
}}}

* Connection count has to be `<= 2` to reproduce the problem.
* One connection is reserved purely for persistent search, i.e. one connection is not enough for sync_ptr.
Comment 1 Dmitri Pal 2013-09-20 17:04:52 UTC 
Details of a specific use case: RHEL 6.4 w/ bind-dyndb-ldap-2.3-2.el6_4.1.x86_64 and ipa-server-3.0.0-26.el6_4.4.x86_64
Comment 3 Petr Spacek 2013-09-23 07:43:54 UTC 
This problem is known upstream and fix for the versions 2.x was deferred.

This bug is fixed in next major version of bind-dyndb-ldap (3.x) as a side-effect of internal refactoring work.

Backport to 2.x would be pretty hard.
Comment 5 Petr Spacek 2013-09-26 12:40:13 UTC 
Created attachment 803383 [details]
0001-Prevent-deadlock-in-PTR-record-synchronization.patch
Comment 8 Namita Soman 2013-10-15 17:54:47 UTC 
Verified using bind-dyndb-ldap-2.3-5.el6.x86_64

Steps taken:
1>  Enable DNS dynamic updates for forward and reverse zone

after ipa server install - it was enabled, didn't have to do it.

# ipa dnszone-show testrelm.com --all
  dn: idnsname=testrelm.com,cn=dns,dc=testrelm,dc=com
  Zone name: testrelm.com
  Authoritative nameserver: abc.testrelm.com.
  Administrator e-mail address: hostmaster.testrelm.com.
  SOA serial: 1381856857
  SOA refresh: 3600
  SOA retry: 900
  SOA expire: 1209600
  SOA minimum: 3600
  BIND update policy: grant TESTRELM.COM krb5-self * A; grant TESTRELM.COM krb5-self * AAAA; grant TESTRELM.COM krb5-self * SSHFP;
  Active zone: TRUE
  Dynamic update: TRUE
  Allow query: any;
  Allow transfer: none;
  nsrecord: abc.testrelm.com.
  objectclass: top, idnsrecord, idnszone

# ipa dnszone-show 4.3.2.in-addr.arpa. --all
  dn: idnsname=98.16.10.in-addr.arpa.,cn=dns,dc=testrelm,dc=com
  Zone name: 98.16.10.in-addr.arpa.
  Authoritative nameserver: abc.testrelm.com.
  Administrator e-mail address: hostmaster.testrelm.com.
  SOA serial: 1381856769
  SOA refresh: 3600
  SOA retry: 900
  SOA expire: 1209600
  SOA minimum: 3600
  BIND update policy: grant TESTRELM.COM krb5-subdomain 4.3.2.in-addr.arpa. PTR;
  Active zone: TRUE
  Dynamic update: TRUE
  Allow query: any;
  Allow transfer: none;
  nsrecord: abc.testrelm.com.
  objectclass: top, idnsrecord, idnszone


2> Enable PTR record synchronization for forward zone.
# ipa dnszone-mod --allow-sync-ptr=1 testrelm.com
  Zone name: testrelm.com
  Authoritative nameserver: abc.testrelm.com.
  Administrator e-mail address: hostmaster.testrelm.com.
  SOA serial: 1381856857
  SOA refresh: 3600
  SOA retry: 900
  SOA expire: 1209600
  SOA minimum: 3600
  Active zone: TRUE
  Allow query: any;
  Allow transfer: none;
  Allow PTR sync: TRUE

3>  Install IPA client with
 ipa-client-install --domain=testrelm.com --realm=TESTRELM.COM -p admin -w Secret123 --server=ipaqa64vmj.testrelm.com
Autodiscovery of servers for failover cannot work with this configuration.
If you proceed with the installation, services will be configured to always access the discovered server for all operations and will not fail over to other servers in case of failure.
Proceed with fixed values and no DNS discovery? [no]: y
Hostname: xyz.testrelm.com
Realm: TESTRELM.COM
DNS Domain: testrelm.com
IPA Server: abc.testrelm.com
BaseDN: dc=testrelm,dc=com

Continue to configure the system with these values? [no]: y
Synchronizing time with KDC...
Unable to sync time with IPA NTP server, assuming the time is in sync. Please check that 123 UDP port is opened.
Enrolled in IPA realm TESTRELM.COM
Created /etc/ipa/default.conf
New SSSD config will be created
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm TESTRELM.COM
trying https://abc.testrelm.com/ipa/xml
Forwarding 'env' to server u'https://abc.testrelm.com/ipa/xml'
Adding SSH public key from /etc/ssh/ssh_host_dsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
Forwarding 'host_mod' to server u'https://abc.testrelm.com/ipa/xml'
SSSD enabled
Configured /etc/openldap/ldap.conf
NTP enabled
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config
Client configuration complete.


Verified that DNS update was successful - didn't see message - "Failed to update DNS records"
On master:
# ipa dnsrecord-find testrelm.com xyz
  Record name: xyz
  A record: 9.8.7.6
  SSHFP record: 2 1 8179641DAC49E7B2D15F0546419891C769170537, 1 1 51BB889CFB53FC8591D7AC7EEBCE79DA43EB30CC
----------------------------
Number of entries returned 1
----------------------------
Comment 9 Petr Spacek 2013-10-16 06:45:41 UTC 
Could you check that PTR record was created properly, please?
Comment 10 Namita Soman 2013-10-17 13:07:12 UTC 
Verified PTR record is created.

# dig -x <$clientip>

<snip>
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24386
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

;; QUESTION SECTION:
;<$clientipreverse>.in-addr.arpa.       IN        PTR

;; ANSWER SECTION:
<$clientipreverse>.in-addr.arpa. 1200        IN        PTR        <$clienthostname>.

;; AUTHORITY SECTION:
<$serveripreverse>.in-addr.arpa.        86400        IN        NS        <$serverhostname>.

;; ADDITIONAL SECTION:
<$serverhostname>. 1200        IN        A        <$serverip>

<snip>


# ipa dnsrecord-find --all <$serverreverseip>.in-addr.arpa.
  dn: idnsName=179,idnsname=<$serverreverseip>.in-addr.arpa.,cn=dns,dc=testrelm,dc=com
  Record name: 179
  Time to live: 1200
  PTR record: <$clienthostname>.
  objectclass: idnsRecord, top
Comment 11 Petr Spacek 2013-10-17 14:23:02 UTC 
Great, thank you!
Comment 12 errata-xmlrpc 2013-11-21 12:11:22 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1636.html