Bug 662930

Summary: named fails to start if using local ldap server
Product: Red Hat Enterprise Linux 6 Reporter: Phil Anderson <pza>
Component: bind-dyndb-ldapAssignee: Adam Tkac <atkac>
Status: CLOSED ERRATA QA Contact: Chandrasekar Kannan <ckannan>
Severity: medium Docs Contact:
Priority: low    
Version: 6.0CC: benl, jgalipea, jturner, mgregg, ovasik
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-05-19 13:35:06 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 667704    

Description Phil Anderson 2010-12-14 08:35:17 UTC 
While this isn't a bug in bind-dyndb-ldap as such, it is only a problem when using the package.

As named is start before slapd in RHEL6, named won't start on system boot if configured to use bind-dyndb-ldap if pointing to an ldap server on the local machine.

named bails out with the following:
Dec 14 19:27:36 dominic named[1250]: registering dynamic ldap driver for ldapdns.
Dec 14 19:27:36 dominic named[1250]: trying to establish LDAP connection to ldap://xxxxxxx
Dec 14 19:27:36 dominic named[1250]: bind to LDAP server failed: Can't contact LDAP server
Dec 14 19:27:36 dominic named[1250]: loading configuration: failure
Dec 14 19:27:36 dominic named[1250]: exiting (due to fatal error)
Comment 3 Jay Turner 2010-12-16 15:06:31 UTC 
What is the proposed solution?  Are we talking about changing the startup sequence of named and slapd?  A change in bind-dyndb-ldap?  QE cannot ack without fully understanding the proposed change.
Comment 4 Adam Tkac 2010-12-16 15:46:24 UTC 
(In reply to comment #3)
> What is the proposed solution?  Are we talking about changing the startup
> sequence of named and slapd?  A change in bind-dyndb-ldap?  QE cannot ack
> without fully understanding the proposed change.

The solution will be to patch bind-dyndb-ldap plugin to periodically reconnect to LDAP server in case connection is lost (or plugin fails to connect to LDAP, which is this case).
Comment 5 Adam Tkac 2011-01-12 15:32:24 UTC 
I created the patch for this issue (https://www.redhat.com/archives/freeipa-devel/2011-January/msg00256.html). With the patch named starts fine but you must call "rndc reload" (you can put it into /etc/rc.local) after LDAP is started to fetch zones stored in LDAP.
Comment 9 Adam Tkac 2011-04-04 12:03:41 UTC 
(In reply to comment #8)
> Please add steps to reproduce/verify this bug. thanks

1. install freeipa server (as described in https://bugzilla.redhat.com/show_bug.cgi?id=666244#c6, "1." and "2.")

2. verify named is running

3. in /etc/named.conf, in dynamic-db "ipa" { }; section, replace line 'arg "uri ldapi:// ...' with 'arg "uri ldap://127.0.0.1";'

4. run `service named restart` and verify named still runs

5. block ports 389 and 636 (LDAP ports) on firewall (for example via `iptables -I INPUT -p tcp --dport 636 -j REJECT`; `iptables -I INPUT -p tcp --dport 389 -j REJECT`)

6. try to restart named. With old plugin named fails to start, with new plugin named starts with following messages in the log:
"bind to LDAP server failed: Can't contact LDAP server"

Feel free to ping me on IRC if you have any question.
Comment 10 Jenny Severance 2011-04-07 18:41:21 UTC 
1) installed IPA

2) service named status
rndc: neither /etc/rndc.conf nor /etc/rndc.key was found
named (pid  12627) is running...

3) with uri ldap://127.0.0.1 - restarted named - okay

4) stopped directory servers
# service dirsrv stop
Shutting down dirsrv: 
    PKI-IPA...[  OK  ]
    TESTRELM...[  OK  ]


5) restarted named
# service named restart
Stopping named: .[  OK  ]
Starting named: [FAILED]


/var/log/messages ...

Apr  7 14:35:02 ipaqa64vmb named[13900]: shutting down
Apr  7 14:35:02 ipaqa64vmb named[13900]: no longer listening on ::#53
Apr  7 14:35:02 ipaqa64vmb named[13900]: no longer listening on 127.0.0.1#53
Apr  7 14:35:02 ipaqa64vmb named[13900]: no longer listening on 10.16.98.183#53
Apr  7 14:35:02 ipaqa64vmb named[13900]: exiting
Apr  7 14:35:04 ipaqa64vmb sssd[be[testrelm]]: LDAP connection error: (null)
Apr  7 14:35:04 ipaqa64vmb named[14106]: starting BIND 9.7.3-RedHat-9.7.3-2.el6 -u named
Apr  7 14:35:04 ipaqa64vmb named[14106]: built with '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-libtool' '--localstatedir=/var' '--enable-threads' '--enable-ipv6' '--with-pic' '--disable-static' '--disable-openssl-version-check' '--with-dlz-ldap=yes' '--with-dlz-postgres=yes' '--with-dlz-mysql=yes' '--with-dlz-filesystem=yes' '--with-gssapi=yes' '--disable-isc-spnego' '--with-docbook-xsl=/usr/share/sgml/docbook/xsl-stylesheets' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'target_alias=x86_64-redhat-linux-gnu' 'CFLAGS= -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' 'CPPFLAGS= -DDIG_SIGCHASE'
Apr  7 14:35:04 ipaqa64vmb named[14106]: adjusted limit on open files from 1024 to 1048576
Apr  7 14:35:04 ipaqa64vmb named[14106]: found 2 CPUs, using 2 worker threads
Apr  7 14:35:04 ipaqa64vmb named[14106]: using up to 4096 sockets
Apr  7 14:35:04 ipaqa64vmb named[14106]: loading configuration from '/etc/named.conf'
Apr  7 14:35:04 ipaqa64vmb named[14106]: using default UDP/IPv4 port range: [1024, 65535]
Apr  7 14:35:04 ipaqa64vmb named[14106]: using default UDP/IPv6 port range: [1024, 65535]
Apr  7 14:35:04 ipaqa64vmb named[14106]: listening on IPv6 interfaces, port 53
Apr  7 14:35:04 ipaqa64vmb named[14106]: listening on IPv4 interface lo, 127.0.0.1#53
Apr  7 14:35:04 ipaqa64vmb named[14106]: listening on IPv4 interface eth0, 10.16.98.183#53
Apr  7 14:35:04 ipaqa64vmb named[14106]: generating session key for dynamic DNS
Apr  7 14:35:04 ipaqa64vmb named[14106]: Failed to init credentials (Generic error (see e-text))
Apr  7 14:35:04 ipaqa64vmb named[14106]: loading configuration: failure
Apr  7 14:35:04 ipaqa64vmb named[14106]: exiting (due to fatal error)


versions tested:

bind-dyndb-ldap-0.2.0-1.el6.x86_64
bind-9.7.3-2.el6.x86_64
ipa-server-2.0.0-20.el6.x86_64
389-ds-base-1.2.8.0-2.el6.x86_64
Comment 15 Adam Tkac 2011-04-11 14:25:06 UTC 
(In reply to comment #10)
> 1) installed IPA
> 
> 2) service named status
> rndc: neither /etc/rndc.conf nor /etc/rndc.key was found
> named (pid  12627) is running...
> 
> 3) with uri ldap://127.0.0.1 - restarted named - okay
> 
> 4) stopped directory servers
> # service dirsrv stop
> Shutting down dirsrv: 
>     PKI-IPA...[  OK  ]
>     TESTRELM...[  OK  ]

Please don't stop dirsrv, block ports on firewall instead.

All Kerberos keys are stored in LDAP (via dirsrv service) so when you stop dirsrv, named daemon is not able to obtain Kerberos credentials and it fails to start. This is considered as fatal error.

When you block LDAP ports on firewall and tell named to use 127.0.0.1 to connect to LDAP, named is able to obtain Kerberos credentials via local ldapi socket and then it fails to obtain zones from LDAP (which is not fatal). The fix should be verified this way.

> 5) restarted named
> # service named restart
> Stopping named: .[  OK  ]
> Starting named: [FAILED]
> 
> 
> /var/log/messages ...
> 
> Apr  7 14:35:02 ipaqa64vmb named[13900]: shutting down
> Apr  7 14:35:02 ipaqa64vmb named[13900]: no longer listening on ::#53
> Apr  7 14:35:02 ipaqa64vmb named[13900]: no longer listening on 127.0.0.1#53
> Apr  7 14:35:02 ipaqa64vmb named[13900]: no longer listening on 10.16.98.183#53
> Apr  7 14:35:02 ipaqa64vmb named[13900]: exiting
> Apr  7 14:35:04 ipaqa64vmb sssd[be[testrelm]]: LDAP connection error: (null)
> Apr  7 14:35:04 ipaqa64vmb named[14106]: starting BIND 9.7.3-RedHat-9.7.3-2.el6
> -u named
> Apr  7 14:35:04 ipaqa64vmb named[14106]: built with
> '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu'
> '--target=x86_64-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr'
> '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin'
> '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include'
> '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib'
> '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-libtool'
> '--localstatedir=/var' '--enable-threads' '--enable-ipv6' '--with-pic'
> '--disable-static' '--disable-openssl-version-check' '--with-dlz-ldap=yes'
> '--with-dlz-postgres=yes' '--with-dlz-mysql=yes' '--with-dlz-filesystem=yes'
> '--with-gssapi=yes' '--disable-isc-spnego'
> '--with-docbook-xsl=/usr/share/sgml/docbook/xsl-stylesheets'
> 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu'
> 'target_alias=x86_64-redhat-linux-gnu' 'CFLAGS= -O2 -g -pipe -Wall
> -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
> --param=ssp-buffer-size=4 -m64 -mtune=generic' 'CPPFLAGS= -DDIG_SIGCHASE'
> Apr  7 14:35:04 ipaqa64vmb named[14106]: adjusted limit on open files from 1024
> to 1048576
> Apr  7 14:35:04 ipaqa64vmb named[14106]: found 2 CPUs, using 2 worker threads
> Apr  7 14:35:04 ipaqa64vmb named[14106]: using up to 4096 sockets
> Apr  7 14:35:04 ipaqa64vmb named[14106]: loading configuration from
> '/etc/named.conf'
> Apr  7 14:35:04 ipaqa64vmb named[14106]: using default UDP/IPv4 port range:
> [1024, 65535]
> Apr  7 14:35:04 ipaqa64vmb named[14106]: using default UDP/IPv6 port range:
> [1024, 65535]
> Apr  7 14:35:04 ipaqa64vmb named[14106]: listening on IPv6 interfaces, port 53
> Apr  7 14:35:04 ipaqa64vmb named[14106]: listening on IPv4 interface lo,
> 127.0.0.1#53
> Apr  7 14:35:04 ipaqa64vmb named[14106]: listening on IPv4 interface eth0,
> 10.16.98.183#53
> Apr  7 14:35:04 ipaqa64vmb named[14106]: generating session key for dynamic DNS
> Apr  7 14:35:04 ipaqa64vmb named[14106]: Failed to init credentials (Generic
> error (see e-text))
> Apr  7 14:35:04 ipaqa64vmb named[14106]: loading configuration: failure
> Apr  7 14:35:04 ipaqa64vmb named[14106]: exiting (due to fatal error)

This error means named failed to init Kerberos credentials which is, as written above, considered as fatal error.
Comment 17 Michael Gregg 2011-04-27 18:27:44 UTC 
Again, with ports blocked:

iptables:
[root@ipaqa64vmb ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
REJECT     tcp  --  anywhere             anywhere            tcp dpt:ldap reject-with icmp-port-unreachable 
REJECT     tcp  --  anywhere             anywhere            tcp dpt:ldaps reject-with icmp-port-unreachable 

Dirsrv is still running:
17150 ?        Sl     1:11 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-PKI-IPA -i /var/run/dirsrv/slapd-PKI-IPA.pid -w /var/run/dirsrv/slapd-PKI-IPA.startpid
17227 ?        Sl     2:28 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-TESTRELM -i /var/run/dirsrv/slapd-TESTRELM.pid -w /var/run/dirsrv/slapd-TESTRELM.startpid


restarting named gives:
Apr 27 14:23:11 ipaqa64vmb sssd[be[testrelm]]: LDAP connection error: (null)
Apr 27 14:23:11 ipaqa64vmb named[4179]: starting BIND 9.7.3-RedHat-9.7.3-2.el6 -u named

Bind is now running, despite the connection error. 

Verified against:
ipa-server-2.0.0-23.el6.x86_64
bind-dyndb-ldap-0.2.0-3.20110426T0344z.el6.x86_64
bind-9.7.3-2.el6.x86_64
Comment 18 errata-xmlrpc 2011-05-19 13:35:06 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0606.html