Bug 1294506

Summary: named.service does not bind to ethernet interface(s) on startup
Product: Red Hat Enterprise Linux 7 Reporter: John L Magee <jlmagee>
Component: bindAssignee: Tomáš Hozza <thozza>
Status: CLOSED ERRATA QA Contact: Petr Sklenar <psklenar>
Severity: medium Docs Contact: Mirek Jahoda <mjahoda>
Priority: medium    
Version: 7.2CC: bkruell, blayo, chlunde, john, jscotka, lnykryn, nparmar, sebastiaan
Target Milestone: rcKeywords: Patch
Target Release: ---   
Hardware: All   
OS: Linux   
Fixed In Version: bind-9.9.4-36.el7 Doc Type: Release Note
Doc Text:
The *named* service now binds to all interfaces With this update, *BIND* is able to react to situations when a new IP address is added to an interface. If the new address is allowed by the configuration, *BIND* will automatically start to listen on that interface.
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-04 01:26:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description John L Magee 2015-12-28 14:59:04 UTC
Description of problem: the named.service does not bind to ethernet interface(s) on startup when the NetworkManager service is not installed

Version-Release number of selected component (if applicable):
CentOS 7 at 1511 level (equivalent to RHEL 7.2)

How reproducible: always

Steps to Reproduce:
1. create operational bind service on minimal VM
2. yum remove NetworkManager
3. reboot VM

Actual results:  named.service only bound to lo interface

Expected results:  named.service bound to all interfaces

Additional info:  changed dependencies in /etc/systemd/system/multi-user.target.wants/named.service  and  the named.service starts as expected


Comment 2 Lukáš Nykrýn 2016-01-05 11:32:55 UTC
This is weird, network.target is After network.service.

Can you please attach output of systemctl show network.target in the case without your workaround?

Can you also add "debug" to kernel cmdline and send me output of journactl -b?

Comment 3 Onmeac 2016-01-05 12:04:57 UTC
similar https://bugzilla.redhat.com/show_bug.cgi?id=1278015?

I had run in this issue as well.

One machine, multiple interfaces.

After a reboot named only listens to, not any of the IPv4 addresses.

sudo systemctl show network.target --no-pager

Before=salt-minion.service named.service rc-local.service sshd.service network-online.target
After=network-pre.target network.service
Documentation=man:systemd.special(7) http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget

Comment 4 Carl Henrik Lunde 2016-02-23 09:47:42 UTC
We've run into the same issue on RHEL 7.2 without NetworkManager with named-chroot-9.9.4-29.el7_2.2.x86_64.

It looks like bind-9.9.4 iterates over available interfaces on startup.  Listening to is not an option[0].  This changes in bind-9.10 [1].

Waiting for network-online.target[2] seems to work:

$ diff -u /usr/lib/systemd/system/named-chroot.service /etc/systemd/system/named-chroot.service
--- /usr/lib/systemd/system/named-chroot.service        2016-01-18 13:57:35.000000000 +0100
+++ /etc/systemd/system/named-chroot.service    2016-02-23 10:12:25.276687016 +0100
@@ -7,7 +7,7 @@


[0] http://permalink.gmane.org/gmane.network.dns.bind.user/31854

[1] https://kb.isc.org/article/AA-01153/0/BIND-9.10.0-Release-Notes.html

 * On operating systems that support routing sockets, including Mac OSX, *BSD and Linux, network interfaces are re-scanned automatically whenever they change.  Use "automatic-interface-scan no;" to disable this feature. [RT #23027]
 * Added "rndc scan" to trigger an interface scan manually. [RT #23027]


[2] https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/

"network.target has very little meaning during start-up. It only indicates that the network management stack is up after it has been reached."

"network-online.target is a target that actively waits until the nework is "up", where the definition of "up" is defined by the network management software. Usually it indicates a configured, routable IP address of some kind."

Comment 5 Tomáš Hozza 2016-05-10 08:16:15 UTC
*** Bug 1301912 has been marked as a duplicate of this bug. ***

Comment 6 Tomáš Hozza 2016-05-10 08:16:43 UTC
*** Bug 1278015 has been marked as a duplicate of this bug. ***

Comment 8 Tomáš Hozza 2016-05-11 17:58:14 UTC

BIND 9.9.4 binds to any addresses, which are available during the server startup and are allowed by the configuration. Versions previous to 9.10.x are not able to automatically detect, that there is a new address on some interface or that some address disappeared (as noted in comment #4).

This is technically solved by the dispatcher script when using NetworkManger. It reloads the server on any network configuration change and thus makes it to bind to new addresses or stop listening on addresses which disappeared. Note that the dispatcher script causes BIND to do complete reload, which is unnecessary and may take long time in case your zone file is really big.

This naturally does not work when you uninstall NetworkManger, as the dispatcher script is not run.

However BIND 9.9.4 is scanning interfaces regularly, but the interval is by default set to 60 minutes. This can be changed using "interface-interval" option in the general "options" section. I don't think that changing the default is really a solution.

Note, that if you wait 60 minutes, BIND will eventually bind to all addresses ;)

I don't think that changing the "network.target" to "network-online.target" is either a good solution. The reason is that for some system services which depend on DNS, it may be too late if BIND starts only after the network is fully configured.

I propose to:
- keep the "After=network.target" in all named service files
- add the automatic scan functionality from BIND 9.10.x to BIND in RHEL-7
- remove the NetworkManager dispatcher script from the package

I backported the needed changes and everything seems to work fine. IOW BIND binds to all addresses even if NetworkManager is not installed.

Comment 15 Blaz Podrzaj 2016-09-02 10:34:50 UTC
(In reply to Carl Henrik Lunde from comment #4)

> "network.target has very little meaning during start-up. It only indicates
> that the network management stack is up after it has been reached."

I wouldn't agree with the first part of this statement, given the fact that there is quite a number of servers still (maybe on purpose) running without NetworkManager.

The problem is in fact with systemd which changed its way of activating network.target and network-online.target somewhere between version 208 and 219 and I've found that totally by accident... I restarted the server after 6 months and Postfix didn't come up. I checked the journal and found out that it was started but died afterwards because the system still haven't got an IP on which Postfix was configured. And I knew that this was not happening on RHEL 7.1.

With version 208 both targets were still working with network.service as well as with NM but with 219 network.target is NM-dependent only whereas network-online.target is still network.service dependent! But nearly all network services still have "After=network.target" in their configuration. So if you run your server without NM (not even installed) then your system ends up without network.target being activated at all and result is network services being started as soon as multi-user.target kicks in and possibly before network interfaces are brought up.

Whether they did that on purpose or was there some greater good in such a decision, I have no idea, but surely such an update messed up a lot of servers. 

So there are two possibilities if you would still like to bring up your network via network.service only... one is reconfiguring all your network services' systemd configs to "After=network-online.target" or two, install NM but leave it disabled. In 2nd case, you would get an info during the boot process that says "systemd[1]: Dependency failed for Network Manager Wait Online." and that's it.

Well, maybe I'm missing something but as I see it right now, the current implementation of systemd in RHEL 7.2 is broken. If you would still like to use network.service only, you would end up without network.target, but if you would like to switch completely to NM (without network.service), you would end up without network-online.target. This was not the case in RHEL 7.1 where you could run either network.service or NM alone or both at the same time and both network.target and network-online.target were active in each and every situation.

Comment 18 errata-xmlrpc 2016-11-04 01:26:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.