Bug 1294506

This is weird, network.target is After network.service.

Can you please attach output of systemctl show network.target in the case without your workaround?

Can you also add "debug" to kernel cmdline and send me output of journactl -b?

similar https://bugzilla.redhat.com/show_bug.cgi?id=1278015?

I had run in this issue as well.
https://lists.centos.org/pipermail/centos-devel/2016-January/014194.html


One machine, multiple interfaces.

After a reboot named only listens to 127.0.0.1, not any of the IPv4 addresses.


sudo systemctl show network.target --no-pager

Id=network.target
Names=network.target
Conflicts=shutdown.target
Before=salt-minion.service named.service rc-local.service sshd.service network-online.target
After=network-pre.target network.service
Documentation=man:systemd.special(7) http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget
Description=Network
LoadState=loaded
ActiveState=inactive
SubState=dead
FragmentPath=/usr/lib/systemd/system/network.target
UnitFileState=static
UnitFilePreset=disabled
InactiveExitTimestampMonotonic=0
ActiveEnterTimestampMonotonic=0
ActiveExitTimestampMonotonic=0
InactiveEnterTimestampMonotonic=0
CanStart=no
CanStop=yes
CanReload=no
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=yes
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=no
IgnoreOnSnapshot=no
NeedDaemonReload=no
JobTimeoutUSec=0
JobTimeoutAction=none
ConditionResult=no
AssertResult=no
ConditionTimestampMonotonic=0
AssertTimestampMonotonic=0
Transient=no

We've run into the same issue on RHEL 7.2 without NetworkManager with named-chroot-9.9.4-29.el7_2.2.x86_64.

It looks like bind-9.9.4 iterates over available interfaces on startup.  Listening to 0.0.0.0 is not an option[0].  This changes in bind-9.10 [1].

Waiting for network-online.target[2] seems to work:

$ diff -u /usr/lib/systemd/system/named-chroot.service /etc/systemd/system/named-chroot.service
--- /usr/lib/systemd/system/named-chroot.service        2016-01-18 13:57:35.000000000 +0100
+++ /etc/systemd/system/named-chroot.service    2016-02-23 10:12:25.276687016 +0100
@@ -7,7 +7,7 @@
 Wants=nss-lookup.target
 Requires=named-chroot-setup.service
 Before=nss-lookup.target
-After=network.target
+After=network-online.target
 After=named-chroot-setup.service

 [Service]

[0] http://permalink.gmane.org/gmane.network.dns.bind.user/31854

[1] https://kb.isc.org/article/AA-01153/0/BIND-9.10.0-Release-Notes.html

 * On operating systems that support routing sockets, including Mac OSX, *BSD and Linux, network interfaces are re-scanned automatically whenever they change.  Use "automatic-interface-scan no;" to disable this feature. [RT #23027]
 * Added "rndc scan" to trigger an interface scan manually. [RT #23027]

  https://source.isc.org/cgi-bin/gitweb.cgi?p=bind9.git;a=blob;f=bin/named/interfacemgr.c;h=ad8046421250663e332daf7f11c003300cbaa3c8;hb=HEAD#l929

[2] https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/

"network.target has very little meaning during start-up. It only indicates that the network management stack is up after it has been reached."

"network-online.target is a target that actively waits until the nework is "up", where the definition of "up" is defined by the network management software. Usually it indicates a configured, routable IP address of some kind."

*** Bug 1301912 has been marked as a duplicate of this bug. ***

*** Bug 1278015 has been marked as a duplicate of this bug. ***

Hello.

BIND 9.9.4 binds to any addresses, which are available during the server startup and are allowed by the configuration. Versions previous to 9.10.x are not able to automatically detect, that there is a new address on some interface or that some address disappeared (as noted in comment #4).

This is technically solved by the dispatcher script when using NetworkManger. It reloads the server on any network configuration change and thus makes it to bind to new addresses or stop listening on addresses which disappeared. Note that the dispatcher script causes BIND to do complete reload, which is unnecessary and may take long time in case your zone file is really big.

This naturally does not work when you uninstall NetworkManger, as the dispatcher script is not run.

However BIND 9.9.4 is scanning interfaces regularly, but the interval is by default set to 60 minutes. This can be changed using "interface-interval" option in the general "options" section. I don't think that changing the default is really a solution.

Note, that if you wait 60 minutes, BIND will eventually bind to all addresses ;)

I don't think that changing the "network.target" to "network-online.target" is either a good solution. The reason is that for some system services which depend on DNS, it may be too late if BIND starts only after the network is fully configured.

I propose to:
- keep the "After=network.target" in all named service files
- add the automatic scan functionality from BIND 9.10.x to BIND in RHEL-7
- remove the NetworkManager dispatcher script from the package

I backported the needed changes and everything seems to work fine. IOW BIND binds to all addresses even if NetworkManager is not installed.

(In reply to Carl Henrik Lunde from comment #4)

> "network.target has very little meaning during start-up. It only indicates
> that the network management stack is up after it has been reached."

I wouldn't agree with the first part of this statement, given the fact that there is quite a number of servers still (maybe on purpose) running without NetworkManager.

The problem is in fact with systemd which changed its way of activating network.target and network-online.target somewhere between version 208 and 219 and I've found that totally by accident... I restarted the server after 6 months and Postfix didn't come up. I checked the journal and found out that it was started but died afterwards because the system still haven't got an IP on which Postfix was configured. And I knew that this was not happening on RHEL 7.1.

With version 208 both targets were still working with network.service as well as with NM but with 219 network.target is NM-dependent only whereas network-online.target is still network.service dependent! But nearly all network services still have "After=network.target" in their configuration. So if you run your server without NM (not even installed) then your system ends up without network.target being activated at all and result is network services being started as soon as multi-user.target kicks in and possibly before network interfaces are brought up.

Whether they did that on purpose or was there some greater good in such a decision, I have no idea, but surely such an update messed up a lot of servers. 

So there are two possibilities if you would still like to bring up your network via network.service only... one is reconfiguring all your network services' systemd configs to "After=network-online.target" or two, install NM but leave it disabled. In 2nd case, you would get an info during the boot process that says "systemd[1]: Dependency failed for Network Manager Wait Online." and that's it.

Well, maybe I'm missing something but as I see it right now, the current implementation of systemd in RHEL 7.2 is broken. If you would still like to use network.service only, you would end up without network.target, but if you would like to switch completely to NM (without network.service), you would end up without network-online.target. This was not the case in RHEL 7.1 where you could run either network.service or NM alone or both at the same time and both network.target and network-online.target were active in each and every situation.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2233.html