Created attachment 1239845 [details]
Example systemd unit file
Description of problem:
Several services that start early and need to bind to Interfaces/IP-Addresses enter the failed state after boot because they try to bind to IP-Adressess which arent yet bound to an interface.
This happens when NetworkManager is disabled or masked and the unit file of the failed service contains After=network.target.
In fact it looks like "After=network.target" is ignored.
Version-Release number of selected component (if applicable):
Every version of software that binds to IP/IF early, like unbound, postfix, a simple netcat listening port and so on...
On every reboot.
Steps to Reproduce:
1. Disable/mask NetworkManager
2. Configure static IP-Address for an interface
3. Enable a systemd unit file containing "After=network.target" and "ExecStart = /usr/bin/ncat -4 -k -l 4711"
The service wont start at boot, check with "systemctl status" or "netstat".
Service starts on boot and can bind to IP-Address on available interface, eg. "After=network.target".
I have disable NetworkManager on a variety of RHEL7 and CentOs7 systems. All seem to behave the same if the software wants to start "After=network.target" and needs to bind to an IP-Address and port.
Created attachment 1239846 [details]
boot messages - unbound binds before interfaces are up
Hmm I have a suspicion that nothing pulls the network.target to transaction in that case. Can you try add Wants=network.target to your service?
(In reply to Lukáš Nykrýn from comment #2)
> Hmm I have a suspicion that nothing pulls the network.target to transaction
> in that case. Can you try add Wants=network.target to your service?
Well that flawlessly worked.
I also tried this with "unbound" which per rpm installed package the systemd unit file reads:
Now after adding network.target:
The boot successfully completes and network interfaces are brought up before unbound ist startet.
So this is a fast fix for the problem occuring, but who delivers the problem? Is it systemd or the unbound shipped unit file? Maybe im in the wrong place here with my bug report. When i start looking at similar software i also see that network.target ist missing from the "Wants" parameter in their unit files.
Well the problem is that the service that provides network should pull the network.target in the transaction. But the network is an initscripts and so it can't do that. We need to think about a solution either in systemd in sysv-generator or in initscripts.
(In reply to Lukáš Nykrýn from comment #5)
> We need to think about a solution either in systemd in
> sysv-generator or in initscripts.
Well thank you for that.
I did find an quick fix that does not need touching all the unit files for software that is affected.
In "/etc/system/systemd/multi-user.target.wants/" we have several symlinks for dependencies. Also there is a symlink for "NetworkManager.service".
But no "network.target" so i have added a symlink for it.
ln -s /usr/lib/systemd/system/network.target /etc/systemd/system/multi-user.target.wants/network.target
Now the programs i have tried to start on boot now all start correctly after network has been initialized and of course with "NetworkManager" still masked/disabled.
I dont know if we can take this as a fix for that problem, although i dont know how this will behave if i reactivate NetworkManager.service now.
I will do a few tests to see the outcome of this. But for now it looks good.
Maybe i should remove the "NetworkManager" symlink in "/etc/system/systemd/multi-user.target.wants/" as in this szenario it shouldn't be needed anymore.
Created attachment 1239937 [details]
Boot log systemd when network.target is symlinked to multi-user.target.wants
I have now found out that its not a good idea to use the "Special Passive System Unit" "network.target" in any "wants". According to https://www.freedesktop.org/software/systemd/man/systemd.special.html and https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/
Well I'm looking forward to a solid solution.
I have removed the symlink as described above now.
The final solution that solved this problem for me was to install the package "sytemd-networkd" and enable it on boot time.
systemctl enable systemd-networkd
It's unit file pulls in network.target actively:
Well as i read this is not best practice, but this solution works out of the box and can be reproduced on productive machines, also a system upgrade wont matter here i think.
If someone comes up with an other solution im happy with that too (if implementet) so far if you disable NetworkManager install systemd-networkd and enable it instead.
To clear things up, this looks like an feature not a bug, NetworkManager is supported by RedHat. Thanks to this reply here: