Bug 1412570 - After=network.target is not happening when NetworkManager is masked/disabled
Summary: After=network.target is not happening when NetworkManager is masked/disabled
Keywords:
Status: NEW
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: systemd
Version: 7.3
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: rc
: ---
Assignee: systemd-maint
QA Contact: qe-baseos-daemons
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-12 10:29 UTC by Patrick
Modified: 2020-04-14 07:27 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)
Example systemd unit file (163 bytes, text/plain)
2017-01-12 10:29 UTC, Patrick
no flags Details
boot messages - unbound binds before interfaces are up (2.84 KB, text/plain)
2017-01-12 10:31 UTC, Patrick
no flags Details
Boot log systemd when network.target is symlinked to multi-user.target.wants (3.74 KB, text/plain)
2017-01-12 13:43 UTC, Patrick
no flags Details

Description Patrick 2017-01-12 10:29:40 UTC
Created attachment 1239845 [details]
Example systemd unit file

Description of problem:
Several services that start early and need to bind to Interfaces/IP-Addresses enter the failed state after boot because they try to bind to IP-Adressess which arent yet bound to an interface. 
This happens when NetworkManager is disabled or masked and the unit file of the failed service contains After=network.target.
In fact it looks like "After=network.target" is ignored.

Version-Release number of selected component (if applicable):
Every version of software that binds to IP/IF early, like unbound, postfix, a simple netcat listening port and so on...

How reproducible:
On every reboot.

Steps to Reproduce:
1. Disable/mask NetworkManager
2. Configure static IP-Address for an interface
3. Enable a systemd unit file containing "After=network.target" and "ExecStart = /usr/bin/ncat -4 -k -l 4711" 
4. Reboot

Actual results:
The service wont start at boot, check with "systemctl status" or "netstat".

Expected results:
Service starts on boot and can bind to IP-Address on available interface, eg. "After=network.target".

Additional info:
I have disable NetworkManager on a variety of RHEL7 and CentOs7 systems. All seem to behave the same if the software wants to start "After=network.target" and needs to bind to an IP-Address and port.

Comment 1 Patrick 2017-01-12 10:31:00 UTC
Created attachment 1239846 [details]
boot messages - unbound binds before interfaces are up

Comment 2 Lukáš Nykrýn 2017-01-12 10:54:14 UTC
Hmm I have a suspicion that nothing pulls the network.target to transaction in that case. Can you try add Wants=network.target to your service?

Comment 3 Patrick 2017-01-12 11:15:17 UTC
(In reply to Lukáš Nykrýn from comment #2)
> Hmm I have a suspicion that nothing pulls the network.target to transaction
> in that case. Can you try add Wants=network.target to your service?

Well that flawlessly worked. 
I also tried this with "unbound" which per rpm installed package the systemd unit file reads:
Wants=nss-lookup.target

Now after adding network.target:
Wants=nss-lookup.target network.target

The boot successfully completes and network interfaces are brought up before unbound ist startet.

So this is a fast fix for the problem occuring, but who delivers the problem? Is it systemd or the unbound shipped unit file? Maybe im in the wrong place here with my bug report. When i start looking at similar software i also see that network.target ist missing from the "Wants" parameter in their unit files.

Thanks!

Comment 5 Lukáš Nykrýn 2017-01-12 13:12:40 UTC
Well the problem is that the service that provides network should pull the network.target in the transaction. But the network is an initscripts and so it can't do that. We need to think about a solution either in systemd in sysv-generator or in initscripts.

Comment 6 Patrick 2017-01-12 13:34:37 UTC
(In reply to Lukáš Nykrýn from comment #5)
> We need to think about a solution either in systemd in
> sysv-generator or in initscripts.

Well thank you for that. 

I did find an quick fix that does not need touching all the unit files for software that is affected.

In "/etc/system/systemd/multi-user.target.wants/" we have several symlinks for dependencies. Also there is a symlink for "NetworkManager.service".

But no "network.target" so i have added a symlink for it. 

ln -s /usr/lib/systemd/system/network.target /etc/systemd/system/multi-user.target.wants/network.target

Now the programs i have tried to start on boot now all start correctly after network has been initialized and of course with "NetworkManager" still masked/disabled.

I dont know if we can take this as a fix for that problem, although i dont know how this will behave if i reactivate NetworkManager.service now. 
I will do a few tests to see the outcome of this. But for now it looks good.

Maybe i should remove the "NetworkManager" symlink in "/etc/system/systemd/multi-user.target.wants/" as in this szenario it shouldn't be needed anymore.

Thanks alot!

Comment 7 Patrick 2017-01-12 13:43:06 UTC
Created attachment 1239937 [details]
Boot log systemd when network.target is symlinked to multi-user.target.wants

Comment 8 Patrick 2017-01-12 15:23:29 UTC
I have now found out that its not a good idea to use the "Special Passive System Unit" "network.target" in any "wants". According to https://www.freedesktop.org/software/systemd/man/systemd.special.html and https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/
Well I'm looking forward to a solid solution.

Comment 9 Patrick 2017-01-13 12:31:50 UTC
I have removed the symlink as described above now.

The final solution that solved this problem for me was to install the package "sytemd-networkd" and enable it on boot time.
systemctl enable systemd-networkd

It's unit file pulls in network.target actively:
Wants=network.target

Well as i read this is not best practice, but this solution works out of the box and can be reproduced on productive machines, also a system upgrade wont matter here i think.

If someone comes up with an other solution im happy with that too (if implementet) so far if you disable NetworkManager install systemd-networkd and enable it instead.

Cheers.

Comment 10 Patrick 2017-01-17 10:19:00 UTC
To clear things up, this looks like an feature not a bug, NetworkManager is supported by RedHat. Thanks to this reply here:
https://lists.centos.org/pipermail/centos/2017-January/162866.html


Note You need to log in before you can comment on or make changes to this bug.