Bug 1412570

Summary: After=network.target is not happening when NetworkManager is masked/disabled
Product: Red Hat Enterprise Linux 7 Reporter: Patrick <patrick.oberdorfer>
Component: systemdAssignee: systemd-maint
Status: CLOSED WONTFIX QA Contact: qe-baseos-daemons
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.3CC: bfinger, systemd-maint-list
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-01-15 07:30:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Example systemd unit file
none
boot messages - unbound binds before interfaces are up
none
Boot log systemd when network.target is symlinked to multi-user.target.wants none

Description Patrick 2017-01-12 10:29:40 UTC
Created attachment 1239845 [details]
Example systemd unit file

Description of problem:
Several services that start early and need to bind to Interfaces/IP-Addresses enter the failed state after boot because they try to bind to IP-Adressess which arent yet bound to an interface. 
This happens when NetworkManager is disabled or masked and the unit file of the failed service contains After=network.target.
In fact it looks like "After=network.target" is ignored.

Version-Release number of selected component (if applicable):
Every version of software that binds to IP/IF early, like unbound, postfix, a simple netcat listening port and so on...

How reproducible:
On every reboot.

Steps to Reproduce:
1. Disable/mask NetworkManager
2. Configure static IP-Address for an interface
3. Enable a systemd unit file containing "After=network.target" and "ExecStart = /usr/bin/ncat -4 -k -l 4711" 
4. Reboot

Actual results:
The service wont start at boot, check with "systemctl status" or "netstat".

Expected results:
Service starts on boot and can bind to IP-Address on available interface, eg. "After=network.target".

Additional info:
I have disable NetworkManager on a variety of RHEL7 and CentOs7 systems. All seem to behave the same if the software wants to start "After=network.target" and needs to bind to an IP-Address and port.

Comment 1 Patrick 2017-01-12 10:31:00 UTC
Created attachment 1239846 [details]
boot messages - unbound binds before interfaces are up

Comment 2 Lukáš Nykrýn 2017-01-12 10:54:14 UTC
Hmm I have a suspicion that nothing pulls the network.target to transaction in that case. Can you try add Wants=network.target to your service?

Comment 3 Patrick 2017-01-12 11:15:17 UTC
(In reply to Lukáš Nykrýn from comment #2)
> Hmm I have a suspicion that nothing pulls the network.target to transaction
> in that case. Can you try add Wants=network.target to your service?

Well that flawlessly worked. 
I also tried this with "unbound" which per rpm installed package the systemd unit file reads:
Wants=nss-lookup.target

Now after adding network.target:
Wants=nss-lookup.target network.target

The boot successfully completes and network interfaces are brought up before unbound ist startet.

So this is a fast fix for the problem occuring, but who delivers the problem? Is it systemd or the unbound shipped unit file? Maybe im in the wrong place here with my bug report. When i start looking at similar software i also see that network.target ist missing from the "Wants" parameter in their unit files.

Thanks!

Comment 5 Lukáš Nykrýn 2017-01-12 13:12:40 UTC
Well the problem is that the service that provides network should pull the network.target in the transaction. But the network is an initscripts and so it can't do that. We need to think about a solution either in systemd in sysv-generator or in initscripts.

Comment 6 Patrick 2017-01-12 13:34:37 UTC
(In reply to Lukáš Nykrýn from comment #5)
> We need to think about a solution either in systemd in
> sysv-generator or in initscripts.

Well thank you for that. 

I did find an quick fix that does not need touching all the unit files for software that is affected.

In "/etc/system/systemd/multi-user.target.wants/" we have several symlinks for dependencies. Also there is a symlink for "NetworkManager.service".

But no "network.target" so i have added a symlink for it. 

ln -s /usr/lib/systemd/system/network.target /etc/systemd/system/multi-user.target.wants/network.target

Now the programs i have tried to start on boot now all start correctly after network has been initialized and of course with "NetworkManager" still masked/disabled.

I dont know if we can take this as a fix for that problem, although i dont know how this will behave if i reactivate NetworkManager.service now. 
I will do a few tests to see the outcome of this. But for now it looks good.

Maybe i should remove the "NetworkManager" symlink in "/etc/system/systemd/multi-user.target.wants/" as in this szenario it shouldn't be needed anymore.

Thanks alot!

Comment 7 Patrick 2017-01-12 13:43:06 UTC
Created attachment 1239937 [details]
Boot log systemd when network.target is symlinked to multi-user.target.wants

Comment 8 Patrick 2017-01-12 15:23:29 UTC
I have now found out that its not a good idea to use the "Special Passive System Unit" "network.target" in any "wants". According to https://www.freedesktop.org/software/systemd/man/systemd.special.html and https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/
Well I'm looking forward to a solid solution.

Comment 9 Patrick 2017-01-13 12:31:50 UTC
I have removed the symlink as described above now.

The final solution that solved this problem for me was to install the package "sytemd-networkd" and enable it on boot time.
systemctl enable systemd-networkd

It's unit file pulls in network.target actively:
Wants=network.target

Well as i read this is not best practice, but this solution works out of the box and can be reproduced on productive machines, also a system upgrade wont matter here i think.

If someone comes up with an other solution im happy with that too (if implementet) so far if you disable NetworkManager install systemd-networkd and enable it instead.

Cheers.

Comment 10 Patrick 2017-01-17 10:19:00 UTC
To clear things up, this looks like an feature not a bug, NetworkManager is supported by RedHat. Thanks to this reply here:
https://lists.centos.org/pipermail/centos/2017-January/162866.html

Comment 13 RHEL Program Management 2021-01-15 07:30:07 UTC
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.