Bug 1402479

Summary: Network services (httpd) couldn't start during boot time if NetworkManager.service is disabled
Product: Red Hat Enterprise Linux 7 Reporter: dbodnarc
Component: httpdAssignee: Luboš Uhliarik <luhliari>
Status: CLOSED NOTABUG QA Contact: BaseOS QE - Apps <qe-baseos-apps>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.3CC: atragler, bgalvani, dmasirka, jorton, luhliari, rkhan, sukulkar
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
RHEL7.3
Last Closed: 2017-05-09 15:08:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1420851    

Description dbodnarc 2016-12-07 16:01:25 UTC
Description of problem:
Network services such as httpd.service, sshd.service with the listening IP address set in their configuration couldn't start during boot time if NetworkManager.service is disabled.

Version-Release number of selected component (if applicable):
NetworkManager-1.4.0-12.el7.x86_64

How reproducible:
# systemctl disable NetworkManager.service
# systemctl disable NetworkManager-wait-online.service

> set exact ip address in service's (i.e. httpd, sshd) configuration files:
/etc/httpd/conf/httpd.conf:
Listen xx.xx.xx.xx:80

Reboot server.

Steps to Reproduce:
1) RHEL7.3 (Linux fastvm-r7-3-73 3.10.0-514.el7.x86_64)
# systemctl -a  |grep "Network Manager"
  NetworkManager-wait-online.service         loaded    inactive dead      Network Manager Wait Online
  NetworkManager.service                     loaded    inactive dead      Network Manager

set exact IP address for listening:
> httpd-2.4.6-45.el7.x86_64

a) /etc/httpd/conf/httpd.conf ::
   Listen 80 >> doesn't cause any issues during boot time 
b) /etc/httpd/conf/httpd.conf ::
   Listen 192.168.22.73:80  >> causes the issue during boot: 
//   httpd[822]: (99)Cannot assign requested address: AH00072: make_sock: could not bind to address 192.168.22.73:80
//   httpd[822]: no listening sockets available, shutting down

Actual results:
//   httpd[822]: (99)Cannot assign requested address: AH00072: make_sock: could not bind to address 192.168.22.73:80

Expected results:
Network service units (like httpd, sshd, ... etc) should run after network-online.target

Additional info:
The mentioned issue comes with the new package NetworkManager-1.4.0-12.el7.x86_64
(after update of NetworkManager-1.0.6-30.el7_2.x86_64 to NetworkManager-1.4.0-12.el7.x86_64)

it comes from the changes applied to the following file:
# rpm -qf /usr/lib/systemd/system/NetworkManager-wait-online.service
NetworkManager-1.4.0-12.el7.x86_64

Here are the differences /usr/lib/systemd/system/NetworkManager-wait-online.service:
NetworkManager-1.4.0-12.el7.x86_64                              NetworkManager-1.0.6-30.el7_2.x86_64
> NEW <                                                         > OLD <
------------------------------------------------------------------------------------------------------------------------
[Unit]								[Unit]
Description=Network Manager Wait Online				Description=Network Manager Wait Online
Documentation=man:nm-online(1)				      <
Requisite=NetworkManager.service				Requisite=NetworkManager.service
After=NetworkManager.service					After=NetworkManager.service
Before=network-online.target				      |	Wants=network.target
							      >	Before=network.target network-online.target

[Service]							[Service]
Type=oneshot							Type=oneshot
ExecStart=/usr/bin/nm-online -s -q --timeout=30			ExecStart=/usr/bin/nm-online -s -q --timeout=30
RemainAfterExit=yes					      <

[Install]							[Install]
WantedBy=network-online.target				      |	WantedBy=multi-user.target
------------------------------------------------------------------------------------------------------------------------

Previously (with previous version of NetworkManager) even with NetworkManager.service disabled, network.target was called from NetworkManager-wait-online.service, so httpd.service (with the listening IP address set in its configuration file) was able to run properly during boot time after network.target reached. (It is not a case when NetworkManager.service is enabled, which means network.target will be reached)

So if you disable NetworkManager.service and will have any other network services (like sshd, httpd, etc) with the ip address specified in their configuration - then we'll get in trouble and the services wouldn't start during boot time

Comment 1 dbodnarc 2016-12-07 16:06:07 UTC
Setting "After=network-online.target" for httpd.service will solve the mentioned issue with NetworkManager.service disabled.

PS: The  issue can be fixed from network packages (such as httpd, sshd, etc) side as well. Please let me know / or update affected components.

Comment 3 Beniamino Galvani 2016-12-09 22:20:25 UTC
By default the httpd unit has "After=network.target", which means that
the service is started only after the network target is reached (if
enabled). Actually, network.target has very little meaning during
start-up and is mainly used to ensure a proper ordering during
shutdown [1]. Usually Apache can be started without addresses
configured on the interfaces and so "After=network.target" is enough.

However, when configured to bind to a specific address, Apache requires
the address to be already configured and thus that network connections
are up. In this case, services should order themselves after the
network-online target, which is triggered after network init-scripts
terminate (or after NetworkManager, when it is enabled).

The default setting is to not let services wait for full connectivity,
because in most cases this is not necessary, but this can be easily
modified using a systemd drop-in:

$ cat > /etc/systemd/system/httpd.service.d/override.conf << EOF
[Unit]
After=network-online.target
EOF


The justification for changes in NetworkManager-wait-online unit
mentioned in comment 0 can be found in bug [2] and I don't think there
is anything to change in that regard.

I'm reassigning this to httpd, so that the package maintainer can confirm
whether the manual update to the unit file is necessary when Apache is
configured to bind to a specific address.


[1] https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/
[2] https://bugzilla.gnome.org/show_bug.cgi?id=746039

Comment 4 dbodnarc 2016-12-13 10:13:45 UTC
Lubos, could you please confirm if the mentioned issue is planned to be fixed for httpd component? 

If so, I'll open bugs against some other network services which falls in the same issue (like sshd).

Comment 7 Pavel Šimerda (pavlix) 2017-01-02 12:07:49 UTC
(In reply to dbodnarc from comment #1)
> Setting "After=network-online.target" for httpd.service will solve the
> mentioned issue with NetworkManager.service disabled.

While "After=network-online.target" is the right setting for all services that aren't capable of starting before the network is configured, I don't see how exactly is it related to the situation with disabled NetworkManager.

Comment 8 Pavel Šimerda (pavlix) 2017-01-02 12:16:00 UTC
Plus After=network-online.target is not enough, Wants=network-online.target is needed as well.

http://unix.stackexchange.com/questions/126009/cause-a-script-to-execute-after-networking-has-started/126146#126146

Comment 9 Beniamino Galvani 2017-01-02 12:50:14 UTC
(In reply to Pavel Šimerda (pavlix) from comment #7)
> (In reply to dbodnarc from comment #1)
> > Setting "After=network-online.target" for httpd.service will solve the
> > mentioned issue with NetworkManager.service disabled.
> 
> While "After=network-online.target" is the right setting for all services
> that aren't capable of starting before the network is configured, I don't
> see how exactly is it related to the situation with disabled NetworkManager.

I think it's because the network.service already pulls in and orders
itself before the network-online target:

 [root@rhel-7-3 ~]# systemctl cat network.service | grep network-online
 Before=network-online.target
 Wants=network-online.target

and thus any service with "After=network-online.target" will be
started after init scripts have already configured the interfaces.

Comment 10 dbodnarc 2017-01-02 16:37:51 UTC
If I understood the behavior right, when NetworkManager.service is disabled in rhel73, network.target became inactive (after reboot):
------------------------------------------------------------------------
# systemctl -a | grep network.target
> network.target    loaded    inactive dead      Network
------------------------------------------------------------------------

And here is httpd.service systemd config part also:
------------------------------------------------------------------------
# systemctl cat httpd.service 
# /usr/lib/systemd/system/httpd.service
[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target
...
------------------------------------------------------------------------

So I think the reason why httpd.service fails to start during boot (with the listening IP address specified) it is network.target inactive.

As far as I understood it is not a bug, but it's a feature.
As for me, the fact that setting httpd service to listen on a IP specific address requires systemd override.conf to be created - was not obvious at all and this wasn't not noted in a man page as well. 

That's why it might be not obvious for other people as well and may require at least some tip in a httpd man page.

Meanwhile I've created solution article regarding this case: 
- https://access.redhat.com/solutions/2815741

Please confirm if I understood everything right and nothing is planned to be fixed in a listed components (httpd/NM).
Thank you in advance

Comment 11 dbodnarc 2017-04-14 08:39:32 UTC
I expect the mentioned behavior to be fixed within recently added BZ#1438749:
> https://bugzilla.redhat.com/show_bug.cgi?id=1438749

Comment 12 Joe Orton 2017-05-09 15:08:18 UTC
We've no intention to make any changes here.  For the minority of configurations which have a dependency on a specific network, a systemd override file should be used as described in Solution 2 of https://access.redhat.com/solutions/2815741