RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1402479 - Network services (httpd) couldn't start during boot time if NetworkManager.service is disabled
Summary: Network services (httpd) couldn't start during boot time if NetworkManager.se...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: httpd
Version: 7.3
Hardware: Unspecified
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Luboš Uhliarik
QA Contact: BaseOS QE - Apps
URL:
Whiteboard:
Depends On:
Blocks: 1420851
TreeView+ depends on / blocked
 
Reported: 2016-12-07 16:01 UTC by dbodnarc
Modified: 2020-05-14 15:27 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
RHEL7.3
Last Closed: 2017-05-09 15:08:18 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2815741 0 None None None 2017-01-20 09:01:38 UTC

Description dbodnarc 2016-12-07 16:01:25 UTC
Description of problem:
Network services such as httpd.service, sshd.service with the listening IP address set in their configuration couldn't start during boot time if NetworkManager.service is disabled.

Version-Release number of selected component (if applicable):
NetworkManager-1.4.0-12.el7.x86_64

How reproducible:
# systemctl disable NetworkManager.service
# systemctl disable NetworkManager-wait-online.service

> set exact ip address in service's (i.e. httpd, sshd) configuration files:
/etc/httpd/conf/httpd.conf:
Listen xx.xx.xx.xx:80

Reboot server.

Steps to Reproduce:
1) RHEL7.3 (Linux fastvm-r7-3-73 3.10.0-514.el7.x86_64)
# systemctl -a  |grep "Network Manager"
  NetworkManager-wait-online.service         loaded    inactive dead      Network Manager Wait Online
  NetworkManager.service                     loaded    inactive dead      Network Manager

set exact IP address for listening:
> httpd-2.4.6-45.el7.x86_64

a) /etc/httpd/conf/httpd.conf ::
   Listen 80 >> doesn't cause any issues during boot time 
b) /etc/httpd/conf/httpd.conf ::
   Listen 192.168.22.73:80  >> causes the issue during boot: 
//   httpd[822]: (99)Cannot assign requested address: AH00072: make_sock: could not bind to address 192.168.22.73:80
//   httpd[822]: no listening sockets available, shutting down

Actual results:
//   httpd[822]: (99)Cannot assign requested address: AH00072: make_sock: could not bind to address 192.168.22.73:80

Expected results:
Network service units (like httpd, sshd, ... etc) should run after network-online.target

Additional info:
The mentioned issue comes with the new package NetworkManager-1.4.0-12.el7.x86_64
(after update of NetworkManager-1.0.6-30.el7_2.x86_64 to NetworkManager-1.4.0-12.el7.x86_64)

it comes from the changes applied to the following file:
# rpm -qf /usr/lib/systemd/system/NetworkManager-wait-online.service
NetworkManager-1.4.0-12.el7.x86_64

Here are the differences /usr/lib/systemd/system/NetworkManager-wait-online.service:
NetworkManager-1.4.0-12.el7.x86_64                              NetworkManager-1.0.6-30.el7_2.x86_64
> NEW <                                                         > OLD <
------------------------------------------------------------------------------------------------------------------------
[Unit]								[Unit]
Description=Network Manager Wait Online				Description=Network Manager Wait Online
Documentation=man:nm-online(1)				      <
Requisite=NetworkManager.service				Requisite=NetworkManager.service
After=NetworkManager.service					After=NetworkManager.service
Before=network-online.target				      |	Wants=network.target
							      >	Before=network.target network-online.target

[Service]							[Service]
Type=oneshot							Type=oneshot
ExecStart=/usr/bin/nm-online -s -q --timeout=30			ExecStart=/usr/bin/nm-online -s -q --timeout=30
RemainAfterExit=yes					      <

[Install]							[Install]
WantedBy=network-online.target				      |	WantedBy=multi-user.target
------------------------------------------------------------------------------------------------------------------------

Previously (with previous version of NetworkManager) even with NetworkManager.service disabled, network.target was called from NetworkManager-wait-online.service, so httpd.service (with the listening IP address set in its configuration file) was able to run properly during boot time after network.target reached. (It is not a case when NetworkManager.service is enabled, which means network.target will be reached)

So if you disable NetworkManager.service and will have any other network services (like sshd, httpd, etc) with the ip address specified in their configuration - then we'll get in trouble and the services wouldn't start during boot time

Comment 1 dbodnarc 2016-12-07 16:06:07 UTC
Setting "After=network-online.target" for httpd.service will solve the mentioned issue with NetworkManager.service disabled.

PS: The  issue can be fixed from network packages (such as httpd, sshd, etc) side as well. Please let me know / or update affected components.

Comment 3 Beniamino Galvani 2016-12-09 22:20:25 UTC
By default the httpd unit has "After=network.target", which means that
the service is started only after the network target is reached (if
enabled). Actually, network.target has very little meaning during
start-up and is mainly used to ensure a proper ordering during
shutdown [1]. Usually Apache can be started without addresses
configured on the interfaces and so "After=network.target" is enough.

However, when configured to bind to a specific address, Apache requires
the address to be already configured and thus that network connections
are up. In this case, services should order themselves after the
network-online target, which is triggered after network init-scripts
terminate (or after NetworkManager, when it is enabled).

The default setting is to not let services wait for full connectivity,
because in most cases this is not necessary, but this can be easily
modified using a systemd drop-in:

$ cat > /etc/systemd/system/httpd.service.d/override.conf << EOF
[Unit]
After=network-online.target
EOF


The justification for changes in NetworkManager-wait-online unit
mentioned in comment 0 can be found in bug [2] and I don't think there
is anything to change in that regard.

I'm reassigning this to httpd, so that the package maintainer can confirm
whether the manual update to the unit file is necessary when Apache is
configured to bind to a specific address.


[1] https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/
[2] https://bugzilla.gnome.org/show_bug.cgi?id=746039

Comment 4 dbodnarc 2016-12-13 10:13:45 UTC
Lubos, could you please confirm if the mentioned issue is planned to be fixed for httpd component? 

If so, I'll open bugs against some other network services which falls in the same issue (like sshd).

Comment 7 Pavel Šimerda (pavlix) 2017-01-02 12:07:49 UTC
(In reply to dbodnarc from comment #1)
> Setting "After=network-online.target" for httpd.service will solve the
> mentioned issue with NetworkManager.service disabled.

While "After=network-online.target" is the right setting for all services that aren't capable of starting before the network is configured, I don't see how exactly is it related to the situation with disabled NetworkManager.

Comment 8 Pavel Šimerda (pavlix) 2017-01-02 12:16:00 UTC
Plus After=network-online.target is not enough, Wants=network-online.target is needed as well.

http://unix.stackexchange.com/questions/126009/cause-a-script-to-execute-after-networking-has-started/126146#126146

Comment 9 Beniamino Galvani 2017-01-02 12:50:14 UTC
(In reply to Pavel Šimerda (pavlix) from comment #7)
> (In reply to dbodnarc from comment #1)
> > Setting "After=network-online.target" for httpd.service will solve the
> > mentioned issue with NetworkManager.service disabled.
> 
> While "After=network-online.target" is the right setting for all services
> that aren't capable of starting before the network is configured, I don't
> see how exactly is it related to the situation with disabled NetworkManager.

I think it's because the network.service already pulls in and orders
itself before the network-online target:

 [root@rhel-7-3 ~]# systemctl cat network.service | grep network-online
 Before=network-online.target
 Wants=network-online.target

and thus any service with "After=network-online.target" will be
started after init scripts have already configured the interfaces.

Comment 10 dbodnarc 2017-01-02 16:37:51 UTC
If I understood the behavior right, when NetworkManager.service is disabled in rhel73, network.target became inactive (after reboot):
------------------------------------------------------------------------
# systemctl -a | grep network.target
> network.target    loaded    inactive dead      Network
------------------------------------------------------------------------

And here is httpd.service systemd config part also:
------------------------------------------------------------------------
# systemctl cat httpd.service 
# /usr/lib/systemd/system/httpd.service
[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target
...
------------------------------------------------------------------------

So I think the reason why httpd.service fails to start during boot (with the listening IP address specified) it is network.target inactive.

As far as I understood it is not a bug, but it's a feature.
As for me, the fact that setting httpd service to listen on a IP specific address requires systemd override.conf to be created - was not obvious at all and this wasn't not noted in a man page as well. 

That's why it might be not obvious for other people as well and may require at least some tip in a httpd man page.

Meanwhile I've created solution article regarding this case: 
- https://access.redhat.com/solutions/2815741

Please confirm if I understood everything right and nothing is planned to be fixed in a listed components (httpd/NM).
Thank you in advance

Comment 11 dbodnarc 2017-04-14 08:39:32 UTC
I expect the mentioned behavior to be fixed within recently added BZ#1438749:
> https://bugzilla.redhat.com/show_bug.cgi?id=1438749

Comment 12 Joe Orton 2017-05-09 15:08:18 UTC
We've no intention to make any changes here.  For the minority of configurations which have a dependency on a specific network, a systemd override file should be used as described in Solution 2 of https://access.redhat.com/solutions/2815741


Note You need to log in before you can comment on or make changes to this bug.