Description of problem: Apache HTTPD service fails to start during startup. Manual start with systemctl start httpd.service works fine. Version-Release number of selected component (if applicable): systemd.x86_64 243.8-1.fc31 How reproducible: Always Steps to Reproduce: 1. Boot the system. 2. Failure error appears on console during startup. 3. systemctl status httpd shows the failure and error. Actual results: This problem appears to be an issue with systemd. The httpd service is trying to start before the IP address has been assigned to the NIC so that httpd cannot bind to that IP address. Error message: [root@yorktown ~]# systemctl status httpd ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Thu 2020-04-16 11:54:37 EDT; 15min ago Docs: man:httpd.service(8) Process: 1101 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE) Main PID: 1101 (code=exited, status=1/FAILURE) Status: "Reading configuration..." CPU: 60ms Apr 16 11:54:35 yorktown.both.org systemd[1]: Starting The Apache HTTP Server... Apr 16 11:54:37 yorktown.both.org httpd[1101]: (99)Cannot assign requested address: AH00072: make_sock: could not bind to address 192.168.0.52:80 Apr 16 11:54:37 yorktown.both.org httpd[1101]: no listening sockets available, shutting down Apr 16 11:54:37 yorktown.both.org httpd[1101]: AH00015: Unable to open logs Apr 16 11:54:37 yorktown.both.org systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE Apr 16 11:54:37 yorktown.both.org systemd[1]: httpd.service: Failed with result 'exit-code'. Apr 16 11:54:37 yorktown.both.org systemd[1]: Failed to start The Apache HTTP Server. [root@yorktown ~]# Expected results: Web server should start without error Additional info: I have circumvented the problem using /etc/systemd/system/httpd.service.d/override.conf with the following contents to provide a 30 second delay of httpd service start during startup. # Trying to delay the startup of httpd so that the network is # fully up and running so that httpd can bind to the correct # IP address # # By David Both, 2020-04-16 [Service] ExecStartPre=/bin/sleep 30 Fedora 31 with all updates installed as of 2020-04-15. ####################################################################### # MOTD for Sat 18 Apr 2020 03:17:33 AM EDT # HOST NAME: yorktown.both.org # Machine Type: physical machine. # Host architecture: X86_64 # Motherboard Mfr: Intel Corporation # Motherboard Model: DZ77SL-50K # Motherboard Serial: BTSL215000SR #---------------------------------------------------------------------- # CPU Model: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz # CPU Data: 1 Quad Core package with 8 CPUs # CPU Architechture: x86_64 # HyperThreading: Yes # Max CPU MHz: 3400.0000 # Current CPU MHz: 3392.320 #---------------------------------------------------------------------- # RAM: 31.258 GB # SWAP: 7.999 GB #---------------------------------------------------------------------- # Install Date: Tue 29 Oct 2019 09:57:41 PM EDT # Linux Distribution: Fedora 31 (Thirty One) X86_64 # Kernel Version: 5.5.15-200.fc31.x86_64 #---------------------------------------------------------------------- # Disk Partition Info # Filesystem Size Used Avail Use% Mounted on # /dev/mapper/fedora-root 3.9G 235M 3.5G 7% / # /dev/mapper/fedora-usr 25G 8.9G 15G 39% /usr # /dev/mapper/fedora-tmp 20G 45M 19G 1% /tmp # /dev/mapper/fedora-var 30G 13G 17G 44% /var # /dev/mapper/fedora-home 55G 5.7G 47G 11% /home # /dev/sda1 2.0G 516M 1.3G 29% /boot #---------------------------------------------------------------------- # LVM Physical Volume Info # PV VG Fmt Attr PSize PFree # /dev/sda2 fedora lvm2 a-- 72.00g 0 # /dev/sda3 fedora lvm2 a-- <224.08g 154.08g ####################################################################### This server also provides DHCPD, BIND, NTPD, and SendMail services.
systemd is just the manager here, and if httpd needs to be ordered after some requirements are met, this would need to be expressed in the unit file. But in this particular case, it sounds like httpd should be using IP_FREEBIND to allow it to bind to the address and not force any ordering.
See "man httpd.service" - Starting the service at boot time The httpd.service and httpd.socket units are disabled by default. To start the httpd service at boot time, run: systemctl enable httpd.service. In the default configuration, the httpd daemon will accept connections on port 80 (and, if mod_ssl is installed, TLS connections on port 443) for any configured IPv4 or IPv6 address. If httpd is configured to depend on any specific IP address (for example, with a "Listen" directive) which may only become available during start-up, or if httpd depends on other services (such as a database daemon), the service must be configured to ensure correct start-up ordering. For example, to ensure httpd is only running after all configured network interfaces are configured, create a drop-in file (as described above) with the following section: [Unit] After=network-online.target Wants=network-online.target See https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ for more information on start-up ordering with systemd.
Thanks for pointing me to this as a better circumvention to the problem. It does work as expected. And thanks for the link to start-up ordering. However I think it is still a bug because it is quite common - at least in my experience - to use a Listen directive. I have always used Listen directives, even on hosts with only a single IP address and it is clearly necessary on hosts with multiple NICs and IP addresses. Adding the lines above to the /usr/lib/systemd/system/httpd.service default file. Adding these entries would cause no problems for configurations not using a Listen directive and would prevent this problem for those that do. In addition I believe that something changed to cause this to occur. It was not a problem in the past. Unfortunately I did not keep good records of when this first started happening. So I do have a question I hope you will answer. Who is responsible for fixing problems with systemd unit files in a situation like this? Is it for the service developers such as those for httpd? Or is it the responsibility of the systemd team? I want to know so I can try to submit to the correct place in the future. Anyway - Thank you.
(In reply to David Both from comment #3) > In addition I believe that something changed to cause this to occur. It was > not a problem in the past. Unfortunately I did not keep good records of when > this first started happening. The httpd behaviour has not changed here since whenever Fedora started using systemd. > So I do have a question I hope you will answer. Who is responsible for > fixing problems with systemd unit files in a situation like this? Is it for > the service developers such as those for httpd? Or is it the responsibility > of the systemd team? I want to know so I can try to submit to the correct > place in the future. In the first instance, the package which owns the relevant unit. In the unlikely case we determine an issue is due to systemd, we can reassign appropriately.
I really think httpd should be using IP_FREEBIND. Anything else is setting up the user for failure. (I guess it could be opt-in or opt-out. But it certainly should be offered as an option.)
I resurrected Lubos' patch to do that upstream. https://bz.apache.org/bugzilla/show_bug.cgi?id=61865
(In reply to Joe Orton from comment #6) > I resurrected Lubos' patch to do that upstream. > https://bz.apache.org/bugzilla/show_bug.cgi?id=61865 Thank you.
(In reply to David Both from comment #3) > Thanks for pointing me to this as a better circumvention to the problem. It > does work as expected. And thanks for the link to start-up ordering. > > However I think it is still a bug because it is quite common - at least in > my experience - to use a Listen directive. I have always used Listen > directives, even on hosts with only a single IP address and it is clearly > necessary on hosts with multiple NICs and IP addresses. Adding the lines > above to the /usr/lib/systemd/system/httpd.service default file. Adding > these entries would cause no problems for configurations not using a Listen > directive and would prevent this problem for those that do. > > In addition I believe that something changed to cause this to occur. It was > not a problem in the past. Unfortunately I did not keep good records of when > this first started happening. > > So I do have a question I hope you will answer. Who is responsible for > fixing problems with systemd unit files in a situation like this? Is it for > the service developers such as those for httpd? Or is it the responsibility > of the systemd team? I want to know so I can try to submit to the correct > place in the future. > > Anyway - Thank you. I really agree: >> However I think it is still a bug because it is quite common - at least in my experience - to use a Listen directive. I have always used Listen directives, even on hosts with only a single IP address and it is clearly necessary on hosts with multiple NICs and IP addresses. Adding the lines above to the /usr/lib/systemd/system/httpd.service default file. Adding these entries would cause no problems for configurations not using a Listen directive and would prevent this problem for those that do. httpd is a "network" service, it's useless without networking, it should start after the network IMO.