Bug 772407

Summary: Systemd breaks rc.local dependencies on NICs being up; rc.local really should be the last thing that runs at startup
Product: [Fedora] Fedora Reporter: Greg Scott <gregscott>
Component: systemdAssignee: systemd-maint
Status: CLOSED DUPLICATE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 16CC: johannbg, lpoetter, metherid, mschmidt, notting, plautrba, systemd-maint
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-01-09 14:11:41 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Greg Scott 2012-01-07 19:54:41 UTC
Description of problem:
I have an rc.local script that sets up several firewall rules and secondary IP Addresses on various network interfaces.  Unfortunately, systemd breaks all this because the network is not yet started when rc.local runs.  Apparently, the network now starts after rc.local.  See below for a test script to demonstrate the problem.  

Version-Release number of selected component (if applicable):  
Fedora 16


How reproducible:
Always

Steps to Reproduce:
1. Set up /etc/rc.d/rc.local to add a secondary IP Address to a NIC.  Or maybe something like, echo "1" > /proc/sys/net/ipv4/ip_forward.
2. Reboot
3. ip addr show will not have the secondary IP Address.  Ip_forward will be 0.  
4. Look at /var/log/boot.log for footprints - sometimes there will really be a boot log, sometimes it will be blank.  When you get a good boot.log, notice command failures because the NICs are not started yet.  
5. Login and run rc.local by hand and it works just fine.  
  
Actual results:

Commands issued on a NIC not yet ifup don't work.  This breaks everything that subsequently depends on the network being up.  With no reliable startup logging, the naive network admin assumes the newly upgraded Fedora firewall booted OK.  After all, the rc.local script runs just fine by hand.  Then the phone rings when internally hosted email/web servers, etc. are offline.  The network admin has a miserable night tracking down a problem with a component that's worked for 10+ years.


Expected results:

By the time the optional rc.local runs, all the other components it depends on should be up and availble.  rc.local should always be the **LAST** thing that happens at startup.

Additional info:

Set IP Addresses appropriately and save this test script as /etc/rc.d/rc.local.  It will run for a very long time at startup because the network does not start until after this script finishes.  And you will see some interesting results in boot.log if you're able to capture it. 

[code]
#!/bin/sh

TRUSTED1_IP="192.168.100.254"
TRUSTED1_IFACE="p3p1"

DMZ_IP="10.10.10.74"
DMZ_IFACE="p2p1"

INET_IP="1.2.20.146"
INET_IFACE="p1p1"

READY="true"
I=0
while [ $I -lt 40 ]; do
    for ADR in $INET_IP $TRUSTED1_IP $DMZ_IP; do
        echo "$I Testing $ADR"
        ping -c 1 -q -w 3 $ADR > /dev/null
        if [ $? != 0 ]
        then
            echo "      $ADR is down"
            READY="false"
            break
        fi
    done
    if [ $READY = "true" ]
    then
        break
    fi
    let "I=I+1"
done

if [ $READY = "false" ]
    then
    echo "At least one interface is not up yet"
fi

[/code]

Comment 1 Michal Schmidt 2012-01-09 14:11:41 UTC
We're changing rc-local to run after network.target in bug 754789, because
talking to the network seems to be a common use-case.

However, running a service "after everything else" is not directly supported by
systemd. Just plan to migrate away from using rc-local. Define your own service
unit with whatever minimal dependencies are truly needed.

*** This bug has been marked as a duplicate of bug 754789 ***