Bug 1145245

Summary: Libreswan appears to start with systemd before all the NICs are up and running.
Product: Red Hat Enterprise Linux 7 Reporter: Paul Wouters <pwouters>
Component: libreswanAssignee: Paul Wouters <pwouters>
Status: CLOSED ERRATA QA Contact: Jaroslav Aster <jaster>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.1CC: amarecek, extras-qa, gregscott, jaster, pwouters
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1144831 Environment:
Last Closed: 2015-03-05 10:22:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1144831    
Bug Blocks:    

Description Paul Wouters 2014-09-22 15:45:21 UTC 
+++ This bug was initially created as a clone of Bug #1144831 +++

Description of problem:

When I use systemd to start libreswan 3.10 and earlier versions that come with Fedora 20, the tunnel does not come up.  But after I login, doing systemctl restart ipsec by hand fires everything up as expected.  So I know my tunnel config is good.  

I have an rc.local file that calls a script named rc.firewall, where I setup a bunch of iptables rules.  The last few lines currently look like this:

echo "Starting IPSEC from here since it looks like timing issues mess it up at boot time."
sleep 10
ip addr show > /firewall-scripts/before.txt
systemctl restart ipsec
ip addr show > /firewall-scripts/after.txt

exit

As you can see, sleeping 10 seconds before starting ipsec does the trick.  This suggests a timing issue with systemd at boot time.


Version-Release number of selected component (if applicable):
Libreswan 3.10 and 3.8

How reproducible:
Always

Steps to Reproduce:
1.  Build an F20 system with multiple NICs
2.  Set up an ipsec tunnel to somewhere
3.  systemctl enable ipsec
4.  Boot - ipsec starts before the NICs are ready and the tunnel does not come up.

Actual results:
IPSEC starts up before the NICs are ready and the tunnel does not come up.  Work around the problem by starting the tunnel in a custom script after sleeping an appropriate length of time.

Expected results:
IPSEC should start properly when called from systemd.

Additional info:

The relevant NICs are named enp2s0 and enp3s0.  NIC enp2s0 is public, enp3s0 is private.  

Waiting 10 seconds during startup seems to work.  While sleeping for only 5 seconds, I noticed during troubleshooting that NIC enp2s0 somehow took on the IP Address that NIC enp3s0 should have, but with a /32 mask.  The only possible way this could happen is some kind of timing interaction with ipsec startup.  So I captured data before and after starting ipsec, as you can see from the script extract above.  

First, when sleeping only 5 seconds - note NIC enp3s0 is not yet up. I dummied up the public IP Address on NIC enp2s0.

[root@superior-fw firewall-scripts]# more before2.txt
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen
1000
    link/ether 00:30:18:c0:6e:26 brd ff:ff:ff:ff:ff:ff
    inet 1.2.3.4/29 brd 1.2.3.7 scope global enp2s0
       valid_lft forever preferred_lft forever
    inet6 fe80::230:18ff:fec0:6e26/64 scope link
       valid_lft forever preferred_lft forever
3: enp3s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:30:18:c0:6e:27 brd ff:ff:ff:ff:ff:ff
4: enp5s4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:30:18:ad:e7:e7 brd ff:ff:ff:ff:ff:ff
5: .
   .
   . (Other non-relevant NICs cut.)

And here is what things look like after starting ipsec.  Note enp3s0 is a little further along in getting started with a temporary state of DOWN.  

[root@superior-fw firewall-scripts]# more after2.txt
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen
1000
    link/ether 00:30:18:c0:6e:26 brd ff:ff:ff:ff:ff:ff
    inet 209.23.151.114/29 brd 209.23.151.119 scope global enp2s0
       valid_lft forever preferred_lft forever
    inet6 fe80::230:18ff:fec0:6e26/64 scope link
       valid_lft forever preferred_lft forever
3: enp3s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default q
len 1000
    link/ether 00:30:18:c0:6e:27 brd ff:ff:ff:ff:ff:ff
4: enp5s4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:30:18:ad:e7:e7 brd ff:ff:ff:ff:ff:ff
5: .
   .
   .
After logging in and doing ip addr show by hand, I noticed NIC enp2s0 had an additional IP Address of 172.21.5.100/32.  I don't have a capture of that. 

Now here is what things look like before and after starting ipsec when sleeping 10 seconds.  

[root@superior-fw firewall-scripts]# more before.txt
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen
1000
    link/ether 00:30:18:c0:6e:26 brd ff:ff:ff:ff:ff:ff
    inet 1.2.3.4/29 brd 1.2.3.7 scope global enp2s0
       valid_lft forever preferred_lft forever
    inet6 fe80::230:18ff:fec0:6e26/64 scope link
       valid_lft forever preferred_lft forever
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen
1000
    link/ether 00:30:18:c0:6e:27 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::230:18ff:fec0:6e27/64 scope link tentative
       valid_lft forever preferred_lft forever
4: enp5s4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:30:18:ad:e7:e7 brd ff:ff:ff:ff:ff:ff
5: .
   .
   .

And after starting ipsec.  Note that NIC enp3s0 now has its proper IP Address.

[root@superior-fw firewall-scripts]# more after.txt
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen
1000
    link/ether 00:30:18:c0:6e:26 brd ff:ff:ff:ff:ff:ff
    inet 1.2.3.4/29 brd 1.2.3.7 scope global enp2s0
       valid_lft forever preferred_lft forever
    inet6 fe80::230:18ff:fec0:6e26/64 scope link
       valid_lft forever preferred_lft forever
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen
1000
    link/ether 00:30:18:c0:6e:27 brd ff:ff:ff:ff:ff:ff
    inet 172.21.5.100/24 brd 172.21.5.255 scope global enp3s0
       valid_lft forever preferred_lft forever
    inet6 fe80::230:18ff:fec0:6e27/64 scope link
       valid_lft forever preferred_lft forever
4: enp5s4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:30:18:ad:e7:e7 brd ff:ff:ff:ff:ff:ff
5: .
   .
   .


And here is the current picture of all NICs when I login and look at them by hand.  This time showing everything, even the irrelevant ones so you can have a complete picture for what this system looks like.

[root@superior-fw firewall-scripts]# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:30:18:c0:6e:26 brd ff:ff:ff:ff:ff:ff
    inet 1.2.3.4/29 brd 1.2.3.7 scope global enp2s0
       valid_lft forever preferred_lft forever
    inet6 fe80::230:18ff:fec0:6e26/64 scope link
       valid_lft forever preferred_lft forever
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:30:18:c0:6e:27 brd ff:ff:ff:ff:ff:ff
    inet 172.21.5.100/24 brd 172.21.5.255 scope global enp3s0
       valid_lft forever preferred_lft forever
    inet6 fe80::230:18ff:fec0:6e27/64 scope link
       valid_lft forever preferred_lft forever
4: enp5s4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:30:18:ad:e7:e7 brd ff:ff:ff:ff:ff:ff
5: enp5s6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:30:18:ad:e7:e8 brd ff:ff:ff:ff:ff:ff
6: enp5s7: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 00:30:18:ad:e7:e9 brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.88/24 brd 10.10.10.255 scope global enp5s7
       valid_lft forever preferred_lft forever
7: wlp1s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether b4:b6:76:b6:39:1f brd ff:ff:ff:ff:ff:ff
8: ip_vti0@NONE: <NOARP> mtu 1332 qdisc noop state DOWN group default
    link/ipip 0.0.0.0 brd 0.0.0.0
[root@superior-fw firewall-scripts]#


Here are the relevant ifcfg-nnn files.  I dummied up the public IP Addresses.  

[root@superior-fw network-scripts]# more ifcfg-enp2s0
TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=enp2s0
UUID=0f07ae8d-4974-4f72-af77-548cea8a0cf5
ONBOOT=yes
HWADDR=00:30:18:C0:6E:26
PEERDNS=yes
PEERROUTES=yes
BROADCAST=1.2.3.7
IPADDR=1.2.3.4
NETMASK=255.255.255.248
NETWORK=1.2.3.0
[root@superior-fw network-scripts]#
[root@superior-fw network-scripts]#
[root@superior-fw network-scripts]# more ifcfg-enp3s0
TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=enp3s0
UUID=a81b3bae-9e9d-4c7b-9e16-fc83f7fa2b56
ONBOOT=yes
HWADDR=00:30:18:C0:6E:27
PEERDNS=yes
PEERROUTES=yes
BROADCAST=172.21.5.255
IPADDR=172.21.5.100
NETMASK=255.255.255.0
NETWORK=172.21.5.0
[root@superior-fw network-scripts]#


The only difference when waiting only 5 seconds is, NIC enp2s0 also ends up with an additional IP Address of 172.21.5.100/32.  

All this has to be a consequence of ipsec starting too soon with systemd.

--- Additional comment from Paul Wouters on 2014-09-22 11:40:52 EDT ---

Can you try this change to the systemd unit file and see if it fixes your issue:

diff --git a/initsystems/systemd/ipsec.service.in b/initsystems/systemd/ipsec.service.in
index 082ef79..9edcbd4 100644
--- a/initsystems/systemd/ipsec.service.in
+++ b/initsystems/systemd/ipsec.service.in
@@ -1,7 +1,6 @@
 [Unit]
 Description=Internet Key Exchange (IKE) Protocol Daemon for IPsec
-After=network.target
-#After=remote-fs.target
+After=network-online.target
 
 [Service]
 Type=simple
Comment 7 errata-xmlrpc 2015-03-05 10:22:51 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0431.html