Can you try this change to the systemd unit file and see if it fixes your issue:

diff --git a/initsystems/systemd/ipsec.service.in b/initsystems/systemd/ipsec.service.in
index 082ef79..9edcbd4 100644
--- a/initsystems/systemd/ipsec.service.in
+++ b/initsystems/systemd/ipsec.service.in
@@ -1,7 +1,6 @@
 [Unit]
 Description=Internet Key Exchange (IKE) Protocol Daemon for IPsec
-After=network.target
-#After=remote-fs.target
+After=network-online.target
 
 [Service]
 Type=simple

Thanks Paul.  I'm out of town for a few days but hope to try it tonight.  Would it be possible to post a copy of the whole unit file so I can just paste it in?  Just makes my testing a little bit easier

thanks

- Greg

Welcome to the Twilight Zone.  I think we're suddenly chasing a different problem now.  Long story short, the timing seems better, but now I can't make ipsec on my left side set up the route back to the right side to save my life.  I did not touch any tunnel parameters.  The misbehaving system is on the left side.

I dummied up the first 2 octets of public IP Addresses below.

Here is what I did to /usr/lib/systemd/system/ipsec.service when I inserted that line you mentioned.  

[root@superior-fw system]# diff ipsec.service ipsec.service-original
3c3
< ##After=network.target
---
> After=network.target
5d4
< After=network-online.target
[root@superior-fw system]#

Here is the complete unit file with the change put in:

root@superior-fw system]# more ipsec.service
[Unit]
Description=Internet Key Exchange (IKE) Protocol Daemon for IPsec
##After=network.target
#After=remote-fs.target
After=network-online.target

[Service]
Type=simple
Restart=always
# backwards compatible with plutorestartoncrash=no
#RestartPreventExitStatus=137 143 SIGTERM SIGKILL
EnvironmentFile=-/etc/sysconfig/pluto
#Environment=IPSEC_SBINDIR=/usr/sbin
#Environment=IPSEC_EXECDIR=/usr/sbin/ipsec
#PIDFile=/var/run/pluto/pluto.pid
#
ExecStartPre=/usr/libexec/ipsec/addconn --config /etc/ipsec.conf --checkconfig
ExecStartPre=/usr/libexec/ipsec/_stackmanager start
ExecStartPre=/usr/sbin/ipsec --checknss
ExecStart=/bin/sh -c 'eval `/usr/libexec/ipsec/pluto --config /etc/ipsec.conf --nofork $PLUTO_OPTIO
NS`'
ExecStop=/usr/libexec/ipsec/whack --shutdown
ExecStopPost=/sbin/ip xfrm policy flush
ExecStopPost=/sbin/ip xfrm state flush
ExecReload=/usr/libexec/ipsec/whack --listen

[Install]
WantedBy=multi-user.target[root@superior-fw ~]

Let's try a reboot on my left side and see what happens.

Doing /var/log/secure -f on the right side, I see:

Sep 24 04:00:05 lme-fw2 pluto[2534]: "Superior-Everywhere" #695: STATE_QUICK_R2: IPsec SA established tunnel mode {ESP=>0x4d85b7d0 <0xb3b21e51 xfrm=AES_128-HMAC_SHA1 NATOA=none NATD=none DPD=none}

This looks good.

But going to the left side where the rubber meets the road - this is the result of a bunch of troubleshooting:

[root@superior-fw ~]# ping 192.168.3.4
PING 192.168.3.4 (192.168.3.4) 56(84) bytes of data.
^C
--- 192.168.3.4 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 3999ms

[root@superior-fw ~]# ip route show
default via 1.2.151.113 dev enp2s0
10.10.10.0/24 dev enp5s7  proto kernel  scope link  src 10.10.10.88
169.254.0.0/16 dev enp2s0  scope link  metric 1002
169.254.0.0/16 dev enp3s0  scope link  metric 1003
169.254.0.0/16 dev enp5s7  scope link  metric 1006
172.21.5.0/24 dev enp3s0  proto kernel  scope link  src 172.21.5.100
1.2.151.112/29 dev enp2s0  proto kernel  scope link  src 209.23.151.114

And to work around the problem:

[root@superior-fw ~]# ip route add 192.168.0.0/16 via 1.2.151.113 dev enp2s0  src 172.21.5.100  mtu 1400
[root@superior-fw ~]# ping 192.168.3.4
PING 192.168.3.4 (192.168.3.4) 56(84) bytes of data.
64 bytes from 192.168.3.4: icmp_seq=1 ttl=31 time=34.2 ms
64 bytes from 192.168.3.4: icmp_seq=2 ttl=31 time=33.3 ms
64 bytes from 192.168.3.4: icmp_seq=3 ttl=31 time=33.8 ms
^C
--- 192.168.3.4 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 33.372/33.808/34.239/0.384 ms
[root@superior-fw ~]#

I have to add the route back to the right side by hand.  That's the only way I can get the right side to answer anything.  

Below worked one time but did not work the second time I tried it.  On the right side, do this:

ipsec auto --delete Superior-Everywhere
ipsec auto --add Superior-Everywhere

And on the left side do this:

systemctl stop ipsec
systemctl start ipsec

This worked the first time I tried it, but not the second time.  (Edit later - it probably worked the first time because I added the route by hand and stopping and restarting ipsec did not get rid of the route.)

I also tried deleting and adding the tunnel on the left side. Here is a snippet from /var/log/secure on the left side when I tried that:

Sep 24 04:01:44 superior-fw pluto[2253]: "Superior-Everywhere" #1: deleting state (STATE_MAIN_I4)
Sep 24 04:01:44 superior-fw pluto[2253]: "Superior-Everywhere": unroute-client output: /usr/libexec/ipsec/_updown.netkey: doroute "ip route del 192.168.0.0/16 via 1.2.151.113 dev enp2s0 mtu 1400PLUTO_ADDTIME=0 " failed (Error: argument "1400PLUTO_ADDTIME=0" is wrong: "mtu" value is invalid)

Hmmm....

Restarting ipsec on my left side, I see this in /var/log/secure:

Sep 24 04:23:13 superior-fw pluto[2677]: "Superior-Everywhere": route-client output: /usr/libexec/ipsec/_updown.netkey: doroute "ip route replace 192.168.0.0/16 via 1.2.151.113 dev enp2s0 mtu 1400PLUTO_ADDTIME=0  src 172.21.5.100" failed (Error: argument "1400PLUTO_ADDTIME=0" is wrong: "mtu" value is invalid)

Why is my MTU suddenly invalid?  This works at 5 other sites and worked at this site up until right now. And it stays broken even when I put the above unit file back the way it was and reboot. 

Here is what this tunnel looks like on the left side:

[root@superior-fw firewall-scripts]# cd /etc/ipsec.d
[root@superior-fw ipsec.d]# more superior-ipsec.conf
# /etc/ipsec.d/superior-ipsec.conf - IPsec configuration file
# The HOME (HQ) office is always on the right.  ("Make yerself RIGHT at home!",
# while the other branch sites have LEFT home.)
#

##version 2.0   # conforms to second version of ipsec.conf specification
# Commented out because version is declared in ipsec.conf

# basic configuration

conn Superior-Everywhere
        type=tunnel
        #
        # Left security gateway, subnet behind it, next hop toward right.
        #
        also=superior
        #
        # Right security gateway, subnet behind it, next hop toward left.
        #
        also=hq
        mtu=1400
        auto=start

include /etc/ipsec.d/sites.conf

And the relevant portion of sites.conf with details about every site.  I have a copy of this at every site.

[root@superior-fw ipsec.d]# more sites.conf
# /etc/ipsec.d/sites.conf - IPsec configuration file describing each site.
# The HOME office is always on the right.  ("Make yerself RIGHT at home!",
# while the other branch sites have LEFT home.)

##version       2.0     # conforms to second version of ipsec.conf specification

# basic configuration

conn hq
        right=3.4.248.50
        rightnexthop=3.4.248.49
        rightsubnet=192.168.0.0/16
        rightsourceip=192.168.3.5
        rightid=
        # rsakey AQPL4uclF   Generated 2014-0918
        rightrsasigkey=0vjcGc2O+7q9...

(Other site descriptions removed.)

conn superior
        left=1.2.151.114
        leftnexthop=1.2.151.113
        leftsubnet=172.21.5.0/24
        leftsourceip=172.21.5.100
        leftid=
        # rsakey AQPpA2gXk
        leftrsasigkey=0sAQ...
[root@superior-fw ipsec.d]#

[root@superior-fw ipsec.d]# ipsec version
Linux Libreswan 3.10 (netkey) on 3.16.2-201.fc20.x86_64
[root@superior-fw ipsec.d]# more /etc/issue
Fedora release 20 (Heisenbug)
Kernel \r on an \m (\l)

[root@superior-fw ipsec.d]#

And here is the right side:

# /etc/ipsec.d/rochester-ipsec.conf - IPsec configuration file.
# The HOME (HQ) office is always on the right.  ("Make yerself RIGHT at home!",
# while the other branch sites have LEFT home.)
#

(Connections to other sites removed.)

conn Superior-Everywhere
        type=tunnel
        #
        # Left security gateway, subnet behind it, next hop toward right.
        #
        also=superior
        #
        # Right security gateway, subnet behind it, next hop toward left.
        #
        also=hq
        mtu=1400
        auto=start

include /etc/ipsec.d/sites.conf

(Sites.conf is the same here as in all the left sides.)

[root@lme-fw2 ipsec.d]# ipsec version
Linux Libreswan 3.8 (netkey) on 3.16.2-200.fc20.x86_64
[root@lme-fw2 ipsec.d]# more /etc/issue
Fedora release 20 (Heisenbug)
Kernel \r on an \m (\l)

The left side in Superior is running version 3.10 (see above).  Looks like its MTU problems started on Sept. 21.  So the MTU problem is not new tonight, I just found it tonight.  

[root@superior-fw ipsec.d]# grep mtu /var/log/secure
Sep 21 04:59:22 superior-fw pluto[1673]: "Superior-Everywhere": route-client output: /usr/libexec/ipsec/_updown.netkey: doroute "ip route replace 192.168.0.0/16 via 209.23.151.113 dev enp2s0 mtu 1400PLUTO_ADDTIME=0  src 172.21.5.100" failed (Error: argument "1400PLUTO_ADDTIME=0" is wrong: "mtu" value is invalid)
.
.
. (lots of others)


The right side is running version 3.8.  No MTU problems on the right side.  

[root@lme-fw2 ipsec.d]# grep mtu /var/log/secure
[root@lme-fw2 ipsec.d]#

confirmed bug. Fix commited upstream.  I'll check if upstream will release 3.11 today, and either upgrade to that, or call a new build with 3.10 and a patch.

Thanks Paul.  Is the confirmed bug the startup timing thing or the MTU thing or both?  No sense doing a special patch - I have a workaround for now and I can test 3.11 when it comes out the other end.

thanks

- Greg

(In reply to Paul Wouters from comment #1)
> Can you try this change to the systemd unit file and see if it fixes your
> issue:

I can confirm that the equivalent in e.g. /etc/systemd/system/ipsec.service.d/network-online.target.conf was enough to fix the interface issue:

[Unit]
After=network-online.target

Is libreswan expected to pay attention to a netlink socket and/or periodically scan for new/changed interfaces?  The above is sufficient to bring up a tunnel shortly after boot, but feels like an incomplete hack...

-Rob

libreswan-3.11-1.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/libreswan-3.11-1.fc21

libreswan-3.11-1.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/libreswan-3.11-1.fc20

libreswan-3.11-1.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/libreswan-3.11-1.fc19

Package libreswan-3.11-1.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libreswan-3.11-1.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-13597/libreswan-3.11-1.fc21
then log in and leave karma (feedback).

I just updated my problem site to 3.11-1.fc20 and restarted.  All seems to work well now.  It starts when it's supposed to and the MTU problem doesn't happen anymore.  Leaving good karma in https://admin.fedoraproject.org/updates/FEDORA-2014-13597/libreswan-3.11-1.fc21

- Greg

thanks for the feedback! Feel free to close this bug :)

libreswan-3.11-1.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

libreswan-3.11-1.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

libreswan-3.11-1.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.