Bug 1158222 - pluto not starting due to 'NSS readonly initialization failed (err -8015)' when using in OpenStack VPNaaS
Summary: pluto not starting due to 'NSS readonly initialization failed (err -8015)' wh...
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: libreswan
Version: 22
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Paul Wouters
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-10-28 20:59 UTC by Stefan Berger
Modified: 2016-07-19 12:19 UTC (History)
4 users (show)

Fixed In Version:
Clone Of:
Environment:
OpenStack Juno installed with Devstack on Fedora 20
Last Closed: 2016-07-19 12:19:55 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Stefan Berger 2014-10-28 20:59:05 UTC
Description of problem:

Pluto does not start when used in OpenStack VPNaaS. I installed OpenStack via devstack with all components on Juno level.

I am not sure whether this is an OpenStack VPNaaS problem or a problem of libreswan.

Version-Release number of selected component (if applicable):

3.10-3.fc20

How reproducible:

I followed these instructions here: https://wiki.openstack.org/wiki/Neutron/VPNaaS/HowToInstall

Actual results:

From the log with additional '--stderrlog' since this shows an error it otherwise would not show. Note, exit code is 0.

Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-250faac2-167b-4861-9d0c-b5710bf02ee2', 'ipsec', 'pluto', '--ctlbase', '/opt/stack/data/neutron/ipsec/250faac2-167b-4861-9d0c-b5710bf02ee2/var/run/pluto', '--ipsecdir', '/opt/stack/data/neutron/ipsec/250faac2-167b-4861-9d0c-b5710bf02ee2/etc', '--use-netkey', '--uniqueids', '--nat_traversal', '--secretsfile', '/opt/stack/data/neutron/ipsec/250faac2-167b-4861-9d0c-b5710bf02ee2/etc/ipsec.secrets', '--virtual_private', '%v4:10.1.0.0/24,%v4:10.2.0.0/24', '--stderrlog']
Exit code: 0
Stdout: ''
Stderr: 'adjusting ipsec.d to /opt/stack/data/neutron/ipsec/250faac2-167b-4861-9d0c-b5710bf02ee2/etc\nwarning: option "--nat_traversal" is obsolete; ignored\nwarning: option "--virtual_private" with \'_\' in its name is obsolete; use \'-\'\nnss directory plutomain: /opt/stack/data/neutron/ipsec/250faac2-167b-4861-9d0c-b5710bf02ee2/etc\nFATAL: NSS readonly initialization ("/opt/stack/data/neutron/ipsec/250faac2-167b-4861-9d0c-b5710bf02ee2/etc") failed (err -8015) \n' from (pid=29756) execute /opt/stack/neutron/neutron/agent/linux/utils.py:81  

The interesting error message seems to be the following one which only appears with --stderrlog:

FATAL: NSS readonly initialization ("/opt/stack/data/neutron/ipsec/250faac2-167b-4861-9d0c-b5710bf02ee2/etc") failed (err -8015)

Additional info:

# rpm -q -a | grep ^nss
nss-3.17.2-1.fc20.x86_64
nss-mdns-0.10-13.fc20.x86_64
nss-tools-3.17.2-1.fc20.x86_64
nss-softokn-3.17.2-1.fc20.x86_64
nss-softokn-freebl-3.17.2-1.fc20.x86_64
nss-sysinit-3.17.2-1.fc20.x86_64
nss-util-3.17.2-1.fc20.x86_64
# getenforce 
Permissive

Comment 1 Paul Wouters 2014-10-29 03:54:46 UTC
nss directory plutomain: /opt/stack/data/neutron/ipsec/250faac2-167b-4861-9d0c-b5710bf02ee2/etc

Seems related to bug #1092047

This is most likely because we are dropping CAP_DAC_OVERRIDE and we cannot read the directory? Is that etc directory mode 700 instead of like 750 (if root in group neutron) or 755 (if root is not in group neutron). Or perhaps a directory above it is missing some rx ?

Comment 2 Stefan Berger 2014-10-29 10:33:14 UTC
Thanks for responding so quickly. I still have the environment running:

# ls -l /opt/stack/data/neutron/ipsec/250faac2-167b-4861-9d0c-b5710bf02ee2/
total 12
drwxr-xr-x. 4 stefanb stefanb 4096 Oct 28 16:42 etc
drwxr-xr-x. 2 stefanb stefanb 4096 Oct 28 16:42 log
drwxr-xr-x. 3 stefanb stefanb 4096 Oct 28 16:42 var

# ls -l /opt/stack/data/neutron/ipsec
total 4
drwxr-xr-x. 5 stefanb stefanb 4096 Oct 28 16:42 250faac2-167b-4861-9d0c-b5710bf02ee2

# ls -l /opt/stack/data/neutron
total 20
drwxr-xr-x. 2 stefanb stefanb 4096 Oct 23 12:26 dhcp
drwxr-xr-x. 3 stefanb stefanb 4096 Oct 23 12:26 external
drwxr-xr-x. 2 stefanb stefanb 4096 Oct 23 12:26 ha_confs
drwxr-xr-x. 3 stefanb stefanb 4096 Oct 28 16:42 ipsec
drwxrwxr-x. 2 stefanb stefanb 4096 Oct 23 12:26 lock
srwxrwxr-x. 1 stefanb stefanb    0 Oct 23 12:26 metadata_proxy

# ls -l /opt/stack/data
total 16
drwxr-xr-x. 3 stefanb stefanb 4096 Oct 23 12:25 cinder
drwxr-xr-x. 4 stefanb stefanb 4096 Oct 23 12:24 glance
drwxr-xr-x. 7 stefanb stefanb 4096 Oct 23 12:30 neutron
drwxr-xr-x. 6 stefanb root    4096 Oct 23 12:26 nova



# ls -l /opt/stack/data/neutron/ipsec/250faac2-167b-4861-9d0c-b5710bf02ee2/etc
total 16
-rw-r--r--.  1 stefanb stefanb 1758 Oct 28 16:42 ipsec.conf
drwxr-xr-x. 11 stefanb stefanb 4096 Oct 28 16:42 ipsec.d
-rw-r--r--.  1 stefanb stefanb   68 Oct 28 16:42 ipsec.secrets
drwxr-xr-x.  3 stefanb stefanb 4096 Oct 28 16:42 pki

# ls -l /opt/stack/data/neutron/ipsec/250faac2-167b-4861-9d0c-b5710bf02ee2/etc/ipsec.d
total 36
drwxr-xr-x. 2 stefanb stefanb 4096 Oct 28 16:42 aacerts
drwxr-xr-x. 2 stefanb stefanb 4096 Oct 28 16:42 acerts
drwxr-xr-x. 2 stefanb stefanb 4096 Oct 28 16:42 cacerts
drwxr-xr-x. 2 stefanb stefanb 4096 Oct 28 16:42 certs
drwxr-xr-x. 2 stefanb stefanb 4096 Oct 28 16:42 crls
drwxr-xr-x. 2 stefanb stefanb 4096 Oct 28 16:42 ocspcerts
drwxr-xr-x. 2 stefanb stefanb 4096 Oct 28 16:42 policies
drwxr-xr-x. 2 stefanb stefanb 4096 Oct 28 16:42 private
drwxr-xr-x. 2 stefanb stefanb 4096 Oct 28 16:42 reqs

Comment 3 Stefan Berger 2014-10-29 10:45:14 UTC
# ls -l / | grep opt
drwxr-xr-x.   3 root root  4096 Oct 17 09:51 opt

# ls -l /opt/stack | grep data
drwxr-xr-x.  6 stefanb root    4096 Oct 23 12:26 data

Comment 4 Paul Wouters 2014-10-29 15:58:31 UTC
hmm, I'll do some testing myself for this.

I guess you are not even using NSS in this case? You only use PSK in ipsec.secrets? And not RSA keys from NSS?

Comment 5 Stefan Berger 2014-10-29 16:08:01 UTC
Not sure, these things are happening under the hood of this VPNaaS. VPNaaS is writing all the config files.

#  cat ipsec.secrets 
# Configuration for myvpn 
172.24.4.226 172.24.4.234 : PSK "secret"


The above pluto command line call seems to be the first one is a sequence of calls. I think it should leave pluto running, but you probably know that better than I do. The following calls then show failures like this:

014-10-29 12:03:20.457 DEBUG neutron.agent.linux.utils [-] Running command: ['sud
o', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exe
c', 'qrouter-250faac2-167b-4861-9d0c-b5710bf02ee2', 'ipsec', 'whack', '--ctlbase',
 '/opt/stack/data/neutron/ipsec/250faac2-167b-4861-9d0c-b5710bf02ee2/var/run/pluto
', '--status'] from (pid=29756) create_process /opt/stack/neutron/neutron/agent/li
nux/utils.py:46
[...]
2014-10-29 12:03:20.815 ERROR neutron.agent.linux.utils [-] 
Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-250faac2-167b-4861-9d0c-b5710bf02ee2', 'ipsec', 'whack', '--ctlbase', '/opt/stack/data/neutron/ipsec/250faac2-167b-4861-9d0c-b5710bf02ee2/var/run/pluto', '--status']
Exit code: 1
Stdout: ''
Stderr: 'whack: Pluto is not running (no "/opt/stack/data/neutron/ipsec/250faac2-167b-4861-9d0c-b5710bf02ee2/var/run/pluto.ctl")\n'


So subsequent calls expect /opt/stack/data/neutron/ipsec/250faac2-167b-4861-9d0c-b5710bf02ee2/var/run/pluto.ctl (probably a socket) to be available.

Comment 6 Stefan Berger 2014-11-04 19:08:05 UTC
Paul, did you find anything?

Comment 7 Paul Wouters 2014-11-05 18:28:44 UTC
Seems that error -8015 is:

#define SEC_ERROR_BASE                          (-0x2000)
#define SEC_ERROR_LEGACY_DATABASE                   =   (SEC_ERROR_BASE + 177),

Could it be that these *.db files were created by an old pluto that created "bad" db files on startup before we made the change to make pluto only use db files readonly and make the systemd/init scripts be responsible for creating those?

Or possibly, check if there are any *.db files in that directory? I would expect to see:

cert8.db  key3.db  secmod.db

If those are not there, then it should be created using:

certutil -N -d /opt/stack/data/neutron/ipsec/250faac2-167b-4861-9d0c-b5710bf02ee2/etc

It could also be that in fact the *.db files live in 
/opt/stack/data/neutron/ipsec/250faac2-167b-4861-9d0c-b5710bf02ee2/etc/ipsec.d and the call you are showing forgot to add "/ipsec.d" to the --ipsecdir parameter

Comment 8 Paul Wouters 2014-11-05 19:25:58 UTC
I really suspect there are no db files there, based on this test:

paul@bofh:~$ mkdir /tmp/empty
paul@bofh:~$ certutil -L -f /tmp/empty
certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format.
paul@bofh:~$ 

So adding the previous certutil command from comment 7 will likely fix this.

Comment 9 Stefan Berger 2014-11-05 19:46:08 UTC
For directory /opt/stack/data/neutron/ipsec/250faac2-167b-4861-9d0c-b5710bf02ee2:

# find
.
./log
./var
./var/run
./var/run/pluto
./etc
./etc/ipsec.conf
./etc/ipsec.secrets
./etc/ipsec.d
./etc/ipsec.d/acerts
./etc/ipsec.d/ocspcerts
./etc/ipsec.d/reqs
./etc/ipsec.d/certs
./etc/ipsec.d/private
./etc/ipsec.d/crls
./etc/ipsec.d/aacerts
./etc/ipsec.d/policies
./etc/ipsec.d/cacerts
./etc/pki
./etc/pki/nssdb

So, as you guessed, these files are missing...

A current experimental patch looks as below. Unfortunately I get a prompt on the command line for entering a password, but once I type enter twice, I see pluto now running...

diff --git a/neutron/services/vpn/device_drivers/ipsec.py b/neutron/services/vpn
index c19b61e..e0381fd 100644
--- a/neutron/services/vpn/device_drivers/ipsec.py
+++ b/neutron/services/vpn/device_drivers/ipsec.py
@@ -373,6 +373,11 @@ class OpenSwanProcess(BaseSwanProcess):
         if not self.namespace:
             return
         virtual_private = self._virtual_privates()
+        import subprocess
+        subprocess.call(['/bin/certutil',
+                        '-N',
+                        '-d', self.etc_dir
+                        ])
         #start pluto IKE keying daemon
         self._execute([self.binary,
                        'pluto',

Comment 10 Stefan Berger 2014-11-05 19:55:21 UTC
When adding the '--empty-password' option, at least we don't need to provide a password then. 

diff --git a/neutron/services/vpn/device_drivers/ipsec.py b/neutron/services/vpn
index c19b61e..26317a5 100644
--- a/neutron/services/vpn/device_drivers/ipsec.py
+++ b/neutron/services/vpn/device_drivers/ipsec.py
@@ -373,6 +373,12 @@ class OpenSwanProcess(BaseSwanProcess):
         if not self.namespace:
             return
         virtual_private = self._virtual_privates()
+        import subprocess
+        subprocess.call(['/bin/certutil',
+                        '-N',
+                        '-d', self.etc_dir,
+                        '--empty-password'
+                        ])
         #start pluto IKE keying daemon
         self._execute([self.binary,
                        'pluto',


Did the behavior of pluto change recently so that I am seeing this problem on Fedora 20 while no one else may see it ?

Comment 11 Paul Wouters 2014-11-05 20:24:43 UTC
you can create an empty nss password file, like we do in the initscript/service file:

 TEMPFILE=$(/bin/mktemp "${IPSEC_CONFDDIR}/nsspw.XXXXXXX")
        [ $? -gt 0 ] && TEMPFILE="${IPSEC_CONFDDIR}/nsspw.$$"
        echo > ${TEMPFILE}
        certutil -N -f "${TEMPFILE}" -d "${IPSEC_CONFDDIR}"
        rm -f ${TEMPFILE}

pluto's change for that went into libreswan-3.6. Perhaps you had old valid db files around from a pluto run before that time?

Comment 12 Paul Wouters 2014-11-05 20:25:32 UTC
actually just noticed the --empty-password option. I wonder if that's new, that's much better :)

Comment 13 Stefan Berger 2014-11-05 20:31:41 UTC
That directory name '250faac2-167b-4861-9d0c-b5710bf02ee2' shown above is related to the UUID of the VPN service. It gets created when the VPN service is created and files are added to it. It is automatically deleted when the VPN service is deleted. There were no files before that and no old db files could have been there. Multiple VPNaaS instances can run concurrently. So these .db files would have to be created every time as it looks.


I have now two IPSecs from two OpenStack installations' VPNaaS talking to each other. Are these packet traces signs that the connection is up? 

5:28:23.509695 IP 172.24.4.226.isakmp > 172.24.4.234.isakmp: isakmp: phase 2/others R inf[E]
15:28:23.509704 IP 172.24.4.226.isakmp > 172.24.4.234.isakmp: isakmp: phase 2/others R inf[E]
15:28:23.510350 IP 172.24.4.234.isakmp > 172.24.4.226.isakmp: isakmp: phase 2/others I inf[E]
15:28:23.510367 IP 172.24.4.234.isakmp > 172.24.4.226.isakmp: isakmp: phase 2/others I inf[E]


Should I see a tunnel interface on each side?

Comment 14 Stefan Berger 2014-11-05 20:41:07 UTC
so, isakmp is port 500. Nothing listens on port 500 on either one of the two sides? Should pluto be listening on this port?

Comment 15 Paul Wouters 2014-11-05 20:41:31 UTC
that just shows some IKE traffic. The best way to check if there is a tunnel up is "ipsec status"

There are no interfaces created for this.

Comment 16 Paul Wouters 2014-11-05 20:42:19 UTC
yes pluto should be listening on udp port 500 and 4500

Comment 17 Stefan Berger 2014-11-05 20:55:41 UTC
Pluto is not listening on any port
# netstat -nap | grep pluto

.. but it's running

# ps aux | grep pluto
root     25733  0.0  0.0 331832  8764 ?        Ssl  15:22   0:00 /usr/libexec/ipsec/pluto --ctlbase /opt/stack/data/neutron/ipsec/250faac2-167b-4861-9d0c-b5710bf02ee2/var/run/pluto --ipsecdir /opt/stack/data/neutron/ipsec/250faac2-167b-4861-9d0c-b5710bf02ee2/etc --use-netkey --uniqueids --nat_traversal --secretsfile /opt/stack/data/neutron/ipsec/250faac2-167b-4861-9d0c-b5710bf02ee2/etc/ipsec.secrets --virtual_private %v4:10.1.0.0/24,%v4:10.2.0.0/24
root     25767  0.0  0.0  31156  1508 ?        S    15:22   0:00 _pluto_adns
root     31510  0.0  0.0 112680  2368 pts/37   S+   15:53   0:00 grep --color=auto pluto

any hint ?

Comment 18 Paul Wouters 2014-11-06 03:42:41 UTC
that is odd. does it start to listen when you issue: ipsec whack --listen

Can you add plutodebug=all and plutostderrlog=/tmp/pluto.log to the generated configuration and send me the logs?

Comment 19 Stefan Berger 2014-11-06 11:31:31 UTC
My bad, one has to apply the namespace:

# ip netns exec qrouter-250faac2-167b-4861-9d0c-b5710bf02ee2 netstat -nap | grep 500
udp        0      0 127.0.0.1:500           0.0.0.0:*                           25733/pluto         
udp        0      0 10.1.0.1:500            0.0.0.0:*                           25733/pluto         
udp        0      0 172.24.4.226:500        0.0.0.0:*                           25733/pluto         
udp        0      0 127.0.0.1:4500          0.0.0.0:*                           25733/pluto         
udp        0      0 10.1.0.1:4500           0.0.0.0:*                           25733/pluto         
udp        0      0 172.24.4.226:4500       0.0.0.0:*                           25733/pluto         
udp6       0      0 ::1:500                 :::*                                25733/pluto         


So it's there...

Will try to add the debugging statements later ...

Comment 20 Stefan Berger 2014-11-06 17:00:19 UTC
Another oddity are configdir=/etc and configfile=/etc/ipsec.conf. The latter probably shouldn't be pointing to this file but to the file /opt/stack/data/neutron/ipsec/f97a22e0-c7a1-4540-896f-5f910843346f/etc/ipsec.conf ?

# ip netns exec qrouter-f97a22e0-c7a1-4540-896f-5f910843346f ipsec whack --ctlbase /opt/stack/data/neutron/ipsec/f97a22e0-c7a1-4540-896f-5f910843346f
/var/run/pluto --status

000 using kernel interface: noklips
000  
000  
000 fips mode=disabled;
000 SElinux=disabled
000  
000 config setup options:
000  
000 configdir=/etc, configfile=/etc/ipsec.conf, secrets=/opt/stack/data/neutron/ipsec/f97a22e0-c7a1-4540-896f-5f910843346f/etc/ipsec.secrets, ipsecdir=/opt/stack/data/neutron/ipsec/f97a22e0-c7a1-4540-896f-5f910843346f/etc, dumpdir=/var/run/pluto, statsbin=unset
000 sbindir=/usr/sbin, libexecdir=/usr/libexec/ipsec
000 pluto_version=3.10, pluto_vendorid=OE-Libreswan-3.10
000 nhelpers=-1, uniqueids=yes, retransmits=yes, force-busy=no
000 ikeport=500, strictcrlpolicy=no, crlcheckinterval=0, listen=<any>
000 secctx-attr-value=32001
000 myid = (none)
000 debug none
000  
000 nat-traversal=yes, keep-alive=20, nat-ikeport=4500
000 virtual-private (%priv):
000 - allowed subnets: 10.1.0.0/24, 10.2.0.0/24
000  
000 ESP algorithms supported:
000  
000  
000 IKE algorithms supported:
000  
000 algorithm IKE encrypt: v1id=5, v1name=OAKLEY_3DES_CBC, v2id=3, v2name=3DES, blocksize=8, keydeflen=192
000 algorithm IKE encrypt: v1id=7, v1name=OAKLEY_AES_CBC, v2id=12, v2name=AES_CBC, blocksize=16, keydeflen=128
000 algorithm IKE encrypt: v1id=65004, v1name=OAKLEY_SERPENT_CBC, v2id=65004, v2name=SERPENT_CBC, blocksize=16, keydeflen=128
000 algorithm IKE encrypt: v1id=65005, v1name=OAKLEY_TWOFISH_CBC, v2id=65005, v2name=TWOFISH_CBC, blocksize=16, keydeflen=128
000 algorithm IKE encrypt: v1id=65289, v1name=OAKLEY_TWOFISH_CBC_SSH, v2id=65289, v2name=TWOFISH_CBC_SSH, blocksize=16, keydeflen=128
000 algorithm IKE hash: id=1, name=OAKLEY_MD5, hashsize=16
000 algorithm IKE hash: id=2, name=OAKLEY_SHA1, hashsize=20
000 algorithm IKE hash: id=4, name=OAKLEY_SHA2_256, hashsize=32
000 algorithm IKE hash: id=5, name=OAKLEY_SHA2_384, hashsize=48
000 algorithm IKE hash: id=6, name=OAKLEY_SHA2_512, hashsize=64
000 algorithm IKE dh group: id=2, name=OAKLEY_GROUP_MODP1024, bits=1024
000 algorithm IKE dh group: id=5, name=OAKLEY_GROUP_MODP1536, bits=1536
000 algorithm IKE dh group: id=14, name=OAKLEY_GROUP_MODP2048, bits=2048
000 algorithm IKE dh group: id=15, name=OAKLEY_GROUP_MODP3072, bits=3072
000 algorithm IKE dh group: id=16, name=OAKLEY_GROUP_MODP4096, bits=4096
000 algorithm IKE dh group: id=17, name=OAKLEY_GROUP_MODP6144, bits=6144
000 algorithm IKE dh group: id=18, name=OAKLEY_GROUP_MODP8192, bits=8192
000 algorithm IKE dh group: id=22, name=OAKLEY_GROUP_DH22, bits=1024
000 algorithm IKE dh group: id=23, name=OAKLEY_GROUP_DH23, bits=2048
000 algorithm IKE dh group: id=24, name=OAKLEY_GROUP_DH24, bits=2048
000  
000 stats db_ops: {curr_cnt, total_cnt, maxsz} :context={0,0,0} trans={0,0,0} attrs={0,0,0} 
000  
000 Connection list:
000  
000 "97c9d9f5-52a9-4598-a5d3-e99cc57fc930/0x1": 10.1.0.0/24===172.24.4.226<172.24.4.226>---172.24.4.225...172.24.4.225---172.24.4.234<172.24.4.234>===10.2.0.0/24; unrouted; eroute owner: #0
000 "97c9d9f5-52a9-4598-a5d3-e99cc57fc930/0x1":     unoriented; my_ip=unset; their_ip=unset
000 "97c9d9f5-52a9-4598-a5d3-e99cc57fc930/0x1":   xauth info: us:none, them:none,  my_xauthuser=[any]; their_xauthuser=[any]
000 "97c9d9f5-52a9-4598-a5d3-e99cc57fc930/0x1":   modecfg info: us:none, them:none, modecfg policy:push, dns1:unset, dns2:unset, domain:unset, banner:unset;
000 "97c9d9f5-52a9-4598-a5d3-e99cc57fc930/0x1":   labeled_ipsec:no, loopback:no; 
000 "97c9d9f5-52a9-4598-a5d3-e99cc57fc930/0x1":    policy_label:unset; 
000 "97c9d9f5-52a9-4598-a5d3-e99cc57fc930/0x1":   ike_life: 3600s; ipsec_life: 3600s; rekey_margin: 540s; rekey_fuzz: 100%; keyingtries: 0;
000 "97c9d9f5-52a9-4598-a5d3-e99cc57fc930/0x1":   sha2_truncbug:no; initial_contact:no; cisco_unity:no; send_vendorid:no;
000 "97c9d9f5-52a9-4598-a5d3-e99cc57fc930/0x1":   policy: PSK+ENCRYPT+TUNNEL+PFS+SAREF_TRACK+IKE_FRAG_ALLOW; 
000 "97c9d9f5-52a9-4598-a5d3-e99cc57fc930/0x1":   conn_prio: 24,24; interface: ; metric: 0; mtu: unset; sa_prio:auto;
000 "97c9d9f5-52a9-4598-a5d3-e99cc57fc930/0x1":   dpd: action:hold; delay:30; timeout:120; nat-t: force_encaps:no; nat_keepalive:yes; ikev1_natt:both
000 "97c9d9f5-52a9-4598-a5d3-e99cc57fc930/0x1":   newest ISAKMP SA: #0; newest IPsec SA: #0; 
000 "97c9d9f5-52a9-4598-a5d3-e99cc57fc930/0x1":   aliases: 97c9d9f5-52a9-4598-a5d3-e99cc57fc930 
000 "97c9d9f5-52a9-4598-a5d3-e99cc57fc930/0x1":   IKE algorithms wanted: AES_CBC(7)_128-SHA1(2)_000-MODP1536(5)
000 "97c9d9f5-52a9-4598-a5d3-e99cc57fc930/0x1":   IKE algorithms found:  AES_CBC(7)_128-SHA1(2)_160-MODP1536(5)
000 "97c9d9f5-52a9-4598-a5d3-e99cc57fc930/0x1":   ESP algorithms wanted: AES(12)_128-SHA1(2)_000; pfsgroup=MODP1536(5)
000 "97c9d9f5-52a9-4598-a5d3-e99cc57fc930/0x1":   ESP algorithms loaded: none
000  
000 Total IPsec connections: loaded 1, active 0
000  
000 State list:
000  
000 Shunt list:
000

Comment 21 Stefan Berger 2014-11-07 17:11:58 UTC
I updated my machines and reinstalled my two OpenStack installations and now I do not see the above network packets for port 500 (isakmp) anymore. Also pluto does not listen on port 500. I see this message in the log:

2014-11-07 11:56:04.369 ERROR neutron.agent.linux.utils [-] 
Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-8a6e7b73-da83-4e62-9637-7443b4a84f07', 'ipsec', 'whack', '--ctlbase', '/opt/stack/data/neutron/ipsec/8a6e7b73-da83-4e62-9637-7443b4a84f07/var/run/pluto', '--listen']
Exit code: 3
Stdout: '002 listening for IKE messages\n003 no public interfaces found\n002 loading secrets from "/opt/stack/data/neutron/ipsec/8a6e7b73-da83-4e62-9637-7443b4a84f07/etc/ipsec.secrets"\n'
Stderr: ''


I also see the following errors (hadn't seen the one related to /var/run/pluto before)


Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-8a6e7b73-da83-4e62-9637-7443b4a84f07', 'ipsec', 'pluto', '--ctlbase', '/opt/stack/data/neutron/ipsec/8a6e7b73-da83-4e62-9637-7443b4a84f07/var/run/pluto', '--ipsecdir', '/opt/stack/data/neutron/ipsec/8a6e7b73-da83-4e62-9637-7443b4a84f07/etc', '--use-netkey', '--uniqueids', '--nat_traversal', '--secretsfile', '/opt/stack/data/neutron/ipsec/8a6e7b73-da83-4e62-9637-7443b4a84f07/etc/ipsec.secrets', '--virtual_private', '%v4:10.1.0.0/24,%v4:10.2.0.0/24']
Exit code: 0
Stdout: ''
Stderr: 'adjusting ipsec.d to /opt/stack/data/neutron/ipsec/8a6e7b73-da83-4e62-9637-7443b4a84f07/etc\nwarning: option "--nat_traversal" is obsolete; ignored\nwarning: option "--virtual_private" with \'_\' in its name is obsolete; use \'-\'\npluto: warning: chdir("/var/run/pluto") to dumpdir failed (2: No such file or directory) \n' from (pid=10097) execute /opt/stack/neutron/neutron/agent/linux/utils.py:81


Adding those debugging options to ipsec.conf had not resulted in the log file being created. I deactivated the modifications for now.

Here's my latest modification:


diff --git a/neutron/services/vpn/device_drivers/ipsec.py b/neutron/services/vpn
index c19b61e..22895fa 100644
--- a/neutron/services/vpn/device_drivers/ipsec.py
+++ b/neutron/services/vpn/device_drivers/ipsec.py
@@ -328,6 +328,14 @@ class OpenSwanProcess(BaseSwanProcess):
             'ipsec.secrets',
             self.conf.openswan.ipsec_secret_template,
             self.vpnservice)
+        if not os.path.isfile(self.etc_dir + '/cert8.db'):
+            import subprocess
+            ret = subprocess.call(['/bin/certutil',
+                        '-N',
+                        '-d', self.etc_dir,
+                        '--empty-password'
+                        ])
+            LOG.info('ooo ret = %s' % str(ret))
 
     def get_status(self):
         return self._execute([self.binary,
diff --git a/neutron/services/vpn/device_drivers/template/openswan/ipsec.conf.te
index 546e27e..b643cd8 100644
--- a/neutron/services/vpn/device_drivers/template/openswan/ipsec.conf.template
+++ b/neutron/services/vpn/device_drivers/template/openswan/ipsec.conf.template
@@ -2,6 +2,8 @@
 config setup
     nat_traversal=yes
     listen={{vpnservice.external_ip}}
+    # plutodebug=all
+    # plutostderrlog=/tmp/pluto.log
 conn %default
     ikelifetime=480m
     keylife=60m

Comment 22 Paul Wouters 2014-11-27 21:15:09 UTC
be careful testing for only cert8.db as the newer format will use cert8.db. You need to test for both.

Comment 23 Paul Wouters 2015-02-27 04:24:20 UTC
this should work with the latest libreswan builds. Can you confirm the issue has gone away?

Comment 24 Jaroslav Reznik 2015-03-03 16:26:49 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 22 development cycle.
Changing version to '22'.

More information and reason for this action is here:
https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora22

Comment 25 Terry Wilson 2015-04-23 18:25:16 UTC
Paul: There is a patch to specifically add support for libreswan to neutron-vpnaas: https://review.openstack.org/#/c/174299/

I've tested it, and it solves the above issue by calling nss-init. For some reason, I had to manually load the af_key kernel module, otherwise pluto would start, but would not listen on ports 500/4500. Is that normal?

Comment 26 Paul Wouters 2015-04-23 18:56:35 UTC
those are things that normally happen in the service file and via ipsec _stackmanager. eg see the systemd service file:

https://github.com/libreswan/libreswan/blob/master/initsystems/systemd/ipsec.service.in

So yeah, you shoudl call ipsec checknss (if you might need to migrate nss) or ipsec initnss (if you always want to create it from scratch). you can specify -d /path/to/your/ipsec.d directory. I just did note a root check there, which I just removedin upstream.

See: https://github.com/libreswan/libreswan/blob/master/programs/ipsec/ipsec.in

note that this code is currently in flux between libreswan 3.12 and git head as we are adding better support for nss database migration.

The first thing you should do when you want to start pluto is to run (as root):

ipsec _stackmanager start

That will ensure all the kernel modules (like af_key) are loaded.

Comment 27 Fedora End Of Life 2016-07-19 12:19:55 UTC
Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.