Bug 1329905

Summary: High CPU load VPNaaS and libreswan (certutil)
Product: Red Hat OpenStack Reporter: kevin.olbrich
Component: openstack-neutron-vpnaasAssignee: Assaf Muller <amuller>
Status: CLOSED WONTFIX QA Contact: Toni Freger <tfreger>
Severity: high Docs Contact:
Priority: low    
Version: 8.0 (Liberty)CC: amuller, apevec, lhh, nyechiel
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: 10.0 (Newton)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-09-14 17:42:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
vpn-agent log
none
ps -ax none

Description kevin.olbrich 2016-04-24 16:52:06 UTC 
Description of problem:
When installing libreswan and neutron-vpnaas-agent, I get 100% cpu load on all cores. Problem exists in Liberty and Mitaka deployments via Packstack.

The process consuming the cpu-cycles is "certutil":

certutil -N -d sql:/etc/ipsec.d --empty-password
It spawns serveral times, sometimes the process dies when swap runs full. Running this command on root shell works flawlessly but the problem returns.


Version-Release number of selected component (if applicable):
[root@testnode1 ~]# rpm -qa | grep openstack
openstack-nova-cert-13.0.0-1.el7.noarch
openstack-neutron-lbaas-8.0.0-1.el7.noarch
openstack-swift-plugin-swift3-1.10-1.el7.noarch
openstack-gnocchi-metricd-2.0.2-1.el7.noarch
openstack-ceilometer-collector-6.0.0-2.el7.noarch
openstack-aodh-listener-2.0.0-1.el7.noarch
centos-release-openstack-mitaka-1-2.el7.centos.noarch
openstack-packstack-puppet-8.0.0-0.7.0rc2.el7.noarch
openstack-keystone-9.0.0-1.el7.noarch
python-django-openstack-auth-2.2.0-1.el7.noarch
openstack-neutron-vpnaas-8.0.0-1.el7.noarch
openstack-swift-account-2.6.0-1.el7.noarch
openstack-cinder-8.0.0-1.el7.noarch
openstack-gnocchi-api-2.0.2-1.el7.noarch
openstack-ceilometer-notification-6.0.0-2.el7.noarch
openstack-aodh-common-2.0.0-1.el7.noarch
openstack-nova-compute-13.0.0-1.el7.noarch
openstack-neutron-metering-agent-8.0.0-1.el7.noarch
openstack-glance-12.0.0-1.el7.noarch
openstack-ceilometer-common-6.0.0-2.el7.noarch
openstack-gnocchi-carbonara-2.0.2-1.el7.noarch
openstack-ceilometer-compute-6.0.0-2.el7.noarch
python2-openstacksdk-0.8.3-1.el7.noarch
openstack-selinux-0.6.58-1.el7.noarch
openstack-utils-2015.2-1.el7.noarch
openstack-nova-common-13.0.0-1.el7.noarch
openstack-ceilometer-polling-6.0.0-2.el7.noarch
openstack-nova-conductor-13.0.0-1.el7.noarch
openstack-nova-scheduler-13.0.0-1.el7.noarch
openstack-neutron-common-8.0.0-1.el7.noarch
openstack-neutron-ml2-8.0.0-1.el7.noarch
openstack-neutron-openvswitch-8.0.0-1.el7.noarch
openstack-swift-object-2.6.0-1.el7.noarch
openstack-gnocchi-common-2.0.2-1.el7.noarch
openstack-gnocchi-statsd-2.0.2-1.el7.noarch
openstack-ceilometer-api-6.0.0-2.el7.noarch
openstack-aodh-notifier-2.0.0-1.el7.noarch
openstack-neutron-fwaas-8.0.0-3.el7.noarch
openstack-swift-2.6.0-1.el7.noarch
openstack-gnocchi-indexer-sqlalchemy-2.0.2-1.el7.noarch
openstack-neutron-8.0.0-1.el7.noarch
openstack-swift-proxy-2.6.0-1.el7.noarch
openstack-ceilometer-central-6.0.0-2.el7.noarch
openstack-aodh-api-2.0.0-1.el7.noarch
openstack-nova-console-13.0.0-1.el7.noarch
python-openstackclient-2.2.0-1.el7.noarch
openstack-puppet-modules-8.0.0-1.el7.noarch
openstack-packstack-8.0.0-0.7.0rc2.el7.noarch
openstack-nova-novncproxy-13.0.0-1.el7.noarch
openstack-dashboard-9.0.0-1.el7.noarch
openstack-swift-container-2.6.0-1.el7.noarch
openstack-nova-api-13.0.0-1.el7.noarch
openstack-aodh-evaluator-2.0.0-1.el7.noarch

[root@testnode1 ~]# rpm -qa | grep libresw
libreswan-3.15-5.el7_1.x86_64

How reproducible:
100% of deployments

Steps to Reproduce:
1. Run Packstack All-In-One on CentOS with VPNaaS enabled
2. Create VPN-SiteToSite-VPN
3. Restart neutron-vpn-agent

Actual results:
100% load, no VPN

Expected results:
normal load, working VPN

Additional info:
Comment 2 kevin.olbrich 2016-04-24 17:00:17 UTC 
Created attachment 1150200 [details]
vpn-agent log
Comment 3 kevin.olbrich 2016-04-24 17:03:38 UTC 
Created attachment 1150201 [details]
ps -ax
Comment 4 kevin.olbrich 2016-04-24 18:05:49 UTC 
[root@testnode1 ~]# ps -ax | grep certutil
 6042 ?        R     14:09 certutil -N -d sql:/etc/ipsec.d --empty-password
 8457 ?        R    156:34 certutil -N -d sql:/etc/ipsec.d --empty-password
12071 ?        R    140:10 certutil -N -d sql:/etc/ipsec.d --empty-password
15747 ?        R    122:42 certutil -N -d sql:/etc/ipsec.d --empty-password
20058 ?        R    100:45 certutil -N -d sql:/etc/ipsec.d --empty-password
26105 ?        R     73:14 certutil -N -d sql:/etc/ipsec.d --empty-password
32342 ?        R     44:09 certutil -N -d sql:/etc/ipsec.d --empty-password
Comment 5 kevin.olbrich 2016-04-24 20:07:08 UTC 
Just tested OpenSwan from EPEL and it seems to work.

vpn_agent.ini:
vpn_device_driver=neutron_vpnaas.services.vpn.device_drivers.fedora_strongswan_ipsec.FedoraStrongSwanDriver
Comment 6 kevin.olbrich 2016-04-24 20:33:10 UTC 
(In reply to kevin.olbrich from comment #5)
> Just tested OpenSwan from EPEL and it seems to work.
> 
> vpn_agent.ini:
> vpn_device_driver=neutron_vpnaas.services.vpn.device_drivers.
> fedora_strongswan_ipsec.FedoraStrongSwanDriver

Sorry, I meant StrongSwan.
Comment 7 Assaf Muller 2016-09-14 17:43:33 UTC 
VPNaaS does not align with our team capacity and prioritization. I'd rather mark this as won't fixed and set expectations rather than let the bug rot open for years.