Bug 1268425 (pain-in-the-*aas)

Summary:	[Docs] [Upgrade] OSP-7 neutron packages split out service plugins (vpnaas, lbaas) in a way that breaks package upgrades
Product:	Red Hat OpenStack	Reporter:	Lars Kellogg-Stedman <lars>
Component:	documentation	Assignee:	Deepti Navale <dnavale>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Martin Lopes <mlopes>
Severity:	high	Docs Contact:
Priority:	high
Version:	7.0 (Kilo)	CC:	adahms, amuller, chrisw, ihrachys, lars, mlopes, nyechiel, yeylon
Target Milestone:	async	Keywords:	Documentation, ZStream
Target Release:	7.0 (Kilo)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-10-21 00:05:08 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1249564, 1277361

Description Lars Kellogg-Stedman 2015-10-02 19:24:07 UTC

I am upgrading an OSP-6 environment to OSP-7.  The starting point is
this version of python-neutron:

    # rpm -q python-neutron
    python-neutron-2014.2.3-12.el7ost.noarch

And this configuration in /etc/neutron/neutron.conf:

    # grep loadbalancer /etc/neutron/neutron.conf
    service_plugins =neutron.services.loadbalancer.plugin.LoadBalancerPlugin,neutron.services.l3_router.l3_router_plugin.L3RouterPlugin,neutron.services.firewall.fwaas_plugin.FirewallPlugin

There are two distinct problems with the Neutron packaging in OSP-7 that prevent a simple "yum upgrade" from working successfully:

1. The osp-7 packages move service plugins into separate packages, which are not installed by default when someone performs a 'yum upgrade'.  This results in the following error:

    INFO neutron.manager [-] Loading Plugin: neutron.services.loadbalancer.plugin.LoadBalancerPlugin
    ERROR neutron.common.config [-] Unable to load neutron from configuration file /usr/share/neutron/api-paste.ini.
    TRACE neutron.common.config Traceback (most recent call last):
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/common/config.py", line 227, in load_paste_app
    TRACE neutron.common.config     app = deploy.loadapp("config:%s" % config_path, name=app_name)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
    TRACE neutron.common.config     return loadobj(APP, uri, name=name, **kw)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
    TRACE neutron.common.config     return context.create()
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
    TRACE neutron.common.config     return self.object_type.invoke(self)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
    TRACE neutron.common.config     **context.local_conf)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call
    TRACE neutron.common.config     val = callable(*args, **kw)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 25, in urlmap_factory
    TRACE neutron.common.config     app = loader.get_app(app_name, global_conf=global_conf)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
    TRACE neutron.common.config     name=name, global_conf=global_conf).create()
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
    TRACE neutron.common.config     return self.object_type.invoke(self)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
    TRACE neutron.common.config     **context.local_conf)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call
    TRACE neutron.common.config     val = callable(*args, **kw)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/auth.py", line 71, in pipeline_factory
    TRACE neutron.common.config     app = loader.get_app(pipeline[-1])
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
    TRACE neutron.common.config     name=name, global_conf=global_conf).create()
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
    TRACE neutron.common.config     return self.object_type.invoke(self)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
    TRACE neutron.common.config     return fix_call(context.object, context.global_conf, **context.local_conf)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call
    TRACE neutron.common.config     val = callable(*args, **kw)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 71, in factory
    TRACE neutron.common.config     return cls(**local_config)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 75, in __init__
    TRACE neutron.common.config     plugin = manager.NeutronManager.get_plugin()
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 222, in get_plugin
    TRACE neutron.common.config     return weakref.proxy(cls.get_instance().plugin)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 216, in get_instance
    TRACE neutron.common.config     cls._create_instance()
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner
    TRACE neutron.common.config     return f(*args, **kwargs)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 202, in _create_instance
    TRACE neutron.common.config     cls._instance = cls()
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 128, in __init__
    TRACE neutron.common.config     self._load_service_plugins()
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 175, in _load_service_plugins
    TRACE neutron.common.config     provider)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 133, in _get_plugin_instance
    TRACE neutron.common.config     mgr = driver.DriverManager(namespace, plugin_provider)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/stevedore/driver.py", line 45, in __init__
    TRACE neutron.common.config     verify_requirements=verify_requirements,
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 55, in __init__
    TRACE neutron.common.config     verify_requirements)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 170, in _load_plugins
    TRACE neutron.common.config     self._on_load_failure_callback(self, ep, err)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/stevedore/driver.py", line 50, in _default_on_load_failure
    TRACE neutron.common.config     raise err
    TRACE neutron.common.config ImportError: No module named neutron_lbaas.services.loadbalancer.plugin

This can be corrected by install the openstack-neutron-{lbaas,vpnaas}
packages:

    # yum install openstack-neutron-{lbaas,vpnaas}

These packages should probably have an appropriate "Obsoletes:"
dependency to ensure that they will be installed when upgrading from
the OSP-6 packages.

2. The OSP-7 plugin packages install new configuration files, such as /etc/neutron/neutron_lbaas.conf.  This by itself isn't a problem, but we also start neutron-server like this:

    /usr/bin/neutron-server \
      --config-file /usr/share/neutron/neutron-dist.conf \
      --config-dir /usr/share/neutron/server \
      --config-file /etc/neutron/neutron.conf \
      --config-file /etc/neutron/plugin.ini \
      --config-dir /etc/neutron/conf.d/common \
      --config-dir /etc/neutron/conf.d/neutron-server \
      --log-file /var/log/neutron/server.log

That's three '--config-file's and three '--config-dir's.  The
directory /usr/share/neutron/server contains symlinks to the plugin
configurations:

    # ls -l /usr/share/neutron/server/
    total 0
    lrwxrwxrwx. 1 root root 31 Oct  2 14:38 neutron_lbaas.conf -> /etc/neutron/neutron_lbaas.conf
    lrwxrwxrwx. 1 root root 32 Oct  2 14:38 neutron_vpnaas.conf -> /etc/neutron/neutron_vpnaas.conf

So when neutron-server starts it will read both
/etc/neutron/neutron.conf as well as (for exampel)
/etc/neutron/neutron_lbaas.conf, resulting in this error:

    2015-10-02 15:23:10.296 13410 TRACE neutron Invalid: Driver neutron_lbaas.services.loadbalancer.drivers.haproxy.plugin_driver.HaproxyOnHostPluginDriver is not unique across providers

Comment 2 Assaf Muller 2015-10-02 19:26:53 UTC

Alerting Ihar.

Comment 3 Ihar Hrachyshka 2015-10-05 08:01:44 UTC

Lars, in the meantime, please attach configuration files that you have.

Comment 4 Ihar Hrachyshka 2015-10-05 13:10:52 UTC

No need for config examples, it's clear what's going on. Discussion on best solution is ongoing.

Comment 5 Ihar Hrachyshka 2015-10-07 08:56:14 UTC

OK, moving the bug to documentation component.

Upgrade docs for OSP6 -> OSP7 should be updated to:

- on controller side (the nodes that run neutron-server), suggest installing openstack-neutron-{vpn,fw,lb}aas packages for corresponding service_plugins enabled in neutron.conf;

- on controller side, for each service_provider entry for {lb,vpn}aas services, move neutron.conf entries into corresponding neutron_{lb,vpn}aas.conf files located in /etc/neutron;

- on every node that runs lbaas agents, install openstack-neutron-lbaas.

Comment 6 Andrew Dahms 2015-10-08 00:31:12 UTC

Assigning to Deepti for review.

Comment 10 Andrew Dahms 2015-10-12 02:17:36 UTC

Assigning Martin as the QA contact.

Martin - could you review the changes made for this bug?

Comment 11 Martin Lopes 2015-10-13 03:15:21 UTC

Added some minor edits. lgtm. 

Moving to 'verified'.

Comment 12 Deepti Navale 2015-10-21 00:05:08 UTC

Content has been published: 

https://access.redhat.com/documentation/en/red-hat-enterprise-linux-openstack-platform/version-7/upgrading-openstack/#section_upgrade-one-by-one-HA