Bug 1268425 - (pain-in-the-*aas) [Docs] [Upgrade] OSP-7 neutron packages split out service plugins (vpnaas, lbaas) in a way that breaks package upgrades
[Docs] [Upgrade] OSP-7 neutron packages split out service plugins (vpnaas, lb...
Status: CLOSED CURRENTRELEASE
Product: Red Hat OpenStack
Classification: Red Hat
Component: documentation (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
high Severity high
: async
: 7.0 (Kilo)
Assigned To: Deepti Navale
Martin Lopes
: Documentation, ZStream
Depends On:
Blocks: track-osp6-osp7 1277361
  Show dependency treegraph
 
Reported: 2015-10-02 15:24 EDT by Lars Kellogg-Stedman
Modified: 2016-04-26 21:49 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-10-20 20:05:08 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Lars Kellogg-Stedman 2015-10-02 15:24:07 EDT
I am upgrading an OSP-6 environment to OSP-7.  The starting point is
this version of python-neutron:

    # rpm -q python-neutron
    python-neutron-2014.2.3-12.el7ost.noarch

And this configuration in /etc/neutron/neutron.conf:

    # grep loadbalancer /etc/neutron/neutron.conf
    service_plugins =neutron.services.loadbalancer.plugin.LoadBalancerPlugin,neutron.services.l3_router.l3_router_plugin.L3RouterPlugin,neutron.services.firewall.fwaas_plugin.FirewallPlugin

There are two distinct problems with the Neutron packaging in OSP-7 that prevent a simple "yum upgrade" from working successfully:

1. The osp-7 packages move service plugins into separate packages, which are not installed by default when someone performs a 'yum upgrade'.  This results in the following error:

    INFO neutron.manager [-] Loading Plugin: neutron.services.loadbalancer.plugin.LoadBalancerPlugin
    ERROR neutron.common.config [-] Unable to load neutron from configuration file /usr/share/neutron/api-paste.ini.
    TRACE neutron.common.config Traceback (most recent call last):
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/common/config.py", line 227, in load_paste_app
    TRACE neutron.common.config     app = deploy.loadapp("config:%s" % config_path, name=app_name)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
    TRACE neutron.common.config     return loadobj(APP, uri, name=name, **kw)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
    TRACE neutron.common.config     return context.create()
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
    TRACE neutron.common.config     return self.object_type.invoke(self)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
    TRACE neutron.common.config     **context.local_conf)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call
    TRACE neutron.common.config     val = callable(*args, **kw)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 25, in urlmap_factory
    TRACE neutron.common.config     app = loader.get_app(app_name, global_conf=global_conf)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
    TRACE neutron.common.config     name=name, global_conf=global_conf).create()
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
    TRACE neutron.common.config     return self.object_type.invoke(self)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
    TRACE neutron.common.config     **context.local_conf)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call
    TRACE neutron.common.config     val = callable(*args, **kw)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/auth.py", line 71, in pipeline_factory
    TRACE neutron.common.config     app = loader.get_app(pipeline[-1])
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
    TRACE neutron.common.config     name=name, global_conf=global_conf).create()
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
    TRACE neutron.common.config     return self.object_type.invoke(self)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
    TRACE neutron.common.config     return fix_call(context.object, context.global_conf, **context.local_conf)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call
    TRACE neutron.common.config     val = callable(*args, **kw)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 71, in factory
    TRACE neutron.common.config     return cls(**local_config)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 75, in __init__
    TRACE neutron.common.config     plugin = manager.NeutronManager.get_plugin()
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 222, in get_plugin
    TRACE neutron.common.config     return weakref.proxy(cls.get_instance().plugin)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 216, in get_instance
    TRACE neutron.common.config     cls._create_instance()
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner
    TRACE neutron.common.config     return f(*args, **kwargs)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 202, in _create_instance
    TRACE neutron.common.config     cls._instance = cls()
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 128, in __init__
    TRACE neutron.common.config     self._load_service_plugins()
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 175, in _load_service_plugins
    TRACE neutron.common.config     provider)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 133, in _get_plugin_instance
    TRACE neutron.common.config     mgr = driver.DriverManager(namespace, plugin_provider)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/stevedore/driver.py", line 45, in __init__
    TRACE neutron.common.config     verify_requirements=verify_requirements,
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 55, in __init__
    TRACE neutron.common.config     verify_requirements)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 170, in _load_plugins
    TRACE neutron.common.config     self._on_load_failure_callback(self, ep, err)
    TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/stevedore/driver.py", line 50, in _default_on_load_failure
    TRACE neutron.common.config     raise err
    TRACE neutron.common.config ImportError: No module named neutron_lbaas.services.loadbalancer.plugin

This can be corrected by install the openstack-neutron-{lbaas,vpnaas}
packages:

    # yum install openstack-neutron-{lbaas,vpnaas}

These packages should probably have an appropriate "Obsoletes:"
dependency to ensure that they will be installed when upgrading from
the OSP-6 packages.

2. The OSP-7 plugin packages install new configuration files, such as /etc/neutron/neutron_lbaas.conf.  This by itself isn't a problem, but we also start neutron-server like this:

    /usr/bin/neutron-server \
      --config-file /usr/share/neutron/neutron-dist.conf \
      --config-dir /usr/share/neutron/server \
      --config-file /etc/neutron/neutron.conf \
      --config-file /etc/neutron/plugin.ini \
      --config-dir /etc/neutron/conf.d/common \
      --config-dir /etc/neutron/conf.d/neutron-server \
      --log-file /var/log/neutron/server.log

That's three '--config-file's and three '--config-dir's.  The
directory /usr/share/neutron/server contains symlinks to the plugin
configurations:

    # ls -l /usr/share/neutron/server/
    total 0
    lrwxrwxrwx. 1 root root 31 Oct  2 14:38 neutron_lbaas.conf -> /etc/neutron/neutron_lbaas.conf
    lrwxrwxrwx. 1 root root 32 Oct  2 14:38 neutron_vpnaas.conf -> /etc/neutron/neutron_vpnaas.conf

So when neutron-server starts it will read both
/etc/neutron/neutron.conf as well as (for exampel)
/etc/neutron/neutron_lbaas.conf, resulting in this error:

    2015-10-02 15:23:10.296 13410 TRACE neutron Invalid: Driver neutron_lbaas.services.loadbalancer.drivers.haproxy.plugin_driver.HaproxyOnHostPluginDriver is not unique across providers
Comment 2 Assaf Muller 2015-10-02 15:26:53 EDT
Alerting Ihar.
Comment 3 Ihar Hrachyshka 2015-10-05 04:01:44 EDT
Lars, in the meantime, please attach configuration files that you have.
Comment 4 Ihar Hrachyshka 2015-10-05 09:10:52 EDT
No need for config examples, it's clear what's going on. Discussion on best solution is ongoing.
Comment 5 Ihar Hrachyshka 2015-10-07 04:56:14 EDT
OK, moving the bug to documentation component.

Upgrade docs for OSP6 -> OSP7 should be updated to:

- on controller side (the nodes that run neutron-server), suggest installing openstack-neutron-{vpn,fw,lb}aas packages for corresponding service_plugins enabled in neutron.conf;

- on controller side, for each service_provider entry for {lb,vpn}aas services, move neutron.conf entries into corresponding neutron_{lb,vpn}aas.conf files located in /etc/neutron;

- on every node that runs lbaas agents, install openstack-neutron-lbaas.
Comment 6 Andrew Dahms 2015-10-07 20:31:12 EDT
Assigning to Deepti for review.
Comment 10 Andrew Dahms 2015-10-11 22:17:36 EDT
Assigning Martin as the QA contact.

Martin - could you review the changes made for this bug?
Comment 11 Martin Lopes 2015-10-12 23:15:21 EDT
Added some minor edits. lgtm. 

Moving to 'verified'.

Note You need to log in before you can comment on or make changes to this bug.