Bug 1459343 - bigswitch component is downgraded after yum update on overcloud components
bigswitch component is downgraded after yum update on overcloud components
Status: CLOSED NOTABUG
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-networking-bigswitch (Show other bugs)
9.0 (Mitaka)
Unspecified Unspecified
unspecified Severity medium
: ---
: ---
Assigned To: RHOS Maint
Ofer Blaut
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-06 17:20 EDT by bigswitch
Modified: 2017-06-12 11:07 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-06-09 13:27:31 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description bigswitch 2017-06-06 17:20:08 EDT
Description of problem:
While doing rhosp 9 update, notice bigswitch component is downgraded from Mitaka release to Liberty causing bsnstacklib error in neutron server.

after deploying RHOSP9 with bigswitch plugin. was trying to run yum update via undercloud update on overcloud   nodes as part of RHOSP9 to RHOSP10 upgrade steps. notice after yum update bigswitch plugin is downgraded from Mitaka to Liberty

Version-Release number of selected component (if applicable):
RHOSP9

How reproducible:
always

Before update:
[root@rhosp9-controller-0 heat-admin]# pip show networking-bigswitch
---
Metadata-Version: 1.1
Name: networking-bigswitch
Version: 8.40.8

After update:

root@rhosp9-controller-0 heat-admin]# pip show networking-bigswitch
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
[root@rhosp9-controller-0 heat-admin]# pip show bsnstacklib
---
Metadata-Version: 1.1
Name: bsnstacklib
Version: 2015.3.8

yum log:
Jun 06 13:06:53 Updated: 1:python-networking-bigswitch-2015.3.8-1.el7ost.noarch
Jun 06 13:07:33 Updated: 1:openstack-neutron-bigswitch-agent-2015.3.8-1.el7ost.noarch
Jun 06 13:07:34 Updated: 1:openstack-neutron-bigswitch-lldp-2015.3.8-1.el7ost.noarch


repos enabled:
[root@rhosp9-controller-0 heat-admin]# yum repolist
Loaded plugins: product-id, search-disabled-repos, subscription-manager
repo id                                                                                                                              repo name                                                                                                                                            status
rhel-7-server-extras-rpms/x86_64                                                                                                     Red Hat Enterprise Linux 7 Server - Extras (RPMs)                                                                                                       498
rhel-7-server-openstack-9-director-rpms/7Server/x86_64                                                                               Red Hat OpenStack Platform 9 director for RHEL 7 (RPMs)                                                                                                  65
rhel-7-server-openstack-9-rpms/7Server/x86_64                                                                                        Red Hat OpenStack Platform 9 for RHEL 7 (RPMs)                                                                                                          810
rhel-7-server-rh-common-rpms/7Server/x86_64                                                                                          Red Hat Enterprise Linux 7 Server - RH Common (RPMs)                                                                                                    228
rhel-7-server-rpms/7Server/x86_64                                                                                                    Red Hat Enterprise Linux 7 Server (RPMs)                                                                                                             14,447


neutron-server log
2017-06-06 13:11:01.700 4212 WARNING neutron.agent.securitygroups_rpc [-] Driver configuration doesn't match with enable_security_group
2017-06-06 13:11:01.705 4212 ERROR stevedore.extension [-] Could not load 'bsn_ml2': No module named bigswitch.db
2017-06-06 13:11:01.706 4212 ERROR stevedore.extension [-] No module named bigswitch.db
2017-06-06 13:11:01.706 4212 ERROR stevedore.extension Traceback (most recent call last):
2017-06-06 13:11:01.706 4212 ERROR stevedore.extension   File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 163, in _load_plugins
2017-06-06 13:11:01.706 4212 ERROR stevedore.extension     verify_requirements,
2017-06-06 13:11:01.706 4212 ERROR stevedore.extension   File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 123, in _load_one_plugin
2017-06-06 13:11:01.706 4212 ERROR stevedore.extension     verify_requirements,
2017-06-06 13:11:01.706 4212 ERROR stevedore.extension   File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 186, in _load_one_plugin
2017-06-06 13:11:01.706 4212 ERROR stevedore.extension     plugin = ep.load(require=verify_requirements)
2017-06-06 13:11:01.706 4212 ERROR stevedore.extension   File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2260, in load
2017-06-06 13:11:01.706 4212 ERROR stevedore.extension     entry = __import__(self.module_name, globals(),globals(), ['__name__'])
2017-06-06 13:11:01.706 4212 ERROR stevedore.extension   File "/usr/lib/python2.7/site-packages/bsnstacklib/plugins/ml2/drivers/mech_bigswitch/driver.py", line 39, in <module>
2017-06-06 13:11:01.706 4212 ERROR stevedore.extension     from bsnstacklib.plugins.bigswitch.db import consistency_db as cdb
2017-06-06 13:11:01.706 4212 ERROR stevedore.extension   File "/usr/lib/python2.7/site-packages/bsnstacklib/plugins/bigswitch/db/consistency_db.py", line 27, in <module>
2017-06-06 13:11:01.706 4212 ERROR stevedore.extension     from neutron.plugins.bigswitch.db import consistency_db
2017-06-06 13:11:01.706 4212 ERROR stevedore.extension ImportError: No module named bigswitch.db
2017-06-06 13:11:01.706 4212 ERROR stevedore.extension
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi [-] Couldn't lookup app: neutron
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi Traceback (most recent call last):
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/oslo_service/wsgi.py", line 353, in load_app
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     return deploy.loadapp("config:%s" % self.config_path, name=name)
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     return loadobj(APP, uri, name=name, **kw)
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     return context.create()
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     return self.object_type.invoke(self)
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     **context.local_conf)
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     val = callable(*args, **kw)
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 25, in urlmap_factory
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     app = loader.get_app(app_name, global_conf=global_conf)
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     name=name, global_conf=global_conf).create()
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     return self.object_type.invoke(self)
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     **context.local_conf)
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     val = callable(*args, **kw)
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/neutron/auth.py", line 71, in pipeline_factory
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     app = loader.get_app(pipeline[-1])
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     name=name, global_conf=global_conf).create()
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     return self.object_type.invoke(self)
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     return fix_call(context.object, context.global_conf, **context.local_conf)
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     val = callable(*args, **kw)
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 71, in factory
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     return cls(**local_config)
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 75, in __init__
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     plugin = manager.NeutronManager.get_plugin()
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 233, in get_plugin
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     return weakref.proxy(cls.get_instance().plugin)
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 227, in get_instance
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     cls._create_instance()
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     return f(*args, **kwargs)
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 213, in _create_instance
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     cls._instance = cls()
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 116, in __init__
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     plugin_provider)
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 149, in _get_plugin_instance
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     return plugin_class()
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/neutron/quota/resource_registry.py", line 121, in wrapper
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     return f(*args, **kwargs)
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 154, in __init__
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     self.mechanism_manager = managers.MechanismManager()
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 308, in __init__
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     name_order=True)
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 56, in __init__
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     self._init_plugins(extensions)
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 112, in _init_plugins
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     self.extensions = [self[n] for n in self._names]
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi   File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 284, in __getitem__
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi     return self._extensions_by_name[name]
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi KeyError: 'bsn_ml2'
2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi
Comment 1 Alex Schultz 2017-06-07 16:13:41 EDT
Can you tell me where you got the python-networking-bigswitch 8.40.8 package? As far as I can tell we haven't released one yet. If it's your own package, make sure the epoch is set to 2 in the packaging. This should prevent it from being 'upgraded' to the 2015.3.8 version.
Comment 2 bigswitch 2017-06-07 16:18:40 EDT
Hi Alex,
the 8.40.8 package is packaged into overcloud.qcow2 image using virt-customize before deploying RHOSP 9.
Comment 3 Alex Schultz 2017-06-07 16:29:25 EDT
Who created the package?  The issue is that the packages needs to be created with epoch > 1 to ensure it's not updated to the old one which is what you are experiencing.  This is not an issue with our existing available packages as we do not currently ship any 8.40.x package. I know one is currently being tested but has yet to be released.
Comment 4 bigswitch 2017-06-07 17:25:57 EDT
Hi Alex,

Thanks for the info, that helps.
The spec file here [1] is what creates the new packages. I see that epoch hasn't changed.

Just curious - does the epoch change for all Openstack packages? Since the major version number change from 2015.x.x (year based version number) to 8.x.x (simply release based) was done for all services, including the plugins.

Also, changing epoch should not require any other changes to the spec file, right? I should probably open the PR for it and discuss it there (spec file has a copy in one of RDOs repos, which actually triggers the update).

[1] https://github.com/openstack/networking-bigswitch/blob/stable/mitaka/rhel/python-networking-bigswitch.spec
Comment 5 Alex Schultz 2017-06-08 09:15:14 EDT
Our packaging uses this as it's upstream version: https://github.com/rdo-packages/networking-bigswitch-distgit/blob/rpm-master/python-networking-bigswitch.spec which as you can see has the epoch set to 2. We had to do this for the year based version switch for many projects. The upstream changed these version numbers around the mitaka timeframe so that is why this is needed.
Comment 6 bigswitch 2017-06-09 06:45:03 EDT
That makes sense. From the same repo, I see that Epoch is set to 2 for master, Ocata and Newton, but not for Mitaka[1]. Probably that's why on upgrade from Liberty to Mitaka, the yum update pulls incorrect version.

I'll open a PR to update that. Thanks again Alex!

[1] https://github.com/rdo-packages/networking-bigswitch-distgit/blob/rpm-mitaka/python-networking-bigswitch.spec#L9
Comment 7 bigswitch 2017-06-12 08:41:42 EDT
Opened a code review to update it: https://review.rdoproject.org/r/#/c/7045/
Please take a look :)
Comment 8 bigswitch 2017-06-12 10:46:30 EDT
Hi Mike, Alex,

I have a follow-up query - since Mitaka is EOL, is it possible to not update networking-bigswitch plugin by removing it from the repo config files?

When upgrading from RHOSP 9 to RHOSP 10, there is a yum update that happens on overcloud and it gets stuck - since overcloud is RHOSP 9 and it gets the liberty package, which stalls the neutron-server process due to version mismatch.
Comment 9 Mike Burns 2017-06-12 11:03:54 EDT
Mitaka is EOL at the RDO level.  OSP 9 is still supported.  We've already bumped epoch for this package to 2 in OSP 9 (it's in a package that has not yet been shipped, but is pending)
Comment 10 bigswitch 2017-06-12 11:07:13 EDT
Ah, I misunderstood EOL as no further updates. If the epoch change goes out in the next update for RHOSP 9, that works perfectly well.
Thanks Mike!

Note You need to log in before you can comment on or make changes to this bug.