Description of problem: While doing rhosp 9 update, notice bigswitch component is downgraded from Mitaka release to Liberty causing bsnstacklib error in neutron server. after deploying RHOSP9 with bigswitch plugin. was trying to run yum update via undercloud update on overcloud nodes as part of RHOSP9 to RHOSP10 upgrade steps. notice after yum update bigswitch plugin is downgraded from Mitaka to Liberty Version-Release number of selected component (if applicable): RHOSP9 How reproducible: always Before update: [root@rhosp9-controller-0 heat-admin]# pip show networking-bigswitch --- Metadata-Version: 1.1 Name: networking-bigswitch Version: 8.40.8 After update: root@rhosp9-controller-0 heat-admin]# pip show networking-bigswitch You are using pip version 8.1.2, however version 9.0.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. [root@rhosp9-controller-0 heat-admin]# pip show bsnstacklib --- Metadata-Version: 1.1 Name: bsnstacklib Version: 2015.3.8 yum log: Jun 06 13:06:53 Updated: 1:python-networking-bigswitch-2015.3.8-1.el7ost.noarch Jun 06 13:07:33 Updated: 1:openstack-neutron-bigswitch-agent-2015.3.8-1.el7ost.noarch Jun 06 13:07:34 Updated: 1:openstack-neutron-bigswitch-lldp-2015.3.8-1.el7ost.noarch repos enabled: [root@rhosp9-controller-0 heat-admin]# yum repolist Loaded plugins: product-id, search-disabled-repos, subscription-manager repo id repo name status rhel-7-server-extras-rpms/x86_64 Red Hat Enterprise Linux 7 Server - Extras (RPMs) 498 rhel-7-server-openstack-9-director-rpms/7Server/x86_64 Red Hat OpenStack Platform 9 director for RHEL 7 (RPMs) 65 rhel-7-server-openstack-9-rpms/7Server/x86_64 Red Hat OpenStack Platform 9 for RHEL 7 (RPMs) 810 rhel-7-server-rh-common-rpms/7Server/x86_64 Red Hat Enterprise Linux 7 Server - RH Common (RPMs) 228 rhel-7-server-rpms/7Server/x86_64 Red Hat Enterprise Linux 7 Server (RPMs) 14,447 neutron-server log 2017-06-06 13:11:01.700 4212 WARNING neutron.agent.securitygroups_rpc [-] Driver configuration doesn't match with enable_security_group 2017-06-06 13:11:01.705 4212 ERROR stevedore.extension [-] Could not load 'bsn_ml2': No module named bigswitch.db 2017-06-06 13:11:01.706 4212 ERROR stevedore.extension [-] No module named bigswitch.db 2017-06-06 13:11:01.706 4212 ERROR stevedore.extension Traceback (most recent call last): 2017-06-06 13:11:01.706 4212 ERROR stevedore.extension File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 163, in _load_plugins 2017-06-06 13:11:01.706 4212 ERROR stevedore.extension verify_requirements, 2017-06-06 13:11:01.706 4212 ERROR stevedore.extension File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 123, in _load_one_plugin 2017-06-06 13:11:01.706 4212 ERROR stevedore.extension verify_requirements, 2017-06-06 13:11:01.706 4212 ERROR stevedore.extension File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 186, in _load_one_plugin 2017-06-06 13:11:01.706 4212 ERROR stevedore.extension plugin = ep.load(require=verify_requirements) 2017-06-06 13:11:01.706 4212 ERROR stevedore.extension File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2260, in load 2017-06-06 13:11:01.706 4212 ERROR stevedore.extension entry = __import__(self.module_name, globals(),globals(), ['__name__']) 2017-06-06 13:11:01.706 4212 ERROR stevedore.extension File "/usr/lib/python2.7/site-packages/bsnstacklib/plugins/ml2/drivers/mech_bigswitch/driver.py", line 39, in <module> 2017-06-06 13:11:01.706 4212 ERROR stevedore.extension from bsnstacklib.plugins.bigswitch.db import consistency_db as cdb 2017-06-06 13:11:01.706 4212 ERROR stevedore.extension File "/usr/lib/python2.7/site-packages/bsnstacklib/plugins/bigswitch/db/consistency_db.py", line 27, in <module> 2017-06-06 13:11:01.706 4212 ERROR stevedore.extension from neutron.plugins.bigswitch.db import consistency_db 2017-06-06 13:11:01.706 4212 ERROR stevedore.extension ImportError: No module named bigswitch.db 2017-06-06 13:11:01.706 4212 ERROR stevedore.extension 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi [-] Couldn't lookup app: neutron 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi Traceback (most recent call last): 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/oslo_service/wsgi.py", line 353, in load_app 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi return deploy.loadapp("config:%s" % self.config_path, name=name) 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi return loadobj(APP, uri, name=name, **kw) 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi return context.create() 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi return self.object_type.invoke(self) 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi **context.local_conf) 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi val = callable(*args, **kw) 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 25, in urlmap_factory 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi app = loader.get_app(app_name, global_conf=global_conf) 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi name=name, global_conf=global_conf).create() 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi return self.object_type.invoke(self) 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi **context.local_conf) 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi val = callable(*args, **kw) 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/neutron/auth.py", line 71, in pipeline_factory 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi app = loader.get_app(pipeline[-1]) 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi name=name, global_conf=global_conf).create() 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi return self.object_type.invoke(self) 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi return fix_call(context.object, context.global_conf, **context.local_conf) 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi val = callable(*args, **kw) 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 71, in factory 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi return cls(**local_config) 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 75, in __init__ 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi plugin = manager.NeutronManager.get_plugin() 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 233, in get_plugin 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi return weakref.proxy(cls.get_instance().plugin) 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 227, in get_instance 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi cls._create_instance() 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi return f(*args, **kwargs) 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 213, in _create_instance 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi cls._instance = cls() 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 116, in __init__ 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi plugin_provider) 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 149, in _get_plugin_instance 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi return plugin_class() 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/neutron/quota/resource_registry.py", line 121, in wrapper 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi return f(*args, **kwargs) 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 154, in __init__ 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi self.mechanism_manager = managers.MechanismManager() 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 308, in __init__ 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi name_order=True) 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 56, in __init__ 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi self._init_plugins(extensions) 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 112, in _init_plugins 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi self.extensions = [self[n] for n in self._names] 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 284, in __getitem__ 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi return self._extensions_by_name[name] 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi KeyError: 'bsn_ml2' 2017-06-06 13:11:01.727 4212 ERROR oslo.service.wsgi
Can you tell me where you got the python-networking-bigswitch 8.40.8 package? As far as I can tell we haven't released one yet. If it's your own package, make sure the epoch is set to 2 in the packaging. This should prevent it from being 'upgraded' to the 2015.3.8 version.
Hi Alex, the 8.40.8 package is packaged into overcloud.qcow2 image using virt-customize before deploying RHOSP 9.
Who created the package? The issue is that the packages needs to be created with epoch > 1 to ensure it's not updated to the old one which is what you are experiencing. This is not an issue with our existing available packages as we do not currently ship any 8.40.x package. I know one is currently being tested but has yet to be released.
Hi Alex, Thanks for the info, that helps. The spec file here [1] is what creates the new packages. I see that epoch hasn't changed. Just curious - does the epoch change for all Openstack packages? Since the major version number change from 2015.x.x (year based version number) to 8.x.x (simply release based) was done for all services, including the plugins. Also, changing epoch should not require any other changes to the spec file, right? I should probably open the PR for it and discuss it there (spec file has a copy in one of RDOs repos, which actually triggers the update). [1] https://github.com/openstack/networking-bigswitch/blob/stable/mitaka/rhel/python-networking-bigswitch.spec
Our packaging uses this as it's upstream version: https://github.com/rdo-packages/networking-bigswitch-distgit/blob/rpm-master/python-networking-bigswitch.spec which as you can see has the epoch set to 2. We had to do this for the year based version switch for many projects. The upstream changed these version numbers around the mitaka timeframe so that is why this is needed.
That makes sense. From the same repo, I see that Epoch is set to 2 for master, Ocata and Newton, but not for Mitaka[1]. Probably that's why on upgrade from Liberty to Mitaka, the yum update pulls incorrect version. I'll open a PR to update that. Thanks again Alex! [1] https://github.com/rdo-packages/networking-bigswitch-distgit/blob/rpm-mitaka/python-networking-bigswitch.spec#L9
Opened a code review to update it: https://review.rdoproject.org/r/#/c/7045/ Please take a look :)
Hi Mike, Alex, I have a follow-up query - since Mitaka is EOL, is it possible to not update networking-bigswitch plugin by removing it from the repo config files? When upgrading from RHOSP 9 to RHOSP 10, there is a yum update that happens on overcloud and it gets stuck - since overcloud is RHOSP 9 and it gets the liberty package, which stalls the neutron-server process due to version mismatch.
Mitaka is EOL at the RDO level. OSP 9 is still supported. We've already bumped epoch for this package to 2 in OSP 9 (it's in a package that has not yet been shipped, but is pending)
Ah, I misunderstood EOL as no further updates. If the epoch change goes out in the next update for RHOSP 9, that works perfectly well. Thanks Mike!