Bug 1258106
| Summary: | ml2 plugin rpm package is not properly installed by virt-customize | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | bigswitch <rhosp-bugs-internal> |
| Component: | rhosp-director | Assignee: | Jiri Stransky <jstransk> |
| Status: | CLOSED WORKSFORME | QA Contact: | yeylon <yeylon> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.0 (Kilo) | CC: | jstransk, mandreou, mburns, rhel-osp-director-maint, srevivo |
| Target Milestone: | --- | ||
| Target Release: | 8.0 (Liberty) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-09-04 18:20:12 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
(In reply to bigswitch from comment #0) > However, problem happens when we use virt-customize. Following is the > steps to constantly reproduce the problem: > > 1. on director node use following command to customize image > $ virt-customize --upload > python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm:/root/python- > networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm --upload > openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm:/root/ > openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm > --run-command "rpm -ivh > /root/python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm;rpm -ivh > /root/openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm; > systemctl enable neutron-bsn-lldp.service" -a overcloud-full.qcow2 > > 2. upload image and install overcloud as usual FYI, for this bug we are concentrating on getting the package installed correctly, you can debug much faster by just launching the (virt-customized) overcloud-full.qcow2 image rather than doing a full deploy. nova boot --flavor baremetal --image overcloud-full --key-name default test (I believe ssh cloud-user@IP is what you need) > > 3. Since at this point, ml2_conf.ini is not populated with bigswitch > specific configuration due to the first bug. We use "puppet apply ml2.pp" to > make changes to ml2_conf.ini. > > 4. on controller node, > $ sudo systemctl restart neutron-server we use pacemaker to manage the openstack-* services so you should probably use that to restart like: pcs resource restart neutron-server > > 5. /var/log/neutron/server.log will show following error. The line number > may twisted a little bit since we added some debug messages. The point is > that the bsn_ml2 mechanism_driver is not picked up. > 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config Traceback (most > recent call last): will do my best to try this before we talk this afternoon. Where can I get the rpms exact that you are using above from please? thanks, marios (In reply to marios from comment #3) > (In reply to bigswitch from comment #0) > will do my best to try this before we talk this afternoon. Where can I get > the rpms exact that you are using above from please? fyi for context, the bug @ https://bugzilla.redhat.com/show_bug.cgi?id=1254828 tracks the packaging effort and links to rpm there (In reply to marios from comment #4) > (In reply to marios from comment #3) > > (In reply to bigswitch from comment #0) > > will do my best to try this before we talk this afternoon. Where can I get > > the rpms exact that you are using above from please? > > fyi for context, the bug @ > https://bugzilla.redhat.com/show_bug.cgi?id=1254828 tracks the packaging > effort and links to rpm there Though I can't quickly see a link to the openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm in that packaging tracking bug sorry for the noise is in the description (am poking at this) 1. the command we used to deploy overcloud $ openstack overcloud deploy -e /home/stack/network-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/templates/bigswitch-config.yaml --neutron-network-type vlan --neutron-network-vlan-ranges datacentre:50:90 --neutron-disable-tunneling --compute-scale 1 --control-scale 3 --ceph-storage-scale 0 --templates --control-flavor control --compute-flavor compute --ntp-server 0.rhel.pool.ntp.org --debug 2>&1 | tee haha.log 2. the command we used to patch image $ virt-customize --upload python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm:/root/python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm --upload openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm:/root/openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm --run-command "rpm -ivh /root/python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm;rpm -ivh /root/openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm;systemctl enable neutron-bsn-lldp.service" -a overcloud-full.qcow2 To summarize what we found about this bug 1. If patch image with Big Switch rpm and then install overcloud, neutron-server can NOT find bsn_ml2 entry point. 2. If patch image with Big Switch rpm, install overcloud and then put "bsn_ml2 = bsnstacklib.plugins.ml2.drivers.mech_bigswitch.driver:BigSwitchMechanismDriver" into "/lib/python2.7/site-packages/neutron-2015.1.0-py2.7.egg-info/entry_points.txt" [neutron.ml2.mechanism_drivers] section, neutron-server can NOT find bsn_ml2 entry point. 3. If install overcloud first and then rpm -ivh Big Switch package, neutron-server CAN find bsn_ml2 entry point. The exact command we used to reproduce this problem is as following 1. deploy overcloud $ openstack overcloud deploy -e /home/stack/network-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/templates/bigswitch-config.yaml --neutron-network-type vlan --neutron-network-vlan-ranges datacentre:50:90 --neutron-disable-tunneling --compute-scale 1 --control-scale 3 --ceph-storage-scale 0 --templates --control-flavor control --compute-flavor compute --ntp-server 0.rhel.pool.ntp.org --debug 2>&1 | tee haha.log 2. patch image virt-customize --upload python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm:/root/python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm --upload openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm:/root/openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm --run-command "rpm -ivh /root/python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm;rpm -ivh /root/openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm;systemctl enable neutron-bsn-lldp.service" -a overcloud-full.qcow2 FYI I am having some luck with virt-customize -a overcloud-full.qcow2 --firstboot install_bigswitch_packages.sh where the firstboot script is simply like rpm -ivh --force /root/python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm rpm -ivh --force /root/openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm; systemctl enable neutron-bsn-lldp.service (the --force here is because it was already installed on that image from the earlier invocation of virt-customize as in the script above) I will test again from scratch for sanity check and update The latest version of the virt-customize command at https://github.com/jistr/tripleo-bigswitch-temporary-setup/pull/1 fixes this I believe. It uses --firstboot-command to install the packages and enable the lldp service. On deploy, everything looks installed ok like [2] below. I then did the neutron server restart: vi /etc/neutron/plugins/ml2/ml2_conf.ini # (^^^ mechanism_drivers =openvswitch,bsn_ml2) pcs resource restart neutron-server neutron-server still won't start, but at least not because of the bsn_ml2 key error. I see [1] for example and a long trace like [0] (if that trace is useful you may want to copy it as the paste will expire) [0] http://paste.openstack.org/raw/444151 [1] 2015-09-03 08:31:33.264 15477 INFO neutron.plugins.ml2.managers [-] Initializing mechanism driver 'bsn_ml2' 2015-09-03 08:31:33.264 15477 DEBUG bsnstacklib.plugins.ml2.drivers.mech_bigswitch.driver [-] Initializing driver initialize /usr/lib/python2.7/site-packages/bsnstacklib/plugins/ml2/drivers/mech_bigswitch/driver.py:62 [2] [root@overcloud-controller-0 heat-admin]# service neutron-bsn-lldp status Redirecting to /bin/systemctl status neutron-bsn-lldp.service neutron-bsn-lldp.service - bsn lldp Loaded: loaded (/usr/lib/systemd/system/neutron-bsn-lldp.service; enabled) Active: active (running) since Thu 2015-09-03 08:14:38 EDT; 1min 9s ago Main PID: 1793 (bsnlldp) CGroup: /system.slice/neutron-bsn-lldp.service └─1793 /usr/bin/python2 /usr/bin/bsnlldp Sep 03 08:14:38 host-192-0-2-20 systemd[1]: Starting bsn lldp... Sep 03 08:14:38 host-192-0-2-20 systemd[1]: Started bsn lldp. Sep 03 08:15:28 overcloud-controller-0.localdomain sudo[11806]: root : TTY=unknown ; PWD=/ ;...w Sep 03 08:15:28 overcloud-controller-0.localdomain sudo[11822]: root : TTY=unknown ; PWD=/ ;...w Sep 03 08:15:28 overcloud-controller-0.localdomain sudo[11845]: root : TTY=unknown ; PWD=/ ;...w Sep 03 08:15:38 overcloud-controller-0.localdomain sudo[12280]: root : TTY=unknown ; PWD=/ ;...w Sep 03 08:15:38 overcloud-controller-0.localdomain sudo[12284]: root : TTY=unknown ; PWD=/ ;...w Hint: Some lines were ellipsized, use -l to show in full. [root@overcloud-controller-0 heat-admin]# yum info *bigswitch* Installed Packages Name : openstack-neutron-bigswitch-lldp Arch : noarch Version : 2015.1.37 Release : 1.el7.centos Size : 10 k Repo : installed Summary : Neutron Big Switch Networks LLDP service URL : https://pypi.python.org/pypi/bsnstacklib License : ASL 2.0 Description : This package contains the Big Switch Networks neutron LLDP agent. Name : python-networking-bigswitch Arch : noarch Version : 2015.1.37 Release : 1.el7.centos Size : 559 k Repo : installed Summary : Big Switch Networks neutron plugin for OpenStack Networking URL : https://pypi.python.org/pypi/bsnstacklib License : ASL 2.0 Description : This package contains Big Switch Networks : neutron plugins and agents neutron-server starts with the fix at https://github.com/jistr/tripleo-bigswitch-temporary-setup/pull/1. We can close this bug now. |
Description of problem: With current packaging scheme, Big Switch generates two rpm packages: python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm contains ml2 plugin and openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm contains the LLDP service. If we use the original OSP image to install overcloud, download these rpms to controller nodes, use rpm -ivh to install these two packages and then restart neutron-server, everything works fine. Which means the rpms are good. However, problem happens when we use virt-customize. Following is the steps to constantly reproduce the problem: 1. on director node use following command to customize image $ virt-customize --upload python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm:/root/python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm --upload openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm:/root/openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm --run-command "rpm -ivh /root/python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm;rpm -ivh /root/openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm;systemctl enable neutron-bsn-lldp.service" -a overcloud-full.qcow2 2. upload image and install overcloud as usual 3. Since at this point, ml2_conf.ini is not populated with bigswitch specific configuration due to the first bug. We use "puppet apply ml2.pp" to make changes to ml2_conf.ini. 4. on controller node, $ sudo systemctl restart neutron-server 5. /var/log/neutron/server.log will show following error. The line number may twisted a little bit since we added some debug messages. The point is that the bsn_ml2 mechanism_driver is not picked up. 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config Traceback (most recent call last): 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/neutron/common/config.py", line 227, in load_paste_app 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config app = deploy.loadapp("config:%s" % config_path, name=app_name) 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config return loadobj(APP, uri, name=name, **kw) 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config return context.create() 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config return self.object_type.invoke(self) 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config **context.local_conf) 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config val = callable(*args, **kw) 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 25, in urlmap_factory 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config app = loader.get_app(app_name, global_conf=global_conf) 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config name=name, global_conf=global_conf).create() 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config return self.object_type.invoke(self) 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config **context.local_conf) 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config val = callable(*args, **kw) 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/neutron/auth.py", line 71, in pipeline_factory 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config app = loader.get_app(pipeline[-1]) 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config name=name, global_conf=global_conf).create() 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config return self.object_type.invoke(self) 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config return fix_call(context.object, context.global_conf, **context.local_conf) 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config val = callable(*args, **kw) 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 71, in factory 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config return cls(**local_config) 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 75, in __init__ 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config plugin = manager.NeutronManager.get_plugin() 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 222, in get_plugin 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config return weakref.proxy(cls.get_instance().plugin) 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 216, in get_instance 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config cls._create_instance() 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config return f(*args, **kwargs) 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 202, in _create_instance 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config cls._instance = cls() 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 117, in __init__ 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config plugin_provider) 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 143, in _get_plugin_instance 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config return plugin_class() 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 137, in __init__ 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config self.mechanism_manager = managers.MechanismManager() 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 277, in __init__ 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config name_order=True) 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 59, in __init__ 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config self._init_plugins(extensions) 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 120, in _init_plugins 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config LOG.error("iiiiiiii %s" % self[n]) 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 284, in __getitem__ 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config return self._extensions_by_name[name] 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config KeyError: 'bsn_ml2' 6. We did further debug. Looks like the problem is at /usr/lib/python2.7/site-packages/pkg_resources.py. What happens is that the entry_point file (/usr/lib/python2.7/site-packages/bsnstacklib-2015.1.37-py2.7.egg-info/entry_points.txt) comes with python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm is not parsed anywhere. We suspect that we may need to something more with virt-customize. Required files ml2.pp https://bigswitch.box.com/shared/static/lvuorf874zk66u0qflolorm7l3fmg4ls.pp python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm https://bigswitch.box.com/shared/static/g8gceah2iqrqyw1fw5vzzio6rfbxlqyt.rpm openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm https://bigswitch.box.com/shared/static/il5adj35azavpol8pk5giotrebgrkwi3.rpm Version-Release number of selected component (if applicable): OSP7 GA with @Jiri's patch https://gist.github.com/jistr/2575b78058fed8be36d9 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: