Bug 1258106 - ml2 plugin rpm package is not properly installed by virt-customize
ml2 plugin rpm package is not properly installed by virt-customize
Status: CLOSED WORKSFORME
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: 8.0 (Liberty)
Assigned To: Jiri Stransky
yeylon@redhat.com
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-29 02:39 EDT by bigswitch
Modified: 2016-04-18 03:12 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-09-04 14:20:12 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description bigswitch 2015-08-29 02:39:37 EDT
Description of problem:

With current packaging scheme, Big Switch generates two rpm packages: python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm contains ml2 plugin and openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm contains the LLDP service. If we use the original OSP image to install overcloud, download these rpms to controller nodes, use rpm -ivh to install these two packages and then restart neutron-server, everything works fine. Which means the rpms are good.
     However, problem happens when we use virt-customize. Following is the steps to constantly reproduce the problem:

1. on director node use following command to customize image
$ virt-customize --upload python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm:/root/python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm --upload openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm:/root/openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm --run-command "rpm -ivh /root/python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm;rpm -ivh /root/openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm;systemctl enable neutron-bsn-lldp.service" -a overcloud-full.qcow2

2. upload image and install overcloud as usual

3. Since at this point, ml2_conf.ini is not populated with bigswitch specific configuration due to the first bug. We use "puppet apply ml2.pp" to make changes to ml2_conf.ini.

4. on controller node, 
$ sudo systemctl restart neutron-server

5. /var/log/neutron/server.log will show following error. The line number may twisted a little bit since we added some debug messages. The point is that the bsn_ml2 mechanism_driver is not picked up. 
2015-08-29 00:05:33.708 28397 TRACE neutron.common.config Traceback (most recent call last):

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/common/config.py", line 227, in load_paste_app

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     app = deploy.loadapp("config:%s" % config_path, name=app_name)

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     return loadobj(APP, uri, name=name, **kw)

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     return context.create()

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     return self.object_type.invoke(self)

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     **context.local_conf)

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     val = callable(*args, **kw)

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 25, in urlmap_factory

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     app = loader.get_app(app_name, global_conf=global_conf)

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     name=name, global_conf=global_conf).create()

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     return self.object_type.invoke(self)

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     **context.local_conf)

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     val = callable(*args, **kw)

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/auth.py", line 71, in pipeline_factory

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     app = loader.get_app(pipeline[-1])

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     name=name, global_conf=global_conf).create()

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     return self.object_type.invoke(self)

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     return fix_call(context.object, context.global_conf, **context.local_conf)

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     val = callable(*args, **kw)

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 71, in factory

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     return cls(**local_config)

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 75, in __init__

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     plugin = manager.NeutronManager.get_plugin()

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 222, in get_plugin

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     return weakref.proxy(cls.get_instance().plugin)

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 216, in get_instance

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     cls._create_instance()

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     return f(*args, **kwargs)

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 202, in _create_instance

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     cls._instance = cls()

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 117, in __init__

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     plugin_provider)

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 143, in _get_plugin_instance

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     return plugin_class()

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 137, in __init__

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     self.mechanism_manager = managers.MechanismManager()

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 277, in __init__

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     name_order=True)

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 59, in __init__

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     self._init_plugins(extensions)

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 120, in _init_plugins

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     LOG.error("iiiiiiii %s" % self[n])

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config   File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 284, in __getitem__

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config     return self._extensions_by_name[name]

2015-08-29 00:05:33.708 28397 TRACE neutron.common.config KeyError: 'bsn_ml2'

6. We did further debug. Looks like the problem is at /usr/lib/python2.7/site-packages/pkg_resources.py. What happens is that the entry_point file (/usr/lib/python2.7/site-packages/bsnstacklib-2015.1.37-py2.7.egg-info/entry_points.txt) comes with python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm is not parsed anywhere. We suspect that we may need to something more with virt-customize.


Required files
ml2.pp
https://bigswitch.box.com/shared/static/lvuorf874zk66u0qflolorm7l3fmg4ls.pp

python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm
https://bigswitch.box.com/shared/static/g8gceah2iqrqyw1fw5vzzio6rfbxlqyt.rpm

openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm
https://bigswitch.box.com/shared/static/il5adj35azavpol8pk5giotrebgrkwi3.rpm


Version-Release number of selected component (if applicable):
OSP7 GA with @Jiri's patch https://gist.github.com/jistr/2575b78058fed8be36d9

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 3 Marios Andreou 2015-08-31 03:00:01 EDT
(In reply to bigswitch from comment #0)

>      However, problem happens when we use virt-customize. Following is the
> steps to constantly reproduce the problem:
> 
> 1. on director node use following command to customize image
> $ virt-customize --upload
> python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm:/root/python-
> networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm --upload
> openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm:/root/
> openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm
> --run-command "rpm -ivh
> /root/python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm;rpm -ivh
> /root/openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm;
> systemctl enable neutron-bsn-lldp.service" -a overcloud-full.qcow2
> 
> 2. upload image and install overcloud as usual


FYI, for this bug we are concentrating on getting the package installed correctly, you can debug much faster by just launching the (virt-customized) overcloud-full.qcow2 image rather than doing a full deploy.

nova boot --flavor baremetal --image overcloud-full   --key-name default test

(I believe ssh cloud-user@IP is what you need)


> 
> 3. Since at this point, ml2_conf.ini is not populated with bigswitch
> specific configuration due to the first bug. We use "puppet apply ml2.pp" to
> make changes to ml2_conf.ini.
> 
> 4. on controller node, 
> $ sudo systemctl restart neutron-server

we use pacemaker to manage the openstack-* services so you should probably use that to restart like:

pcs resource restart neutron-server

> 
> 5. /var/log/neutron/server.log will show following error. The line number
> may twisted a little bit since we added some debug messages. The point is
> that the bsn_ml2 mechanism_driver is not picked up. 
> 2015-08-29 00:05:33.708 28397 TRACE neutron.common.config Traceback (most
> recent call last):

will do my best to try this before we talk this afternoon. Where can I get the rpms exact that you are using above from please?

thanks, marios
Comment 4 Marios Andreou 2015-08-31 06:51:24 EDT
(In reply to marios from comment #3)
> (In reply to bigswitch from comment #0)
> will do my best to try this before we talk this afternoon. Where can I get
> the rpms exact that you are using above from please?

fyi for context, the bug @ https://bugzilla.redhat.com/show_bug.cgi?id=1254828 tracks the packaging effort and links to rpm there
Comment 5 Marios Andreou 2015-08-31 07:02:06 EDT
(In reply to marios from comment #4)
> (In reply to marios from comment #3)
> > (In reply to bigswitch from comment #0)
> > will do my best to try this before we talk this afternoon. Where can I get
> > the rpms exact that you are using above from please?
> 
> fyi for context, the bug @
> https://bugzilla.redhat.com/show_bug.cgi?id=1254828 tracks the packaging
> effort and links to rpm there


Though I can't quickly see a link to the openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm in that packaging tracking bug
Comment 6 Marios Andreou 2015-08-31 08:06:24 EDT
sorry for the noise is in the description (am poking at this)
Comment 7 bigswitch 2015-09-02 11:06:46 EDT
1. the command we used to deploy overcloud
$ openstack overcloud deploy -e /home/stack/network-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/templates/bigswitch-config.yaml --neutron-network-type vlan --neutron-network-vlan-ranges datacentre:50:90 --neutron-disable-tunneling --compute-scale 1 --control-scale 3 --ceph-storage-scale 0 --templates --control-flavor control --compute-flavor compute --ntp-server 0.rhel.pool.ntp.org --debug 2>&1 | tee haha.log
Comment 8 bigswitch 2015-09-02 11:08:08 EDT
2. the command we used to patch image
$ virt-customize --upload python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm:/root/python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm --upload openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm:/root/openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm --run-command "rpm -ivh /root/python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm;rpm -ivh /root/openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm;systemctl enable neutron-bsn-lldp.service" -a overcloud-full.qcow2
Comment 9 bigswitch 2015-09-02 12:46:21 EDT
To summarize what we found about this bug
1. If patch image with Big Switch rpm and then install overcloud, neutron-server can NOT find bsn_ml2 entry point.
2. If patch image with Big Switch rpm, install overcloud and then put "bsn_ml2 = bsnstacklib.plugins.ml2.drivers.mech_bigswitch.driver:BigSwitchMechanismDriver" into "/lib/python2.7/site-packages/neutron-2015.1.0-py2.7.egg-info/entry_points.txt" [neutron.ml2.mechanism_drivers] section, neutron-server can NOT find bsn_ml2 entry point.
3. If install overcloud first and then rpm -ivh Big Switch package, neutron-server CAN find bsn_ml2 entry point.


The exact command we used to reproduce this problem is as following
1. deploy overcloud
$ openstack overcloud deploy -e /home/stack/network-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/templates/bigswitch-config.yaml --neutron-network-type vlan --neutron-network-vlan-ranges datacentre:50:90 --neutron-disable-tunneling --compute-scale 1 --control-scale 3 --ceph-storage-scale 0 --templates --control-flavor control --compute-flavor compute --ntp-server 0.rhel.pool.ntp.org --debug 2>&1 | tee haha.log

2. patch image
virt-customize --upload python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm:/root/python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm --upload openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm:/root/openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm --run-command "rpm -ivh /root/python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm;rpm -ivh /root/openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm;systemctl enable neutron-bsn-lldp.service" -a overcloud-full.qcow2
Comment 10 Marios Andreou 2015-09-03 07:19:27 EDT
FYI I am having some luck with

 virt-customize -a overcloud-full.qcow2 --firstboot install_bigswitch_packages.sh 

where the firstboot script is simply like

rpm -ivh --force /root/python-networking-bigswitch-2015.1.37-1.el7.centos.noarch.rpm
rpm -ivh --force /root/openstack-neutron-bigswitch-lldp-2015.1.37-1.el7.centos.noarch.rpm;
systemctl enable neutron-bsn-lldp.service

(the --force here is because it was already installed on that image from the earlier invocation of virt-customize as in the script above)

I will test again from scratch for sanity check and update
Comment 11 Marios Andreou 2015-09-03 08:50:44 EDT
The latest version of the virt-customize command at https://github.com/jistr/tripleo-bigswitch-temporary-setup/pull/1 fixes this I believe. It  uses --firstboot-command to install the packages and enable the lldp service.

On deploy, everything looks installed ok like [2] below. I then did the neutron server restart:

vi /etc/neutron/plugins/ml2/ml2_conf.ini
# (^^^ mechanism_drivers =openvswitch,bsn_ml2)
pcs resource restart neutron-server

neutron-server still won't start, but at least not because of the bsn_ml2 key error. I see [1] for example and a long trace like [0] (if that trace is useful you may want to copy it as the paste will expire)

[0] 
http://paste.openstack.org/raw/444151

[1] 
2015-09-03 08:31:33.264 15477 INFO neutron.plugins.ml2.managers [-] Initializing mechanism driver 'bsn_ml2'
2015-09-03 08:31:33.264 15477 DEBUG bsnstacklib.plugins.ml2.drivers.mech_bigswitch.driver [-] Initializing driver initialize /usr/lib/python2.7/site-packages/bsnstacklib/plugins/ml2/drivers/mech_bigswitch/driver.py:62

[2]
[root@overcloud-controller-0 heat-admin]# service neutron-bsn-lldp status
Redirecting to /bin/systemctl status  neutron-bsn-lldp.service
neutron-bsn-lldp.service - bsn lldp
   Loaded: loaded (/usr/lib/systemd/system/neutron-bsn-lldp.service; enabled)
   Active: active (running) since Thu 2015-09-03 08:14:38 EDT; 1min 9s ago
 Main PID: 1793 (bsnlldp)
   CGroup: /system.slice/neutron-bsn-lldp.service
           └─1793 /usr/bin/python2 /usr/bin/bsnlldp

Sep 03 08:14:38 host-192-0-2-20 systemd[1]: Starting bsn lldp...
Sep 03 08:14:38 host-192-0-2-20 systemd[1]: Started bsn lldp.
Sep 03 08:15:28 overcloud-controller-0.localdomain sudo[11806]: root : TTY=unknown ; PWD=/ ;...w
Sep 03 08:15:28 overcloud-controller-0.localdomain sudo[11822]: root : TTY=unknown ; PWD=/ ;...w
Sep 03 08:15:28 overcloud-controller-0.localdomain sudo[11845]: root : TTY=unknown ; PWD=/ ;...w
Sep 03 08:15:38 overcloud-controller-0.localdomain sudo[12280]: root : TTY=unknown ; PWD=/ ;...w
Sep 03 08:15:38 overcloud-controller-0.localdomain sudo[12284]: root : TTY=unknown ; PWD=/ ;...w
Hint: Some lines were ellipsized, use -l to show in full.

[root@overcloud-controller-0 heat-admin]# yum info *bigswitch*
Installed Packages
Name        : openstack-neutron-bigswitch-lldp
Arch        : noarch
Version     : 2015.1.37
Release     : 1.el7.centos
Size        : 10 k
Repo        : installed
Summary     : Neutron Big Switch Networks LLDP service
URL         : https://pypi.python.org/pypi/bsnstacklib
License     : ASL 2.0
Description : This package contains the Big Switch Networks neutron LLDP agent.

Name        : python-networking-bigswitch
Arch        : noarch
Version     : 2015.1.37
Release     : 1.el7.centos
Size        : 559 k
Repo        : installed
Summary     : Big Switch Networks neutron plugin for OpenStack Networking
URL         : https://pypi.python.org/pypi/bsnstacklib
License     : ASL 2.0
Description : This package contains Big Switch Networks
            : neutron plugins and agents
Comment 12 bigswitch 2015-09-04 00:57:32 EDT
neutron-server starts with the fix at https://github.com/jistr/tripleo-bigswitch-temporary-setup/pull/1. We can close this bug now.

Note You need to log in before you can comment on or make changes to this bug.