Bug 1467398 - During RHOSP 9 to RHOSP 10 upgrade, networking-bigswitch is upgraded to Newton release
During RHOSP 9 to RHOSP 10 upgrade, networking-bigswitch is upgraded to Newto...
Status: NEW
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron (Show other bugs)
10.0 (Newton)
Unspecified Unspecified
unspecified Severity urgent
: ---
: ---
Assigned To: Assaf Muller
Toni Freger
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-03 12:17 EDT by bigswitch
Modified: 2017-07-05 16:17 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description bigswitch 2017-07-03 12:17:20 EDT
Description of problem:
Hi
I am trying to upgrade rhosp 9 to rhosp 10. At chapter 3.4.6 (upgrading controller nodes), networking-bigswitch plugin was upgraded from mitaka release to newton release. This causes neutron server to fail to start with the below error

[stack@vm-240 ~]$ heat deployment-show 250c8ea5-5502-4938-a852-24a2cbd99c45
WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead
{
  "status": "FAILED",
  "server_id": "262df95c-b4b3-483d-beb7-4eb205c7bb31",
  "config_id": "1b23eb12-608f-4d96-8d4c-7b37504fd23a",
  "output_values": {
    "deploy_stdout": "Sun Jul  2 21:34:52 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl start httpd\nSun Jul  2 21:34:53 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl enable httpd\nSun Jul  2 21:34:53 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to check_resource_systemd for httpd to be started\nhttpd is started\nSun Jul  2 21:34:54 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl start memcached\nSun Jul  2 21:34:54 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl enable memcached\nSun Jul  2 21:34:54 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to check_resource_systemd for memcached to be started\nmemcached is started\nSun Jul  2 21:34:55 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl start mongod\nSun Jul  2 21:34:55 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl enable mongod\nSun Jul  2 21:34:55 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to check_resource_systemd for mongod to be started\nmongod is started\nSun Jul  2 21:34:55 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl start neutron-dhcp-agent\nSun Jul  2 21:34:56 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl enable neutron-dhcp-agent\nSun Jul  2 21:34:56 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to check_resource_systemd for neutron-dhcp-agent to be started\nneutron-dhcp-agent is started\nSun Jul  2 21:34:57 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl start neutron-l3-agent\nSun Jul  2 21:34:57 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl enable neutron-l3-agent\nSun Jul  2 21:34:57 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to check_resource_systemd for neutron-l3-agent to be started\nneutron-l3-agent is started\nSun Jul  2 21:34:58 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl start neutron-metadata-agent\nSun Jul  2 21:34:58 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl enable neutron-metadata-agent\nSun Jul  2 21:34:58 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to check_resource_systemd for neutron-metadata-agent to be started\nneutron-metadata-agent is started\nSun Jul  2 21:34:59 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl start neutron-netns-cleanup\nSun Jul  2 21:35:00 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl enable neutron-netns-cleanup\nSun Jul  2 21:35:00 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to check_resource_systemd for neutron-netns-cleanup to be started\nneutron-netns-cleanup is started\nSun Jul  2 21:35:00 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl start neutron-openvswitch-agent\nSun Jul  2 21:35:01 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl enable neutron-openvswitch-agent\nSun Jul  2 21:35:01 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to check_resource_systemd for neutron-openvswitch-agent to be started\nneutron-openvswitch-agent is started\nSun Jul  2 21:35:02 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl start neutron-ovs-cleanup\nSun Jul  2 21:35:02 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl enable neutron-ovs-cleanup\nSun Jul  2 21:35:03 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to check_resource_systemd for neutron-ovs-cleanup to be started\nneutron-ovs-cleanup is started\nSun Jul  2 21:35:03 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl start neutron-server\n",
    "deploy_stderr": "Job for neutron-server.service failed because the control process exited with error code. See \"systemctl status neutron-server.service\" and \"journalctl -xe\" for details.\n",
    "deploy_status_code": 1
  },
  "creation_time": "2017-07-03T04:34:17Z",
  "updated_time": "2017-07-03T04:35:02Z",
  "input_values": {
    "update_identifier": "1498684273",
    "deploy_identifier": "1499055564"
  },
  "action": "CREATE",
  "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1",
  "id": "250c8ea5-5502-4938-a852-24a2cbd99c45"
}


Before the deploy
[root@rhosp9-controller-0 heat-admin]# pip show networking-bigswitch
---
Metadata-Version: 1.1
Name: networking-bigswitch
Version: 8.40.8
Summary: Big Switch Networks Plugins for OpenStack Networking
Home-page: http://www.bigswitch.com/
Author: Big Switch Networks
Author-email: support@bigswitch.com
License: UNKNOWN
Location: /usr/lib/python2.7/site-packages
Requires:
Classifiers:
  Environment :: OpenStack
  Intended Audience :: Information Technology
  Intended Audience :: System Administrators
  License :: OSI Approved :: Apache Software License
  Operating System :: POSIX :: Linux
  Programming Language :: Python
  Programming Language :: Python :: 2
  Programming Language :: Python :: 2.7
Entry-points:
  [neutron.service_plugins]
  bsn_service_plugin = networking_bigswitch.plugins.bigswitch.bsn_service_plugin:BSNServicePlugin
  bsn_l3 = networking_bigswitch.plugins.bigswitch.l3_router_plugin:L3RestProxy
  [neutron.db.alembic_migrations]
  bsn_extensions = networking_bigswitch.plugins.bigswitch.db.migration:alembic_migrations
  [neutronclient.extension]
  bsn_extensions = networking_bigswitch.plugins.bigswitch.neutronclient.v2_0._bsn_plugin_client
  [neutron.ml2.mechanism_drivers]
  bsn_ml2 = networking_bigswitch.plugins.ml2.drivers.mech_bigswitch.driver:BigSwitchMechanismDriver
  [console_scripts]
  neutron-bsn-agent = networking_bigswitch.plugins.bigswitch.agent.restproxy_agent:main
  bsnlldp = networking_bigswitch.bsnlldp.bsnlldp:main
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
[root@rhosp9-controller-0 heat-admin]#

After the deploy
[root@rhosp9-controller-0 heat-admin]# pip show networking-bigswitch
---
Metadata-Version: 1.1
Name: networking-bigswitch
Version: 9.42.1
Summary: Big Switch Networks Plugins for OpenStack Networking
Home-page: http://www.bigswitch.com/
Author: Big Switch Networks
Author-email: support@bigswitch.com
License: UNKNOWN
Location: /usr/lib/python2.7/site-packages
Requires:
Classifiers:
  Environment :: OpenStack
  Intended Audience :: Information Technology
  Intended Audience :: System Administrators
  License :: OSI Approved :: Apache Software License
  Operating System :: POSIX :: Linux
  Programming Language :: Python
  Programming Language :: Python :: 2
  Programming Language :: Python :: 2.7
Entry-points:
  [console_scripts]
  bsnlldp = networking_bigswitch.bsnlldp.bsnlldp:main
  neutron-bsn-agent = networking_bigswitch.plugins.bigswitch.agent.restproxy_agent:main
  [neutron.db.alembic_migrations]
  bsn_extensions = networking_bigswitch.plugins.bigswitch.db.migration:alembic_migrations
  [neutron.ml2.mechanism_drivers]
  bsn_ml2 = networking_bigswitch.plugins.ml2.drivers.mech_bigswitch.driver:BigSwitchMechanismDriver
  [neutron.service_plugins]
  bsn_l3 = networking_bigswitch.plugins.bigswitch.l3_router_plugin:L3RestProxy
  bsn_service_plugin = networking_bigswitch.plugins.bigswitch.bsn_service_plugin:BSNServicePlugin
  [neutronclient.extension]
  bsn_extensions = networking_bigswitch.plugins.bigswitch.neutronclient.v2_0._bsn_plugin_client
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
[root@rhosp9-controller-0 heat-admin]#


Version-Release number of selected component (if applicable):
RHOSP 10

How reproducible:
Always

This is the deploy command
time openstack overcloud deploy --stack rhosp9 -e /home/stack/templates/bigswitch-config.yaml -e /home/stack/templates/network-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/templates/timezone.yaml  -e /home/stack/templates/rhel-registration/environment-rhel-registration.yaml -e /home/stack/templates/rhel-registration/rhel-registration-resource-registry.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker.yaml --neutron-network-type vlan --neutron-network-vlan-ranges rhosp:1001:2000 --neutron-bridge-mappings "rhosp:br-ex" --neutron-disable-tunneling --compute-scale 2 --control-scale 1 --templates --control-flavor control --compute-flavor compute --ntp-server 0.rhel.pool.ntp.org --timeout $timeout --debug 2>&1 | tee haha-$date.log
Comment 1 bigswitch 2017-07-03 12:22:17 EDT
sosreport from overcloud controller is uploaded to box

https://bigswitch.box.com/s/7hx4b4xndwpxe4wizctarnfnd3k8ta4x
Comment 2 Mike Burns 2017-07-05 08:58:13 EDT
Upgrade from the mitaka(osp9) to newton(osp10) version sounds like it's doing the right thing.  During the upgrade, all packages get updated to their OSP 10 versions.  This sounds a lot like an issue between neutron and the bigswitch plugin doesn't it?  Shouldn't this be handled by BigSwitch?
Comment 3 bigswitch 2017-07-05 16:17:01 EDT
Hi Mike,

We debugged this further and found the issue but not in a way that we can proceed with upgrade.
The Big Switch Networks plugin needs to read properties from under 'keystone_authtoken' in neutron.conf. However, there has been a change in the naming of properties there from Mitaka to Newton. Namely,

RHOSP 9 mitaka    > RHOSP 10 newton
admin_user        > username
admin_password    > password
auth_uri          > auth_url
admin_tenant_name > project_name
NA                > project_domain_name
NA                > user_domain_name

Now the upgrade process upgrades neutron and networking-bigswitch, but the properties file is still the same old one from Mitaka. So plugin fails to find the properties and neutron-server startup fails.
We patched the local env to do it right and neutron-server starts running. But it is not possible to resume upgrade from that state.

Can we add some extra_properties in one of the RHOSP related yaml files to specify those properties and make sure they're there when upgrade happens?

Either that, or we wait until we patch the Newton version of our plugin and the git tag update reaches the RDO repo which hosts and provides the packages during upgrade.

Note You need to log in before you can comment on or make changes to this bug.