Description of problem:

OSP9 -> OSP10 upgrade fails during major-upgrade-pacemaker.yaml with sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'ovs_neutron.bsn_routerrules' doesn't exist")


Version-Release number of selected component (if applicable):
python-networking-bigswitch-9.42.1-2.el7ost.noarch
openstack-neutron-bigswitch-agent-9.42.1-2.el7ost.noarch
openstack-neutron-bigswitch-lldp-9.42.1-2.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP9
2. Upgrade to OSP10 latest

Actual results:
Upgrade fails with:

[stack@undercloud-0 ~]$ openstack stack failures list overcloud
overcloud.UpdateWorkflow.ControllerPacemakerUpgradeDeployment_Step3.0:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: 3daaf7d7-40c0-40f8-b4a3-862fb1193fbf
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  deploy_stdout: |
    ...
    mongod is started
    HTTP/1.1 200 OK
    Content-Type: text/plain
    Connection: close
    Content-Length: 32
    
    Galera cluster node is synced.
    Running upgrade for neutron ...
    OK
    Running upgrade for bsn_extensions ...
    (truncated, view all with --long)
  deploy_stderr: |
    ...
        result.read()
      File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1312, in read
        first_packet = self.connection._read_packet()
      File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet
        packet.check_error()
      File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
        err.raise_mysql_exception(self._data)
      File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
        raise errorclass(errno, errval)
    sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'ovs_neutron.bsn_routerrules' doesn't exist") [SQL: u'ALTER TABLE bsn_routerrules ADD COLUMN tenant_id VARCHAR(255)']
    (truncated, view all with --long)


Expected results:
Upgrade doesn't fail.

Additional info:
Full trace:

Running upgrade for neutron ...
OK
Running upgrade for bsn_extensions ...
[2017-05-24 13:13:53,722] (heat-config) [DEBUG] No handlers could be found for logger "ceilometer.cmd.storage"
2017-05-24 13:13:46.346 8073 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2017-05-24 13:13:46.362 8073 INFO ceilometer.storage.mongo.utils [-] Connecting to mongodb on [('10.0.0.15', 27017), ('10.0.0.14', 27017), ('10.0.0.16', 27017)]
2017-05-24 13:13:46.378 8073 INFO ceilometer.storage.mongo.utils [-] Connecting to mongodb on [('10.0.0.15', 27017), ('10.0.0.14', 27017), ('10.0.0.16', 27017)]
2017-05-24 13:13:46.386 8073 INFO ceilometer.cmd.storage [-] Skipping Gnocchi resource types upgrade
Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".
2017-05-24 13:13:48.082 8151 INFO migrate.versioning.api [-] 72 -> 73...
2017-05-24 13:13:48.119 8151 INFO migrate.versioning.api [-] done
2017-05-24 13:13:48.120 8151 INFO migrate.versioning.api [-] 73 -> 74...
2017-05-24 13:13:48.131 8151 INFO migrate.versioning.api [-] done
2017-05-24 13:13:48.131 8151 INFO migrate.versioning.api [-] 74 -> 75...
2017-05-24 13:13:48.175 8151 INFO migrate.versioning.api [-] done
2017-05-24 13:13:48.175 8151 INFO migrate.versioning.api [-] 75 -> 76...
2017-05-24 13:13:48.198 8151 INFO migrate.versioning.api [-] done
2017-05-24 13:13:48.199 8151 INFO migrate.versioning.api [-] 76 -> 77...
2017-05-24 13:13:48.221 8151 INFO migrate.versioning.api [-] done
2017-05-24 13:13:48.222 8151 INFO migrate.versioning.api [-] 77 -> 78...
2017-05-24 13:13:48.279 8151 INFO migrate.versioning.api [-] done
2017-05-24 13:13:48.279 8151 INFO migrate.versioning.api [-] 78 -> 79...
2017-05-24 13:13:48.337 8151 INFO migrate.versioning.api [-] done
Option "verbose" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:1171: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade
expire_on_commit=expire_on_commit, _conf=conf)
2017-05-24 13:13:49.979 8251 INFO migrate.versioning.api [-] 71 -> 72...
2017-05-24 13:13:50.013 8251 INFO migrate.versioning.api [-] done
2017-05-24 13:13:50.014 8251 INFO migrate.versioning.api [-] 72 -> 73...
2017-05-24 13:13:50.053 8251 INFO migrate.versioning.api [-] done
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 0e66c5227a8a -> 45f8dd33480b, qos dscp db addition
INFO  [alembic.runtime.migration] Running upgrade 45f8dd33480b -> 5abc0278ca73, Add support for VLAN trunking
INFO  [alembic.runtime.migration] Running upgrade 5abc0278ca73 -> d3435b514502, Add device_id index to Port
INFO  [alembic.runtime.migration] Running upgrade d3435b514502 -> 30107ab6a3ee, provisioning_blocks.py
INFO  [alembic.runtime.migration] Running upgrade 30107ab6a3ee -> c415aab1c048, add revisions table
INFO  [alembic.runtime.migration] Running upgrade c415aab1c048 -> a963b38d82f4, add dns name to portdnses
INFO  [alembic.runtime.migration] Running upgrade a963b38d82f4 -> 3d0e74aa7d37, Add flavor_id to Router
INFO  [alembic.runtime.migration] Running upgrade 3d0e74aa7d37 -> 030a959ceafa, uniq_routerports0port_id
INFO  [alembic.runtime.migration] Running upgrade 030a959ceafa -> a5648cfeeadf, Add support for Subnet Service Types
INFO  [alembic.runtime.migration] Running upgrade a5648cfeeadf -> 0f5bef0f87d4, add_qos_minimum_bandwidth_rules
INFO  [alembic.runtime.migration] Running upgrade 0f5bef0f87d4 -> 67daae611b6e, add standardattr to qos policies
INFO  [alembic.runtime.migration] Running upgrade 4ffceebfcdc -> 7bbb25278f53, device_owner_ha_replicate_int
INFO  [alembic.runtime.migration] Running upgrade 7bbb25278f53 -> 89ab9a816d70, Rename ml2_network_segments table
INFO  [alembic.runtime.migration] Running upgrade 89ab9a816d70 -> c879c5e1ee90, Add segment_id to subnet
INFO  [alembic.runtime.migration] Running upgrade c879c5e1ee90 -> 8fd3918ef6f4, Add segment_host_mapping table.
INFO  [alembic.runtime.migration] Running upgrade 8fd3918ef6f4 -> 4bcd4df1f426, Rename ml2_dvr_port_bindings
INFO  [alembic.runtime.migration] Running upgrade 4bcd4df1f426 -> b67e765a3524, Remove mtu column from networks.
INFO  [alembic.runtime.migration] Running upgrade b67e765a3524 -> a84ccf28f06a, migrate dns name from port
INFO  [alembic.runtime.migration] Running upgrade a84ccf28f06a -> 7d9d8eeec6ad, rename tenant to project
INFO  [alembic.runtime.migration] Running upgrade 7d9d8eeec6ad -> a8b517cff8ab, Add routerport bindings for L3 HA
INFO  [alembic.runtime.migration] Running upgrade a8b517cff8ab -> 3b935b28e7a0, migrate to pluggable ipam
INFO  [alembic.runtime.migration] Running upgrade 3b935b28e7a0 -> b12a3ef66e62, add standardattr to qos policies
INFO  [alembic.runtime.migration] Running upgrade b12a3ef66e62 -> 97c25b0d2353, Add Name and Description to the networksegments table
INFO  [alembic.runtime.migration] Running upgrade 97c25b0d2353 -> 2e0d7a8a1586, Add binding index to RouterL3AgentBinding
INFO  [alembic.runtime.migration] Running upgrade 2e0d7a8a1586 -> 5c85685d616d, Remove availability ranges.
INFO  [alembic.runtime.migration] Running upgrade 67daae611b6e -> 6b461a21bcfc, uniq_floatingips0floating_network_id0fixed_port_id0fixed_ip_addr
INFO  [alembic.runtime.migration] Running upgrade 6b461a21bcfc -> 5cd92597d11d, Add ip_allocation to port
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade kilo -> 1086543dfc0f, update column type for testpath result
INFO  [alembic.runtime.migration] Running upgrade 1086543dfc0f -> 1ef57200f387, testpath modify uniq constraint
INFO  [alembic.runtime.migration] Running upgrade 1ef57200f387 -> 34f888285695, support multiple logical routers
Traceback (most recent call last):
File "/usr/bin/neutron-db-manage", line 10, in <module>
sys.exit(main())
File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 686, in main
return_val |= bool(CONF.command.func(config, CONF.command.name))
File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 207, in do_upgrade
desc=branch, sql=CONF.command.sql)
File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 108, in do_alembic_command
getattr(alembic_command, cmd)(config, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/alembic/command.py", line 174, in upgrade
script.run_env()
File "/usr/lib/python2.7/site-packages/alembic/script/base.py", line 407, in run_env
util.load_python_file(self.dir, 'env.py')
File "/usr/lib/python2.7/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
module = load_module_py(module_id, path)
File "/usr/lib/python2.7/site-packages/alembic/util/compat.py", line 79, in load_module_py
mod = imp.load_source(module_id, path, fp)
File "/usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/env.py", line 86, in <module>
run_migrations_online()
File "/usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/env.py", line 77, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/usr/lib/python2.7/site-packages/alembic/runtime/environment.py", line 797, in run_migrations
self.get_context().run_migrations(**kw)
File "/usr/lib/python2.7/site-packages/alembic/runtime/migration.py", line 312, in run_migrations
step.migration_fn(**kw)
File "/usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/34f888285695_support_multiple_logical_routers.py", line 38, in upgrade
sa.Column('tenant_id', sa.String(255), nullable=True))
File "<string>", line 8, in add_column
File "<string>", line 3, in add_column
File "/usr/lib/python2.7/site-packages/alembic/operations/ops.py", line 1535, in add_column
return operations.invoke(op)
File "/usr/lib/python2.7/site-packages/alembic/operations/base.py", line 318, in invoke
return fn(self, operation)
File "/usr/lib/python2.7/site-packages/alembic/operations/toimpl.py", line 123, in add_column
schema=schema
File "/usr/lib/python2.7/site-packages/alembic/ddl/impl.py", line 172, in add_column
self._exec(base.AddColumn(table_name, column, schema=schema))
File "/usr/lib/python2.7/site-packages/alembic/ddl/impl.py", line 118, in _exec
return conn.execute(construct, *multiparams, **params)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
return meth(self, multiparams, params)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
return connection._execute_ddl(self, multiparams, params)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 968, in _execute_ddl
compiled
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
context)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
util.raise_from_cause(newraise, exc_info)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
context)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
cursor.execute(statement, parameters)
File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute
result = self._query(query)
File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query
conn.query(q)
File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 841, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1029, in _read_query_result
result.read()
File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1312, in read
first_packet = self.connection._read_packet()
File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet
packet.check_error()
File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
err.raise_mysql_exception(self._data)
File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
raise errorclass(errno, errval)
sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'ovs_neutron.bsn_routerrules' doesn't exist") [SQL: u'ALTER TABLE bsn_routerrules ADD COLUMN tenant_id VARCHAR(255)']