Bug 1445917
| Summary: | rhosp-director: Upgrade osp9->10->11 fails with Error: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]: Failed to call refresh: neutron-db-manage upgrade heads returned 1 instead of one of [0] | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Alexander Chuzhoy <sasha> | |
| Component: | python-networking-bigswitch | Assignee: | mathieu bultel <mbultel> | |
| Status: | CLOSED ERRATA | QA Contact: | Amit Ugol <augol> | |
| Severity: | unspecified | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 11.0 (Ocata) | CC: | dbecker, dmacpher, ihrachys, jcoufal, jschluet, lbopf, lhh, mandreou, mbultel, mburns, mcornea, morazi, rhel-osp-director-maint, rhosp-bugs-internal, sathlang, sclewis, slinaber | |
| Target Milestone: | async | Keywords: | Triaged, ZStream | |
| Target Release: | 11.0 (Ocata) | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | python-networking-bigswitch-10.0.4-5.el7ost | Doc Type: | Known Issue | |
| Doc Text: |
A database upgrade issue hindered the OpenStack Networking (neutron) upgrade. This hindered the upgrade from OpenStack Platform 10 to 11. This fix corrects the neutron database upgrade issue.
This issue affects customers who previously upgraded from OpenStack Platform 9 to 10 and now aim to upgrade to OpenStack Platform 11.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1446169 (view as bug list) | Environment: | ||
| Last Closed: | 2017-06-15 16:56:13 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
[root@overcloud-controller-0 heat-admin]# rpm -qa | grep bigswitch openstack-neutron-bigswitch-lldp-10.0.4-3.el7ost.noarch python-networking-bigswitch-10.0.4-3.el7ost.noarch openstack-neutron-bigswitch-agent-10.0.4-3.el7ost.noarch [root@overcloud-controller-0 heat-admin]# yum info python-networking-bigswitch Loaded plugins: product-id, search-disabled-repos, subscription-manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Repository rhelosp-fdp-pending is listed more than once in the configuration Installed Packages Name : python-networking-bigswitch Arch : noarch Epoch : 2 Version : 10.0.4 Release : 3.el7ost Size : 888 k Repo : installed From repo : rhelosp-11.0-puddle Summary : Big Switch Networks neutron plugin for OpenStack Networking URL : https://pypi.python.org/pypi/networking-bigswitch License : ASL 2.0 Description : This package contains Big Switch Networks : neutron plugins and agents is this similar/duplicate for https://bugzilla.redhat.com/show_bug.cgi?id=1425432 or https://bugzilla.redhat.com/show_bug.cgi?id=1434843 ) Asking Ihar Hrachyshka as he solved the error in the marios mentioned bz in https://review.openstack.org/#/c/442663/ (In reply to marios from comment #3) > is this similar/duplicate for > https://bugzilla.redhat.com/show_bug.cgi?id=1425432 or > https://bugzilla.redhat.com/show_bug.cgi?id=1434843 ) so they are related (both fail on neutron alembic migrations failing) but the trace here is for the whole table missing bsn_routerrules sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'ovs_neutron.bsn_routerrules' doesn't exist") [SQL: u'ALTER TABLE bsn_routerrules ADD COLUMN tenant_id VARCHAR(255)'] but at BZ 1434843 its about missing column trying to update that tablebsn_routerrules : oslo_db.exception.DBError: (pymysql.err.InternalError) (1054, u"Unknown column 'upstream.tenant_id' in 'field list'") [SQL: 'UPDATE bsn_routerrules bsn INNER JOIN routers upstream ON bsn.router_id = upstream.id SET bsn.tenant_id = upstream.tenant_id;'] SO there is definitely a missing migration here - trying to make sense of the https://review.openstack.org/#/c/442663/ to see if that would also fix here but we really need someone from bigswitch involved here as chem said ihar is our best lead atm at least red hat side By the look of it, it seems that the db migration in https://github.com/openstack/networking-bigswitch/blob/10.0.4/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/kilo_add_base_bsn_plugin.py#L115..L127 isn't applied before 34f888285695_support_multiple_logical_routers.py (present in the stack trace) When we "blame" the file :
https://github.com/openstack/networking-bigswitch/blame/stable/mitaka/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/kilo_add_base_bsn_plugin.py
we can see that the bsn_* table creation were added later. Checking on the package we see that this commit:
https://github.com/openstack/networking-bigswitch/commit/2c33957b08478ac6b92f924c9fdbcaccacd4f4c4
is missing in the osp9 bigswitch package.
That's why this problem happens only when you start the upgrade from 9, the initial kilo db migration doesn't create the bsn_* tables. Then when we update them in osp11, it fails.
This should be a 9.z for the python-networking-bigswitch-2015.3.8-1.el7ost.noarch package.
This is blocking OSP9->OSP10->OSP11 upgrade. Could you please check this? Thank you. thanks to mcornea via irc might be related to this: 13:45 < mcornea> marios: bug 1444536 which is for osp10 is in modified so it is not in the latest osp10 puddle 13:45 < bhjf> Bug http://bugzilla.redhat.com/show_bug.cgi?id=1444536 urgent, unspecified, z3, jjoyce, python-networking-bigswitch, MODIFIED , python-networking-bigswitch package version in OSP9 is higher than the one in OSP10 or OSP11 might explain the 'missing' migration we don't actually have a patch here yet changing to Assigned Hi, this might be because one of the DB upgrade scripts was left behind in a different directory during git merge. We moved it to the right place (https://github.com/openstack/networking-bigswitch/commit/9491ff48b53fec70ad68dbd5ae804e3793751aac) but didn't update the version string, which is why its causing trouble. We'll update it right away. This will fix upgrade from 9 to 10. But 10 to 11 is pending since there are a few UT failures which block any change. The patch is on the way (https://review.openstack.org/#/c/459858/). Hi, the patch has merged for both newton and ocata. And the tag matches the version string as well. Could you give the systems a nudge to retry? Adjusting external tracker. > But 10 to 11 is pending since there are a few UT failures which block any change. The patch is on the way (https://review.openstack.org/#/c/459858/).
Sorry about the misunderstanding here - OSP 11 = Ocata and not master branch. The version bump in Newton and Ocata have merged. So 9 > 10 > 11 is a go now :)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:1474 |
rhosp-director: Upgrade osp9->10->11 fails with Error: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]: Failed to call refresh: neutron-db-manage upgrade heads returned 1 instead of one of [0] Environment: openstack-neutron-bigswitch-agent-10.0.4-3.el7ost.noarch openstack-neutron-bigswitch-lldp-10.0.4-3.el7ost.noarch python-networking-bigswitch-10.0.4-3.el7ost.noarch openstack-tripleo-heat-templates-compat-5.2.0-7.el7ost.noarch openstack-puppet-modules-10.0.0-1.el7ost.noarch openstack-tripleo-heat-templates-6.0.0-7.el7ost.noarch instack-undercloud-6.0.0-6.el7ost.noarch Steps to reproduce: Attempt to upgrade a setup (upgraded from OSP9 to OSP10) to OSP11. The setup was deployed with: openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --ceph-storage-scale 1 --neutron-network-type vxlan --neutron-tunnel-types vxlan --ntp-server 10.5.26.10 --timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml -e /home/stack/ssl-heat-templates/environments/enable-tls.yaml -e /home/stack/ssl-heat-templates/environments/inject-trust-anchor.yaml Upgrade command: openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml -e /home/stack/ssl-heat-templates/environments/enable-tls.yaml -e /home/stack/ssl-heat-templates/environments/inject-trust-anchor.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/tls-endpoints-public-ip.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps.yaml -e init-repo.yaml Result: resource_type: OS::Heat::StructuredDeployment physical_resource_id: 7d5869cb-1dff-4952-b18c-6f0753c929fa status: CREATE_FAILED status_reason: | Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6 deploy_stdout: | ... Notice: /Stage[main]/Ceilometer::Wsgi::Apache/Openstacklib::Wsgi::Apache[ceilometer_wsgi]/Apache::Vhost[ceilometer_wsgi]/Concat[10-ceilometer_wsgi.conf]/File[/etc/httpd/conf.d/10-ceilometer_wsgi.conf]/ensure: defined content as '{md5}61df7442c66de070e1efcfc7de53593f' Notice: /Stage[main]/Gnocchi::Wsgi::Apache/Openstacklib::Wsgi::Apache[gnocchi_wsgi]/Apache::Vhost[gnocchi_wsgi]/Concat[10-gnocchi_wsgi.conf]/File[/etc/httpd/conf.d/10-gnocchi_wsgi.conf]/ensure: defined content as '{md5}61ad09d00b642d95d8f3f3267ceca05b' Notice: /Stage[main]/Apache/Concat[/etc/httpd/conf/ports.conf]/File[/etc/httpd/conf/ports.conf]/content: content changed '{md5}7a382e7ed5dd62e2b1b4b35aeddfe373' to '{md5}9ddd0ca848715af7b4476dadf20b7062' Notice: /Stage[main]/Keystone::Deps/Anchor[keystone::config::end]: Triggered 'refresh' from 2 events Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]: Triggered 'refresh' from 1 events Notice: /Stage[main]/Keystone::Deps/Anchor[keystone::dbsync::end]: Triggered 'refresh' from 1 events Notice: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: Triggered 'refresh' from 1 events Notice: /Stage[main]/Keystone::Deps/Anchor[keystone::service::begin]: Triggered 'refresh' from 3 events Notice: /Stage[main]/Apache::Service/Service[httpd]: Triggered 'refresh' from 2 events Notice: Applied catalog in 392.16 seconds (truncated, view all with --long) deploy_stderr: | ... Warning: Scope(Oslo::Messaging::Rabbit[cinder_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead. Warning: Scope(Oslo::Messaging::Rabbit[keystone_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead. Warning: Scope(Oslo::Messaging::Rabbit[glance_api_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead. Warning: Scope(Oslo::Messaging::Rabbit[glance_registry_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead. Warning: Scope(Oslo::Messaging::Rabbit[neutron_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead. Warning: Scope(Oslo::Messaging::Rabbit[ceilometer_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead. Warning: Scope(Oslo::Messaging::Rabbit[aodh_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead. Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications. Error: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]: Failed to call refresh: neutron-db-manage upgrade heads returned 1 instead of one of [0] Error: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]: neutron-db-manage upgrade heads returned 1 instead of one of [0] Running neutron-db-manage upgrade heads results in: [root@overcloud-controller-0 heat-admin]# neutron-db-manage upgrade heads INFO [alembic.runtime.migration] Context impl MySQLImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. Running upgrade for neutron ... INFO [alembic.runtime.migration] Context impl MySQLImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. OK INFO [alembic.runtime.migration] Context impl MySQLImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. Running upgrade for bsn_extensions ... INFO [alembic.runtime.migration] Context impl MySQLImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. INFO [alembic.runtime.migration] Running upgrade 1ef57200f387 -> 34f888285695, support multiple logical routers Traceback (most recent call last): File "/bin/neutron-db-manage", line 10, in <module> sys.exit(main()) File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 686, in main return_val |= bool(CONF.command.func(config, CONF.command.name)) File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 207, in do_upgrade desc=branch, sql=CONF.command.sql) File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 108, in do_alembic_command getattr(alembic_command, cmd)(config, *args, **kwargs) File "/usr/lib/python2.7/site-packages/alembic/command.py", line 174, in upgrade script.run_env() File "/usr/lib/python2.7/site-packages/alembic/script/base.py", line 407, in run_env util.load_python_file(self.dir, 'env.py') File "/usr/lib/python2.7/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file module = load_module_py(module_id, path) File "/usr/lib/python2.7/site-packages/alembic/util/compat.py", line 79, in load_module_py mod = imp.load_source(module_id, path, fp) File "/usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/env.py", line 86, in <module> run_migrations_online() File "/usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/env.py", line 77, in run_migrations_online context.run_migrations() File "<string>", line 8, in run_migrations File "/usr/lib/python2.7/site-packages/alembic/runtime/environment.py", line 797, in run_migrations self.get_context().run_migrations(**kw) File "/usr/lib/python2.7/site-packages/alembic/runtime/migration.py", line 312, in run_migrations step.migration_fn(**kw) File "/usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/34f888285695_support_multiple_logical_routers.py", line 38, in upgrade sa.Column('tenant_id', sa.String(255), nullable=True)) File "<string>", line 8, in add_column File "<string>", line 3, in add_column File "/usr/lib/python2.7/site-packages/alembic/operations/ops.py", line 1535, in add_column return operations.invoke(op) File "/usr/lib/python2.7/site-packages/alembic/operations/base.py", line 318, in invoke return fn(self, operation) File "/usr/lib/python2.7/site-packages/alembic/operations/toimpl.py", line 123, in add_column schema=schema File "/usr/lib/python2.7/site-packages/alembic/ddl/impl.py", line 172, in add_column self._exec(base.AddColumn(table_name, column, schema=schema)) File "/usr/lib/python2.7/site-packages/alembic/ddl/impl.py", line 118, in _exec return conn.execute(construct, *multiparams, **params) File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute return meth(self, multiparams, params) File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection return connection._execute_ddl(self, multiparams, params) File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 968, in _execute_ddl compiled File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context context) File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception util.raise_from_cause(newraise, exc_info) File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause reraise(type(exception), exception, tb=exc_tb) File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context context) File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute cursor.execute(statement, parameters) File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute result = self._query(query) File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query conn.query(q) File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 841, in query self._affected_rows = self._read_query_result(unbuffered=unbuffered) File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1029, in _read_query_result result.read() File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1312, in read first_packet = self.connection._read_packet() File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet packet.check_error() File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error err.raise_mysql_exception(self._data) File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception raise errorclass(errno, errval) sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'ovs_neutron.bsn_routerrules' doesn't exist") [SQL: u'ALTER TABLE bsn_routerrules ADD COLUMN tenant_id VARCHAR(255)']