Bug 1692838
| Summary: | FFU: overcloud upgrade run fails on openstack-neutron-bigswitch-agent.noarch not present | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Ronnie Rasouli <rrasouli> | |
| Component: | python-networking-bigswitch | Assignee: | Nate Johnston <njohnston> | |
| Status: | CLOSED ERRATA | QA Contact: | Ofer Blaut <oblaut> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | urgent | |||
| Version: | 10.0 (Newton) | CC: | amodi, amuller, cswanson, jschluet, jthomas, lhh, mburns, mflusche, morazi, nalmond, njohnston, rbartal, sdunne, slinaber, tquinlan, yprokule, yrachman | |
| Target Milestone: | --- | Keywords: | Regression, Triaged, ZStream | |
| Target Release: | 11.0 (Ocata) | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | python-networking-bigswitch-10.46.4-6.el7ost | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1709893 (view as bug list) | Environment: | ||
| Last Closed: | 2019-06-18 17:34:58 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1614361, 1709893 | |||
Yurii, Do you have the file networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/7db8cd315b95_clear_consistency_db.py in this deployment? I see that migration existing in the code tree; I don't see why alembic would not be able to find it. Upstream: https://opendev.org/openstack/networking-bigswitch/src/branch/stable/newton/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/7db8cd315b95_clear_consistency_db.py Thanks, Nate (In reply to Nate Johnston from comment #3) > Yurii, > > Do you have the file > networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/ > versions/7db8cd315b95_clear_consistency_db.py in this deployment? I see > that migration existing in the code tree; I don't see why alembic would not > be able to find it. > > Upstream: > https://opendev.org/openstack/networking-bigswitch/src/branch/stable/newton/ > networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/ > versions/7db8cd315b95_clear_consistency_db.py > > Thanks, > > Nate Hi Nate, Unfortunately I don't have env to check it, but hopefully Ronnie can assist with it --- Yurii On controller-0 we can see the bigswitch rpm's [root@controller-0 ~]# rpm -qa | grep big openstack-neutron-bigswitch-agent-10.0.8-1.el7ost.noarch jbigkit-libs-2.0-11.el7.x86_64 openstack-neutron-bigswitch-lldp-10.0.8-1.el7ost.noarch python-networking-bigswitch-10.0.8-1.el7ost.noarch rubygem-bigdecimal-1.2.0-34.el7_6.x86_64 The value of version_num in bsn_alembic_version is set to "7db8cd315b95", which is not a migration contained in the version of the python-networking-bigswitch package that is being upgraded to. In fact, the string "7db8cd315b95" does not exist *anywhere* on the filesystem of controller-0 except in the MariaDB database file. I did a "grep -rl 7db8cd315b95 /" on controller-0 and it came back with nothing but MariaDB data files. This means that the string must be coming from somewhere else, but I can't see anywhere what that would be. Can you check and see if the Here's MariaDB: > MariaDB [(none)]> use ovs_neutron; > Reading table information for completion of table and column names > You can turn off this feature to get a quicker startup with -A > > Database changed > MariaDB [ovs_neutron]> select * from bsn_alembic_version; > +--------------+ > | version_num | > +--------------+ > | 7db8cd315b95 | > +--------------+ > 1 row in set (0.00 sec) The contents of the package that the upgrade is installing, which corresponds to OSP 11: > [root@controller-0 tmp]# rpm -ql -p /var/tmp/python-networking-bigswitch-10.0.8-1.el7ost.noarch.rpm | grep alembic > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/README > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/__init__.py > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/__init__.pyc > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/__init__.pyo > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/env.py > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/env.pyc > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/env.pyo > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/script.py.mako > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/1086543dfc0f_update_column_type_for_testpath_result.py > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/1086543dfc0f_update_column_type_for_testpath_result.pyc > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/1086543dfc0f_update_column_type_for_testpath_result.pyo > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/1ef57200f387_testpath_modify_uniq_constraint.py > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/1ef57200f387_testpath_modify_uniq_constraint.pyc > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/1ef57200f387_testpath_modify_uniq_constraint.pyo > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/2dc6f1b7c0a1_testpath_add_column_logical_path.py > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/2dc6f1b7c0a1_testpath_add_column_logical_path.pyc > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/2dc6f1b7c0a1_testpath_add_column_logical_path.pyo > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/34f888285695_support_multiple_logical_routers.py > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/34f888285695_support_multiple_logical_routers.pyc > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/34f888285695_support_multiple_logical_routers.pyo > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/HEAD > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/f774eae87144_rename_tenant_to_project.py > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/f774eae87144_rename_tenant_to_project.pyc > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/f774eae87144_rename_tenant_to_project.pyo > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/kilo_add_base_bsn_plugin.py > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/kilo_add_base_bsn_plugin.pyc > /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/kilo_add_base_bsn_plugin.pyo The value in the bsn_alembic_version table is dictated by the contents of the file networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions/HEAD, which is "2dc6f1b7c0a1". So I set the database correctly: > MariaDB [(none)]> use ovs_neutron; > Reading table information for completion of table and column names > You can turn off this feature to get a quicker startup with -A > > Database changed > MariaDB [ovs_neutron]> update bsn_alembic_version set version_num = "2dc6f1b7c0a1"; > Query OK, 1 row affected (0.00 sec) > Rows matched: 1 Changed: 1 Warnings: 0 > > MariaDB [ovs_neutron]> select * from bsn_alembic_version; > +--------------+ > | version_num | > +--------------+ > | 2dc6f1b7c0a1 | > +--------------+ > 1 row in set (0.00 sec) Once I do that, the alembic migrations proceed without issue: > [root@controller-0 tmp]# neutron-db-manage upgrade head > INFO [alembic.runtime.migration] Context impl MySQLImpl. > INFO [alembic.runtime.migration] Will assume non-transactional DDL. > Running upgrade for neutron ... > INFO [alembic.runtime.migration] Context impl MySQLImpl. > INFO [alembic.runtime.migration] Will assume non-transactional DDL. > OK > INFO [alembic.runtime.migration] Context impl MySQLImpl. > INFO [alembic.runtime.migration] Will assume non-transactional DDL. > Running upgrade for bsn_extensions ... > INFO [alembic.runtime.migration] Context impl MySQLImpl. > INFO [alembic.runtime.migration] Will assume non-transactional DDL. > OK > INFO [alembic.runtime.migration] Context impl MySQLImpl. > INFO [alembic.runtime.migration] Will assume non-transactional DDL. > Running upgrade for networking-cisco ... > INFO [alembic.runtime.migration] Context impl MySQLImpl. > INFO [alembic.runtime.migration] Will assume non-transactional DDL. > OK > INFO [alembic.runtime.migration] Context impl MySQLImpl. > INFO [alembic.runtime.migration] Will assume non-transactional DDL. > Running upgrade for networking-odl ... > INFO [alembic.runtime.migration] Context impl MySQLImpl. > INFO [alembic.runtime.migration] Will assume non-transactional DDL. > OK > INFO [alembic.runtime.migration] Context impl MySQLImpl. > INFO [alembic.runtime.migration] Will assume non-transactional DDL. > Running upgrade for neutron-lbaas ... > INFO [alembic.runtime.migration] Context impl MySQLImpl. > INFO [alembic.runtime.migration] Will assume non-transactional DDL. > OK The remaining question is where the spurious value "7db8cd315b95" came from. There is no evidence to show this anywhere on controller-0, nor on undercloud-0. Do you have any other suggestions for where to look? At this point it would have to have come from something external to the host which did not leave an artifact of it's presence on the host. I will further note that the file in question was added in networking-bigswitch version 10.46.10 (see [1] for 10.46.9 and [2] for 10.46.10 to see the file appear). I don't believe we have these versions available in our environment but if there was something that was trying to get the most recent version with the same major version number then you would end up with references to 7db8cd315b95 on the system. I don't have the slightest clue if that is possible, because we have wandered out my the scope of my expertise. Perhaps someone skilled in the automation aspect can comment. [1] https://opendev.org/openstack/networking-bigswitch/src/tag/10.46.9/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions [2] https://opendev.org/openstack/networking-bigswitch/src/tag/10.46.10/networking_bigswitch/plugins/bigswitch/db/migration/alembic_migrations/versions OK, I have located the issue. It appears that the file 7db8cd315b95_clear_consistency_db.py exists in our OSP 10 version of the python-network-bigswitch package but does not exist in the OSP 11 version. I don't see anything in the git log that explains why this is so. I have created a downstream-only change [1] to fix this, as we appear to have deviated from upstream at some point by deleting this file. [1] https://code.engineering.redhat.com/gerrit/167469 *** Bug 1702767 has been marked as a duplicate of this bug. *** Build successful for python-networking-bigswitch-10.46.4-3.el7ost; thanks eggs! The bigswitch issue has been resolved, however neutron DB migrate still failing. https://bugzilla.redhat.com/show_bug.cgi?id=1710222 Approving hotfix. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1541 |
Description of problem: FFU upgrade run fails on Neutron package update task due to broken dependencies with openstack-neutron-bigswitch-agent.noarch Version-Release number of selected component (if applicable): core_puddle: 2019-03-21.2 How reproducible: 100% Steps to Reproduce: 1.deploy osp10 simple 3 controllers 1 compute 2.perform FFU prepare 3. preform FFU upgrade run Actual results: The upgrade process fails with the error above Expected results: proceed with FFU process without package resoultion errors Additional info: u'TASK [Neutron package update] **************************************************', u'Saturday 23 March 2019 11:28:31 -0400 (0:00:00.467) 0:13:38.433 ******** ', u'changed: [controller-0] => {"changed": true, "cmd": "yum -y update openstack-neutron*", "delta": "0:00:20.264738", "end": "2019-03-23 15:28:51.448857", "rc": 0, "start": "2019-03-23 15:28:31.184119", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: product-id, search-disabled-repos, subscription-manager\\nThis system is not registered with an entitlement server. You can use subscription-manager to register.\\nResolving Dependencies\\n--> Running transaction check\\n---> Package openstack-neutron.noarch 1:9.4.1-38.el7ost will be updated\\n---> Package openstack-neutron.noarch 1:10.0.5-1.el7ost will be an update\\n---> Package openstack-neutron-bigswitch-agent.noarch 2:9.42.14-1.el7ost will be updated\\n---> Package openstack-neutron-bigswitch-agent.noarch 2:10.0.8-1.el7ost will be an update\\n--> Processing Dependency: python-networking-bigswitch = 2:10.0.8-1.el7ost for package: 2:openstack-neutron-bigswitch-agent-10.0.8-1.el7ost.noarch\\n---> Package openstack-neutron-bigswitch-lldp.noarch 2:9.42.14-1.el7ost will be updated\\n---> Package openstack-neutron-bigswitch-lldp.noarch 2:10.0.8-1.el7ost will be an update\\n---> Package openstack-neutron-common.noarch 1:9.4.1-38.el7ost will be updated\\n---> Package openstack-neutron-common.noarch 1:10.0.5-1.el7ost will be an update\\n--> Processing Dependency: python-neutron = 1:10.0.5-1.el7ost for package: 1:openstack-neutron-common-10.0.5-1.el7ost.noarch\\n---> Package openstack-neutron-lbaas.noarch 1:9.2.2-8.el7ost will be updated\\n---> Package openstack-neutron-lbaas.noarch 1:10.0.1-13.el7ost will be an update\\n--> Processing Dependency: python-neutron-lbaas = 1:10.0.1-13.el7ost for package: 1:openstack-neutron-lbaas-10.0.1-13.el7ost.noarch\\n--> Processing Dependency: python-neutron-lib >= 1.1.0 for package: 1:openstack-neutron-lbaas-10.0.1-13.el7ost.noarch\\n---> Package openstack-neutron-metering-agent.noarch 1:9.4.1-38.el7ost will be updated\\n---> Package openstack-neutron-metering-agent.noarch 1:10.0.5-1.el7ost will be an update\\n---> Package openstack-neutron-ml2.noarch 1:9.4.1-38.el7ost will be updated\\n---> Package openstack-neutron-ml2.noarch 1:10.0.5-1.el7ost will be an update\\n---> Package openstack-neutron-openvswitch.noarch 1:9.4.1-38.el7ost will be updated\\n---> Package openstack-neutron-openvswitch.noarch 1:10.0.5-1.el7ost will be an update\\n---> Package openstack-neutron-sriov-nic-agent.noarch 1:9.4.1-38.el7ost will be updated\\n---> Package openstack-neutron-sriov-nic-agent.noarch 1:10.0.5-1.el7ost will be an update\\n--> Running transaction check\\n---> Package python-networking-bigswitch.noarch 2:9.42.14-1.el7ost will be updated\\n---> Package python-networking-bigswitch.noarch 2:10.0.8-1.el7ost will be an update\\n---> Package python-neutron.noarch 1:9.4.1-38.el7ost will be updated\\n--> Processing Dependency: python-neutron = 1:9.4.1-38.el7ost for package: 1:python-neutron-tests-9.4.1-38.el7ost.noarch\\n---> Package python-neutron.noarch 1:10.0.5-1.el7ost will be an update\\n--> Processing Dependency: python-weakrefmethod >= 1.0.2 for package: 1:python-neutron-10.0.5-1.el7ost.noarch\\n--> Processing Dependency: python-pyroute2 >= 0.4.3 for package: 1:python-neutron-10.0.5-1.el7ost.noarch\\n---> Package python-neutron-lbaas.noarch 1:9.2.2-8.el7ost will be updated\\n---> Package python-neutron-lbaas.noarch 1:10.0.1-13.el7ost will be an update\\n---> Package python-neutron-lib.noarch 0:0.4.0-1.el7ost will be updated\\n---> Package python-neutron-lib.noarch 0:1.1.0-1.el7ost will be an update\\n--> Running transaction check\\n---> Package python-neutron-tests.noarch 1:9.4.1-38.el7ost will be updated\\n---> Package python-neutron-tests.noarch 1:10.0.5-1.el7ost will be an update\\n--> Processing Dependency: python-oslo-db-tests >= 4.10.0 for package: 1:python-neutron-tests-10.0.5-1.el7ost.noarch\\n---> Package python-pyroute2.noarch 0:0.4.8-1.el7ost will be installed\\n---> Package python-weakrefmethod.noarch 0:1.0.2-3.el7ost will be installed\\n--> Running transaction check\\n---> Package python-oslo-db-tests.noarch 0:4.17.1-1.el7ost will be installed\\n--> Finished Dependency Resolution\\n\\nDependencies Resolved\\n\\n======