Hide Forgot
Description of problem: UPGRADE STEP 3: major-upgrade-pacemaker.yaml (upgrade the controllers) runs into this issue on one test machine Version-Release number of selected component (if applicable): 7.3 -> 8.0 How reproducible: On one machine (snapshot) all of the time after rollback to before upgrade to 8.0; not on other machines Steps to Reproduce: 1. follow internal upgrade guide 2.openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates --stack overcloud -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e ${template_base_dir}/network-environment.yaml -e ${template_base_dir}/enable-tls.yaml -e ${template_base_dir}/cloudname.yaml -e ${template_base_dir}/inject-trust-anchor.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml -e ${template_base_dir}/rhos-release-8.yaml --control-flavor control --compute-flavor compute --control-scale 3 --compute-scale 1 --ceph-storage-scale 0 --ntp-server 10.5.26.10 --neutron-network-type vxlan --neut Actual results: Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates ERROR: Remote error: DBError (pymysql.err.InternalError) (1054, u"Unknown column 'resource.root_stack_id' in 'field list'") [SQL: u'SELECT resource.created_at AS resource_created_at, resource.action AS resource_action, resource.status AS resource_status, resource.status_reason AS resource_status_reason, resource.id AS resource_id, resource.uuid AS resource_uuid, resource.name AS resource_name, resource.nova_instance AS resource_nova_instance, resource.rsrc_metadata AS resource_rsrc_metadata, resource.stack_id AS resource_stack_id, resource.root_stack_id AS resource_root_stack_id, resource.updated_at AS resource_updated_at, resource.properties_data AS resource_properties_data, resource.properties_data_encrypted AS resource_properties_data_encrypted, resource.engine_id AS resource_engine_id, resource.atomic_key AS resource_atomic_key, resource.needed_by AS resource_needed_by, resource.requires AS resource_requires, resource.replaces AS resource_replaces, resource.replaced_by AS resource_replaced_by, resource.current_template_id AS resource_current_template_id, resource_data_1.created_at AS resource_data_1_created_at, resource_data_1.updated_at AS resource_data_1_updated_at, resource_data_1.id AS resource_data_1_id, resource_data_1.`key` AS resource_data_1_key, resource_data_1.value AS resource_data_1_value, resource_data_1.redact AS resource_data_1_redact, resource_data_1.decrypt_method AS resource_data_1_decrypt_method, resource_data_1.resource_id AS resource_data_1_resource_id \nFROM resource LEFT OUTER JOIN resource_data AS resource_data_1 ON resource.id = resource_data_1.resource_id \nWHERE resource.stack_id = %s'] [parameters: (u'662faabb-5a9e-41cd-85bf-397d80580e39',)] MariaDB [heat]> describe resource; +---------------------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +---------------------------+--------------+------+-----+---------+----------------+ | nova_instance | varchar(255) | YES | | NULL | | | name | varchar(255) | YES | | NULL | | | created_at | datetime | YES | | NULL | | | updated_at | datetime | YES | | NULL | | | status | varchar(255) | YES | | NULL | | | status_reason | text | YES | | NULL | | | stack_id | varchar(36) | NO | MUL | NULL | | | rsrc_metadata | longtext | YES | | NULL | | | action | varchar(255) | YES | | NULL | | | properties_data | longtext | YES | | NULL | | | uuid | varchar(36) | YES | UNI | NULL | | | id | int(11) | NO | PRI | NULL | auto_increment | | engine_id | varchar(36) | YES | | NULL | | | atomic_key | int(11) | YES | | NULL | | | needed_by | longtext | YES | | NULL | | | requires | longtext | YES | | NULL | | | replaces | int(11) | YES | | NULL | | | replaced_by | int(11) | YES | | NULL | | | current_template_id | int(11) | YES | MUL | NULL | | | properties_data_encrypted | tinyint(1) | YES | | NULL | | +---------------------------+--------------+------+-----+---------+----------------+ 20 rows in set (0.00 sec) Expected results: MariaDB [heat]> describe resource; +---------------------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +---------------------------+--------------+------+-----+---------+----------------+ | nova_instance | varchar(255) | YES | | NULL | | | name | varchar(255) | YES | | NULL | | | created_at | datetime | YES | | NULL | | | updated_at | datetime | YES | | NULL | | | status | varchar(255) | YES | | NULL | | | status_reason | text | YES | | NULL | | | stack_id | varchar(36) | NO | MUL | NULL | | | rsrc_metadata | longtext | YES | | NULL | | | action | varchar(255) | YES | | NULL | | | properties_data | longtext | YES | | NULL | | | uuid | varchar(36) | YES | UNI | NULL | | | id | int(11) | NO | PRI | NULL | auto_increment | | engine_id | varchar(36) | YES | | NULL | | | atomic_key | int(11) | YES | | NULL | | | needed_by | longtext | YES | | NULL | | | requires | longtext | YES | | NULL | | | replaces | int(11) | YES | | NULL | | | replaced_by | int(11) | YES | | NULL | | | current_template_id | int(11) | YES | MUL | NULL | | | properties_data_encrypted | tinyint(1) | YES | | NULL | | | root_stack_id | varchar(36) | YES | MUL | NULL | | +---------------------------+--------------+------+-----+---------+----------------+ 21 rows in set (0.00 sec) Additional info:
It sounds like the undercloud heat packages have been upgraded but the db sync hasn't been run. This can be done by running: heat-manage db_sync Is it known why this step was missed during the undercloud upgrade?
It may not be a docs bug - we need elaboration on exactly what was done in this part: > How reproducible: > On one machine (snapshot) all of the time after rollback to before upgrade > to 8.0; not on other machines > > Steps to Reproduce: > 1. follow internal upgrade guide
This bug did not make the OSP 8.0 release. It is being deferred to OSP 10.
Closing for no more info. Please re-open if more info available.
Hi, We bumped into this during an upgrade from 7 to 8 recently. After the undercloud upgrade, had run into: ~~~ [stack@undercloud ~]$ heat resource-list overcloud ERROR: Remote error: DBError (pymysql.err.InternalError) (1054, u"Unknown column 'resource.root_stack_id' in 'field list'") [SQL: u'SELECT resource.created_at AS resource_created_at, resource.action AS resource_action, resource.status AS resource_status, resource.status_reason AS resource_status_reason, resource.id AS resource_id, resource.uuid AS resource_uuid, resource.name AS resource_name, resource.nova_instance AS resource_nova_instance, resource.rsrc_metadata AS resource_rsrc_metadata, resource.stack_id AS resource_stack_id, resource.root_stack_id AS resource_root_stack_id, resource.updated_at AS resource_updated_at, resource.properties_data AS resource_properties_data, resource.properties_data_encrypted AS resource_properties_data_encrypted, resource.engine_id AS resource_engine_id, resource.atomic_key AS resource_atomic_key, resource.needed_by AS resource_needed_by, resource.requires AS resource_requires, resource.replaces AS resource_replaces, resource.replaced_by AS resource_replaced_by, resource.current_template_id AS resource_current_template_id, resource_data_1.created_at AS resource_data_1_created_at, resource_data_1.updated_at AS resource_data_1_updated_at, resource_data_1.id AS resource_data_1_id, resource_data_1.`key` AS resource_data_1_key, resource_data_1.value AS resource_data_1_value, resource_data_1.redact AS resource_data_1_redact, resource_data_1.decrypt_method AS resource_data_1_decrypt_method, resource_data_1.resource_id AS resource_data_1_resource_id \nFROM resource LEFT OUTER JOIN resource_data AS resource_data_1 ON resource.id = resource_data_1.resource_id \nWHERE resource.stack_id = %s'] [parameters: (u'0240e07c-d27b-46c8-a177-1f59c6cf5650',)] [u' [stack@ussmkostk5000 ~]$ ~~~ This command from the heat-manage db_sync failed: ~~~ ALTER TABLE raw_template DROP FOREIGN KEY predecessor_fkey_ref ~~~ compared your DB in staging to the production DB: Here is the one for the QA director: ~~~ MariaDB [heat]> show create table raw_template -> ; +--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Table | Create Table | +--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | raw_template | CREATE TABLE `raw_template` ( `id` int(11) NOT NULL AUTO_INCREMENT, `created_at` datetime DEFAULT NULL, `updated_at` datetime DEFAULT NULL, `template` longtext, `files` longtext, `environment` longtext, `predecessor` int(11) DEFAULT NULL, PRIMARY KEY (`id`), KEY `predecessor_fkey_ref` (`predecessor`) ) ENGINE=InnoDB AUTO_INCREMENT=29242 DEFAULT CHARSET=utf8 | +--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec) MariaDB [heat]> ~~~ Checked the prodction db: ~~~ MariaDB [heat]> show create table raw_template -> ; +--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Table | Create Table | +--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | raw_template | CREATE TABLE `raw_template` ( `id` int(11) NOT NULL AUTO_INCREMENT, `created_at` datetime DEFAULT NULL, `updated_at` datetime DEFAULT NULL, `template` longtext, `files` longtext, `environment` longtext, `predecessor` int(11) DEFAULT NULL, PRIMARY KEY (`id`), KEY `predecessor_fkey_ref` (`predecessor`), CONSTRAINT `predecessor_fkey_ref` FOREIGN KEY (`predecessor`) REFERENCES `raw_template` (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=57857 DEFAULT CHARSET=utf8 | +--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.02 sec) MariaDB [heat]> ~~~ added the constraint again: ~~~ ALTER TABLE `raw_template` ADD CONSTRAINT `predecessor_fkey_ref` FOREIGN KEY (`predecessor`) REFERENCES `raw_template` (`id`) ; ~~~ After which the migration of the db version from 63 to 65 worked: ~~~ heat-manage db_version 65 ~~~ > Versions are in : /usr/lib/python2.7/site-packages/head/db/sqlalchemy/migrate_repo/versions
This was the failure from heat-manage db_sync that had failed and that lead to the above procedure: [stack@undercloud ~]$ sudo heat-manage db_sync 2018-07-17 18:49:52.930 8499 WARNING oslo_config.cfg [-] Option "verbose" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future. 2018-07-17 18:49:52.964 8499 INFO migrate.versioning.api [-] 63 -> 64... 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters [-] DBAPIError exception wrapped from (pymysql.err.InternalError) (1025, u"Error on rename of './heat/raw_template' to './heat/#sql2-ce4-1c4d' (errno: 152)") [SQL: u'ALTER TABLE raw_template DROP FOREIGN KEY predecessor_fkey_ref'] 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last): 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters context) 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters) 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 146, in execute 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters result = self._query(query) 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 296, in _query 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters conn.query(q) 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 781, in query 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 942, in _read_query_result 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters result.read() 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1138, in read 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters first_packet = self.connection._read_packet() 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 906, in _read_packet 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters packet.check_error() 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 367, in check_error 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters err.raise_mysql_exception(self._data) 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 120, in raise_mysql_exception 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters _check_mysql_exception(errinfo) 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 115, in _check_mysql_exception 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters raise InternalError(errno, errorvalue) 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters InternalError: (1025, u"Error on rename of './heat/raw_template' to './heat/#sql2-ce4-1c4d' (errno: 152)") 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters ERROR: (pymysql.err.InternalError) (1025, u"Error on rename of './heat/raw_template' to './heat/#sql2-ce4-1c4d' (errno: 152)") [SQL: u'ALTER TABLE raw_template DROP FOREIGN KEY predecessor_fkey_ref']