| Summary: | Unknown column 'resource.root_stack_id' on upgrade from OSP 7.3 to 8 | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Andreas Karis <akaris> |
| Component: | openstack-tripleo | Assignee: | James Slagle <jslagle> |
| Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | Arik Chernetsky <achernet> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.0 (Kilo) | CC: | akaris, jcoufal, mburns, rhel-osp-director-maint, rybrown, sbaker |
| Target Milestone: | --- | ||
| Target Release: | 10.0 (Newton) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-10-10 04:03:14 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
It sounds like the undercloud heat packages have been upgraded but the db sync hasn't been run. This can be done by running: heat-manage db_sync Is it known why this step was missed during the undercloud upgrade? It may not be a docs bug - we need elaboration on exactly what was done in this part:
> How reproducible:
> On one machine (snapshot) all of the time after rollback to before upgrade
> to 8.0; not on other machines
>
> Steps to Reproduce:
> 1. follow internal upgrade guide
This bug did not make the OSP 8.0 release. It is being deferred to OSP 10. Closing for no more info. Please re-open if more info available. Hi,
We bumped into this during an upgrade from 7 to 8 recently.
After the undercloud upgrade, had run into:
~~~
[stack@undercloud ~]$ heat resource-list overcloud
ERROR: Remote error: DBError (pymysql.err.InternalError) (1054, u"Unknown column 'resource.root_stack_id' in 'field list'") [SQL: u'SELECT resource.created_at AS resource_created_at, resource.action AS resource_action, resource.status AS resource_status, resource.status_reason AS resource_status_reason, resource.id AS resource_id, resource.uuid AS resource_uuid, resource.name AS resource_name, resource.nova_instance AS resource_nova_instance, resource.rsrc_metadata AS resource_rsrc_metadata, resource.stack_id AS resource_stack_id, resource.root_stack_id AS resource_root_stack_id, resource.updated_at AS resource_updated_at, resource.properties_data AS resource_properties_data, resource.properties_data_encrypted AS resource_properties_data_encrypted, resource.engine_id AS resource_engine_id, resource.atomic_key AS resource_atomic_key, resource.needed_by AS resource_needed_by, resource.requires AS resource_requires, resource.replaces AS resource_replaces, resource.replaced_by AS resource_replaced_by, resource.current_template_id AS resource_current_template_id, resource_data_1.created_at AS resource_data_1_created_at, resource_data_1.updated_at AS resource_data_1_updated_at, resource_data_1.id AS resource_data_1_id, resource_data_1.`key` AS resource_data_1_key, resource_data_1.value AS resource_data_1_value, resource_data_1.redact AS resource_data_1_redact, resource_data_1.decrypt_method AS resource_data_1_decrypt_method, resource_data_1.resource_id AS resource_data_1_resource_id \nFROM resource LEFT OUTER JOIN resource_data AS resource_data_1 ON resource.id = resource_data_1.resource_id \nWHERE resource.stack_id = %s'] [parameters: (u'0240e07c-d27b-46c8-a177-1f59c6cf5650',)]
[u'
[stack@ussmkostk5000 ~]$
~~~
This command from the heat-manage db_sync failed:
~~~
ALTER TABLE raw_template DROP FOREIGN KEY predecessor_fkey_ref
~~~
compared your DB in staging to the production DB:
Here is the one for the QA director:
~~~
MariaDB [heat]> show create table raw_template
-> ;
+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| raw_template | CREATE TABLE `raw_template` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`created_at` datetime DEFAULT NULL,
`updated_at` datetime DEFAULT NULL,
`template` longtext,
`files` longtext,
`environment` longtext,
`predecessor` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `predecessor_fkey_ref` (`predecessor`)
) ENGINE=InnoDB AUTO_INCREMENT=29242 DEFAULT CHARSET=utf8 |
+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
MariaDB [heat]>
~~~
Checked the prodction db:
~~~
MariaDB [heat]> show create table raw_template
-> ;
+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| raw_template | CREATE TABLE `raw_template` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`created_at` datetime DEFAULT NULL,
`updated_at` datetime DEFAULT NULL,
`template` longtext,
`files` longtext,
`environment` longtext,
`predecessor` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `predecessor_fkey_ref` (`predecessor`),
CONSTRAINT `predecessor_fkey_ref` FOREIGN KEY (`predecessor`) REFERENCES `raw_template` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=57857 DEFAULT CHARSET=utf8 |
+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.02 sec)
MariaDB [heat]>
~~~
added the constraint again:
~~~
ALTER TABLE `raw_template` ADD CONSTRAINT `predecessor_fkey_ref` FOREIGN KEY (`predecessor`) REFERENCES `raw_template` (`id`) ;
~~~
After which the migration of the db version from 63 to 65 worked:
~~~
heat-manage db_version
65
~~~
> Versions are in : /usr/lib/python2.7/site-packages/head/db/sqlalchemy/migrate_repo/versions
This was the failure from heat-manage db_sync that had failed and that lead to the above procedure: [stack@undercloud ~]$ sudo heat-manage db_sync 2018-07-17 18:49:52.930 8499 WARNING oslo_config.cfg [-] Option "verbose" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future. 2018-07-17 18:49:52.964 8499 INFO migrate.versioning.api [-] 63 -> 64... 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters [-] DBAPIError exception wrapped from (pymysql.err.InternalError) (1025, u"Error on rename of './heat/raw_template' to './heat/#sql2-ce4-1c4d' (errno: 152)") [SQL: u'ALTER TABLE raw_template DROP FOREIGN KEY predecessor_fkey_ref'] 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last): 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters context) 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters) 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 146, in execute 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters result = self._query(query) 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 296, in _query 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters conn.query(q) 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 781, in query 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 942, in _read_query_result 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters result.read() 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1138, in read 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters first_packet = self.connection._read_packet() 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 906, in _read_packet 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters packet.check_error() 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 367, in check_error 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters err.raise_mysql_exception(self._data) 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 120, in raise_mysql_exception 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters _check_mysql_exception(errinfo) 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 115, in _check_mysql_exception 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters raise InternalError(errno, errorvalue) 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters InternalError: (1025, u"Error on rename of './heat/raw_template' to './heat/#sql2-ce4-1c4d' (errno: 152)") 2018-07-17 18:49:57.920 8499 ERROR oslo_db.sqlalchemy.exc_filters ERROR: (pymysql.err.InternalError) (1025, u"Error on rename of './heat/raw_template' to './heat/#sql2-ce4-1c4d' (errno: 152)") [SQL: u'ALTER TABLE raw_template DROP FOREIGN KEY predecessor_fkey_ref'] |
Description of problem: UPGRADE STEP 3: major-upgrade-pacemaker.yaml (upgrade the controllers) runs into this issue on one test machine Version-Release number of selected component (if applicable): 7.3 -> 8.0 How reproducible: On one machine (snapshot) all of the time after rollback to before upgrade to 8.0; not on other machines Steps to Reproduce: 1. follow internal upgrade guide 2.openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates --stack overcloud -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e ${template_base_dir}/network-environment.yaml -e ${template_base_dir}/enable-tls.yaml -e ${template_base_dir}/cloudname.yaml -e ${template_base_dir}/inject-trust-anchor.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml -e ${template_base_dir}/rhos-release-8.yaml --control-flavor control --compute-flavor compute --control-scale 3 --compute-scale 1 --ceph-storage-scale 0 --ntp-server 10.5.26.10 --neutron-network-type vxlan --neut Actual results: Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates ERROR: Remote error: DBError (pymysql.err.InternalError) (1054, u"Unknown column 'resource.root_stack_id' in 'field list'") [SQL: u'SELECT resource.created_at AS resource_created_at, resource.action AS resource_action, resource.status AS resource_status, resource.status_reason AS resource_status_reason, resource.id AS resource_id, resource.uuid AS resource_uuid, resource.name AS resource_name, resource.nova_instance AS resource_nova_instance, resource.rsrc_metadata AS resource_rsrc_metadata, resource.stack_id AS resource_stack_id, resource.root_stack_id AS resource_root_stack_id, resource.updated_at AS resource_updated_at, resource.properties_data AS resource_properties_data, resource.properties_data_encrypted AS resource_properties_data_encrypted, resource.engine_id AS resource_engine_id, resource.atomic_key AS resource_atomic_key, resource.needed_by AS resource_needed_by, resource.requires AS resource_requires, resource.replaces AS resource_replaces, resource.replaced_by AS resource_replaced_by, resource.current_template_id AS resource_current_template_id, resource_data_1.created_at AS resource_data_1_created_at, resource_data_1.updated_at AS resource_data_1_updated_at, resource_data_1.id AS resource_data_1_id, resource_data_1.`key` AS resource_data_1_key, resource_data_1.value AS resource_data_1_value, resource_data_1.redact AS resource_data_1_redact, resource_data_1.decrypt_method AS resource_data_1_decrypt_method, resource_data_1.resource_id AS resource_data_1_resource_id \nFROM resource LEFT OUTER JOIN resource_data AS resource_data_1 ON resource.id = resource_data_1.resource_id \nWHERE resource.stack_id = %s'] [parameters: (u'662faabb-5a9e-41cd-85bf-397d80580e39',)] MariaDB [heat]> describe resource; +---------------------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +---------------------------+--------------+------+-----+---------+----------------+ | nova_instance | varchar(255) | YES | | NULL | | | name | varchar(255) | YES | | NULL | | | created_at | datetime | YES | | NULL | | | updated_at | datetime | YES | | NULL | | | status | varchar(255) | YES | | NULL | | | status_reason | text | YES | | NULL | | | stack_id | varchar(36) | NO | MUL | NULL | | | rsrc_metadata | longtext | YES | | NULL | | | action | varchar(255) | YES | | NULL | | | properties_data | longtext | YES | | NULL | | | uuid | varchar(36) | YES | UNI | NULL | | | id | int(11) | NO | PRI | NULL | auto_increment | | engine_id | varchar(36) | YES | | NULL | | | atomic_key | int(11) | YES | | NULL | | | needed_by | longtext | YES | | NULL | | | requires | longtext | YES | | NULL | | | replaces | int(11) | YES | | NULL | | | replaced_by | int(11) | YES | | NULL | | | current_template_id | int(11) | YES | MUL | NULL | | | properties_data_encrypted | tinyint(1) | YES | | NULL | | +---------------------------+--------------+------+-----+---------+----------------+ 20 rows in set (0.00 sec) Expected results: MariaDB [heat]> describe resource; +---------------------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +---------------------------+--------------+------+-----+---------+----------------+ | nova_instance | varchar(255) | YES | | NULL | | | name | varchar(255) | YES | | NULL | | | created_at | datetime | YES | | NULL | | | updated_at | datetime | YES | | NULL | | | status | varchar(255) | YES | | NULL | | | status_reason | text | YES | | NULL | | | stack_id | varchar(36) | NO | MUL | NULL | | | rsrc_metadata | longtext | YES | | NULL | | | action | varchar(255) | YES | | NULL | | | properties_data | longtext | YES | | NULL | | | uuid | varchar(36) | YES | UNI | NULL | | | id | int(11) | NO | PRI | NULL | auto_increment | | engine_id | varchar(36) | YES | | NULL | | | atomic_key | int(11) | YES | | NULL | | | needed_by | longtext | YES | | NULL | | | requires | longtext | YES | | NULL | | | replaces | int(11) | YES | | NULL | | | replaced_by | int(11) | YES | | NULL | | | current_template_id | int(11) | YES | MUL | NULL | | | properties_data_encrypted | tinyint(1) | YES | | NULL | | | root_stack_id | varchar(36) | YES | MUL | NULL | | +---------------------------+--------------+------+-----+---------+----------------+ 21 rows in set (0.00 sec) Additional info: