For the fast forward upgrades project, we need to have a minimum of compatibility when using an undercloud on N+3 and overcloud on N. The basic operations we need is removing a node and adding new ones. When removing a node, I am hitting this error now: 2018-01-17 08:01:36.550 19877 DEBUG heat.engine.parameter_groups [req-45bad641-e0d3-4612-8365-b7c24b9f59a9 - admin - default default] Validating Parameter Groups: OS::project_id, ControllerCount, ServerMetadata, PublicVirtualFixedIPs, StorageMgmtVirtualFixedIPs, InternalApiVirtualFixedIPs, BlockStorageRemovalPolicies, CloudNameInternal, CephStorageSchedulerHints, ControllerHostnameFormat, CloudDomain, ControllerRemovalPolicies, ComputeCount, ControllerSchedulerHints, NeutronControlPlaneID, ObjectStorageRemovalPolicies, CephStorageServices, ComputeHostnameFormat, ComputeRemovalPolicies, RedisVirtualFixedIPs, ComputeServices, HypervisorNeutronPublicInterface, ObjectStorageCount, BlockStorageServices, CloudName, CloudNameCtlplane, CephStorageRemovalPolicies, RabbitCookieSalt, CloudNameStorageManagement, DeployIdentifier, NeutronPublicInterface, BlockStorageCount, BlockStorageSchedulerHints, ControlFixedIPs, StorageVirtualFixedIPs, CephStorageCount, BlockStorageHostnameFormat, ObjectStorageSchedulerHints, ObjectStorageHostnameFormat, OS::stack_id, ObjectStorageServices, ControllerServices, NovaComputeSchedulerHints, CephStorageHostnameFormat, HypervisorNeutronPhysicalBridge, CloudNameStorage, UpdateIdentifier validate /usr/lib/python2.7/site-packages/heat/engine/parameter_groups.py:42 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters [req-45bad641-e0d3-4612-8365-b7c24b9f59a9 - admin - default default] DBAPIError exception wrapped from (pymysql.err.IntegrityError) (1048, u"Column 'resource_id' cannot be null") [SQL: u'INSERT INTO resource_data (created_at, updated_at, `key`, value, redact, decrypt_method, resource_id) VALUES (%(created_at)s, %(updated_at)s, %(key)s, %(value)s, %(redact)s, %(decrypt_method)s, %(resource_id)s)'] [parameters: {'resource_id': None, 'created_at': datetime.datetime(2018, 1, 17, 8, 1, 38, 426685), 'updated_at': None, 'value': u'1', 'redact': 0, 'decrypt_method': '', 'key': 'name_blacklist'}]: IntegrityError: (1048, u"Column 'resource_id' cannot be null") 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last): 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters context) 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters) 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters result = self._query(query) 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters conn.query(q) 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters result.read() 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters first_packet = self.connection._read_packet() 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1014, in _read_packet 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters packet.check_error() 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters err.raise_mysql_exception(self._data) 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters raise errorclass(errno, errval) 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters IntegrityError: (1048, u"Column 'resource_id' cannot be null") 2018-01-17 08:01:38.427 19877 ERROR oslo_db.sqlalchemy.exc_filters 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server [req-45bad641-e0d3-4612-8365-b7c24b9f59a9 - admin - default default] Exception during message handling: DBError: (pymysql.err.IntegrityError) (1048, u"Column 'resource_id' cannot be null") [SQL: u'INSERT INTO resource_data (created_at, updated_at, `key`, value, redact, decrypt_method, resource_id) VALUES (%(created_at)s, %(updated_at)s, %(key)s, %(value)s, %(redact)s, %(decrypt_method)s, %(resource_id)s)'] [parameters: {'resource_id': None, 'created_at': datetime.datetime(2018, 1, 17, 8, 1, 38, 426685), 'updated_at': None, 'value': u'1', 'redact': 0, 'decrypt_method': '', 'key': 'name_blacklist'}] 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 157, in wrapper 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 409, in wrapped 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server return func(self, ctx, *args, **kwargs) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 1287, in validate_template 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server stack.preview_resources() 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1052, in preview_resources 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server for resource in six.itervalues(self.resources)] 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 190, in preview 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server child_template = self.child_template() 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 704, in child_template 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server names = self._resource_names() 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 367, in _resource_names 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server name_blacklist = self._name_blacklist(update_rsrc_data) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 363, in _name_blacklist 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server self.data_set('name_blacklist', ','.join(rsrc_names)) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 2584, in data_set 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server resource_data_objects.ResourceData.set(self, key, value, redact) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/heat/objects/resource_data.py", line 75, in set 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server **kwargs 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/heat/db/sqlalchemy/api.py", line 418, in resource_data_set 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server current.save(session=context.session) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/models.py", line 50, in save 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server session.flush() 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2171, in flush 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server self._flush(objects) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2291, in _flush 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server transaction.rollback(_capture_exception=True) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server compat.reraise(exc_type, exc_value, exc_tb) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2255, in _flush 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server flush_context.execute() 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server rec.execute(self) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server uow 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 181, in save_obj 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server mapper, table, insert) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 835, in _emit_insert_statements 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server execute(statement, params) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 945, in execute 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server return meth(self, multiparams, params) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server return connection._execute_clauseelement(self, multiparams, params) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server compiled_sql, distilled_params 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server context) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1398, in _handle_dbapi_exception 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server util.raise_from_cause(newraise, exc_info) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server reraise(type(exception), exception, tb=exc_tb, cause=cause) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server context) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server cursor.execute(statement, parameters) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server result = self._query(query) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server conn.query(q) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server result.read() 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server first_packet = self.connection._read_packet() 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1014, in _read_packet 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server packet.check_error() 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server err.raise_mysql_exception(self._data) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server raise errorclass(errno, errval) 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server DBError: (pymysql.err.IntegrityError) (1048, u"Column 'resource_id' cannot be null") [SQL: u'INSERT INTO resource_data (created_at, updated_at, `key`, value, redact, decrypt_method, resource_id) VALUES (%(created_at)s, %(updated_at)s, %(key)s, %(value)s, %(redact)s, %(decrypt_method)s, %(resource_id)s)'] [parameters: {'resource_id': None, 'created_at': datetime.datetime(2018, 1, 17, 8, 1, 38, 426685), 'updated_at': None, 'value': u'1', 'redact': 0, 'decrypt_method': '', 'key': 'name_blacklist'}] 2018-01-17 08:01:38.430 19877 ERROR oslo_messaging.rpc.server I double checked the code, and I could locate where is it happening: http://git.openstack.org/cgit/openstack/heat/tree/heat/engine/resources/openstack/heat/resource_group.py#n362 I commented lines 362 and 363, and the node was successfully deleted.
I can successfully remove nodes now
Undercloud upgraded to OSP13: (undercloud) [stack@undercloud-0 ~]$ openstack server list +--------------------------------------+--------------+--------+------------------------+----------------+------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+--------------+--------+------------------------+----------------+------------+ | 141981d8-b105-414b-b928-fa2345c49891 | ceph-2 | ACTIVE | ctlplane=192.168.24.11 | overcloud-full | ceph | | 41b70313-261c-4672-90aa-504d2df1edd6 | controller-1 | ACTIVE | ctlplane=192.168.24.12 | overcloud-full | controller | | 5f59f10a-145a-4c51-8f00-b787ec96de02 | controller-0 | ACTIVE | ctlplane=192.168.24.18 | overcloud-full | controller | | a750cc0e-d4ee-4a93-bf53-a35ba9d1334e | controller-2 | ACTIVE | ctlplane=192.168.24.20 | overcloud-full | controller | | b333b4a2-becf-4602-ba04-7322091d1c5a | ceph-0 | ACTIVE | ctlplane=192.168.24.14 | overcloud-full | ceph | | 301a8dfb-882c-452b-95a9-dc08edfb341f | ceph-1 | ACTIVE | ctlplane=192.168.24.10 | overcloud-full | ceph | | 6edc54f8-ae50-4704-9859-04c01ec3c3fb | compute-1 | ACTIVE | ctlplane=192.168.24.8 | overcloud-full | compute | | 75b70295-b768-43bd-b223-50c332d145b5 | compute-0 | ACTIVE | ctlplane=192.168.24.17 | overcloud-full | compute | +--------------------------------------+--------------+--------+------------------------+----------------+------------+ (undercloud) [stack@undercloud-0 ~]$ (undercloud) [stack@undercloud-0 ~]$ (undercloud) [stack@undercloud-0 ~]$ (undercloud) [stack@undercloud-0 ~]$ (undercloud) [stack@undercloud-0 ~]$ openstack overcloud node delete --templates /home/stack/openstack-tripleo-heat-templates-newton 75b70295-b768-43bd-b223-50c332d145b5 Deleting the following nodes from stack overcloud: - 75b70295-b768-43bd-b223-50c332d145b5 Started Mistral Workflow tripleo.scale.v1.delete_node. Execution ID: c2d068b3-70e9-4b86-811b-f74c398b19b3 Waiting for messages on queue 'tripleo' with no timeout. (undercloud) [stack@undercloud-0 ~]$ (undercloud) [stack@undercloud-0 ~]$ openstack server list +--------------------------------------+--------------+--------+------------------------+----------------+------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+--------------+--------+------------------------+----------------+------------+ | 141981d8-b105-414b-b928-fa2345c49891 | ceph-2 | ACTIVE | ctlplane=192.168.24.11 | overcloud-full | ceph | | 41b70313-261c-4672-90aa-504d2df1edd6 | controller-1 | ACTIVE | ctlplane=192.168.24.12 | overcloud-full | controller | | 5f59f10a-145a-4c51-8f00-b787ec96de02 | controller-0 | ACTIVE | ctlplane=192.168.24.18 | overcloud-full | controller | | a750cc0e-d4ee-4a93-bf53-a35ba9d1334e | controller-2 | ACTIVE | ctlplane=192.168.24.20 | overcloud-full | controller | | b333b4a2-becf-4602-ba04-7322091d1c5a | ceph-0 | ACTIVE | ctlplane=192.168.24.14 | overcloud-full | ceph | | 301a8dfb-882c-452b-95a9-dc08edfb341f | ceph-1 | ACTIVE | ctlplane=192.168.24.10 | overcloud-full | ceph | | 6edc54f8-ae50-4704-9859-04c01ec3c3fb | compute-1 | ACTIVE | ctlplane=192.168.24.8 | overcloud-full | compute | +--------------------------------------+--------------+--------+------------------------+----------------+------------+
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2086