Description of problem:
- This is an upgrade from Ocata to Queens. 
- DB migration fails to go beyond 868cb606a74a because the ports.version is NULL on all ports. Also, physical_network is NULL, but it should be using the "baremetal" network.
- They might have hit this bug [1] during the update process.
- [2] shows all the ironic tables with a summary of their version colums; conductor_hardware_interfaces and ports have NULL strings in their version columns.
- From the debug logs [3], the conductor_hardware_interface table is not submitted to this validation.
- A quick solution would probably be to set the port version to 1.7 and physical_network to "baremetal" but we chose to validate with engineering to make sure nothing breaks.


[1] https://bugs.launchpad.net/ironic/+bug/1715190
[2]
~~~
$ for t in $(mysql -h 127.0.0.1 -N -s -D ironic -e "show tables;" | grep -v alembic); do echo $t;mysql -h 127.0.0.1 -D ironic -e "select version,count(*) from $t group by version;";done
chassis
conductor_hardware_interfaces
+---------+----------+
| version | count(*) |
+---------+----------+
| NULL    |       90 |
+---------+----------+
conductors
+---------+----------+
| version | count(*) |
+---------+----------+
| 1.2     |        3 |
+---------+----------+
node_tags
nodes
+---------+----------+
| version | count(*) |
+---------+----------+
| 1.21    |        8 |
+---------+----------+
portgroups
ports
+---------+----------+
| version | count(*) |
+---------+----------+
| NULL    |        8 |
+---------+----------+
volume_connectors
volume_targets
~~~

[3]
~~~

[root@overcloud-controller-1 ironic]#  ironic-dbsync -d upgrade 2>&1 | less
Option "rpc_backend" from group "DEFAULT" is deprecated for removal (Replaced by [DEFAULT]/transport_url).  Its value may be silently ignored in the future.
2018-09-02 17:59:20.469 417673 DEBUG oslo_db.api [-] Loading backend 'sqlalchemy' from 'ironic.db.sqlalchemy.api' _load_backend /usr/lib/python2.7/site-packages/oslo_db/api.py:234
2018-09-02 17:59:20.744 417673 DEBUG oslo_db.sqlalchemy.engines [-] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:290
2018-09-02 17:59:20.748 417673 INFO alembic.runtime.migration [-] Context impl MySQLImpl.
2018-09-02 17:59:20.748 417673 INFO alembic.runtime.migration [-] Will assume non-transactional DDL.
2018-09-02 17:59:20.753 417673 INFO ironic.db.sqlalchemy.api [-] DVD checking Chassis Model Version Chassis.version in {'Node': set(['1.23', '1.22', '1.21']), 'Conductor': set(['1.2']), 'Trait': set(['1.0']), 'VolumeTarget': set(['1.0']), 'Port': set(['1.6', '1.7']), 'VolumeConnector': set(['1.0']), 'Chassis': set(['1.3']), 'Portgroup': set(['1.3']), 'TraitList': set(['1.0'])}
2018-09-02 17:59:20.759 417673 INFO ironic.db.sqlalchemy.api [-] DVD query out: SELECT chassis.version AS chassis_version
FROM chassis
WHERE chassis.version IS NULL OR chassis.version NOT IN (%(version_1)s) COUNT 0
2018-09-02 17:59:20.763 417673 INFO ironic.db.sqlalchemy.api [-] DVD checking Conductor Model Version Conductor.version in {'Node': set(['1.23', '1.22', '1.21']), 'Conductor': set(['1.2']), 'Trait': set(['1.0']), 'VolumeTarget': set(['1.0']), 'Port': set(['1.6', '1.7']), 'VolumeConnector': set(['1.0']), 'Chassis': set(['1.3']), 'Portgroup': set(['1.3']), 'TraitList': set(['1.0'])}
2018-09-02 17:59:20.767 417673 INFO ironic.db.sqlalchemy.api [-] DVD checking ConductorHardwareInterfaces Model Version ConductorHardwareInterfaces.version in {'Node': set(['1.23', '1.22', '1.21']), 'Conductor': set(['1.2']), 'Trait': set(['1.0']), 'VolumeTarget': set(['1.0']), 'Port': set(['1.6', '1.7']), 'VolumeConnector': set(['1.0']), 'Chassis': set(['1.3']), 'Portgroup': set(['1.3']), 'TraitList': set(['1.0'])}
2018-09-02 17:59:20.768 417673 INFO ironic.db.sqlalchemy.api [-] DVD checking Node Model Version Node.version in {'Node': set(['1.23', '1.22', '1.21']), 'Conductor': set(['1.2']), 'Trait': set(['1.0']), 'VolumeTarget': set(['1.0']), 'Port': set(['1.6', '1.7']), 'VolumeConnector': set(['1.0']), 'Chassis': set(['1.3']), 'Portgroup': set(['1.3']), 'TraitList': set(['1.0'])}
2018-09-02 17:59:20.773 417673 INFO ironic.db.sqlalchemy.api [-] DVD query out: SELECT nodes.version AS nodes_version
FROM nodes
WHERE nodes.version IS NULL OR nodes.version NOT IN (%(version_1)s, %(version_2)s, %(version_3)s) COUNT 0
2018-09-02 17:59:20.777 417673 INFO ironic.db.sqlalchemy.api [-] DVD checking Port Model Version Port.version in {'Node': set(['1.23', '1.22', '1.21']), 'Conductor': set(['1.2']), 'Trait': set(['1.0']), 'VolumeTarget': set(['1.0']), 'Port': set(['1.6', '1.7']), 'VolumeConnector': set(['1.0']), 'Chassis': set(['1.3']), 'Portgroup': set(['1.3']), 'TraitList': set(['1.0'])}
2018-09-02 17:59:20.783 417673 INFO ironic.db.sqlalchemy.api [-] DVD query out: SELECT ports.version AS ports_version
FROM ports
WHERE ports.version IS NULL OR ports.version NOT IN (%(version_1)s, %(version_2)s) COUNT 8
The database is not compatible with this release of ironic (10.1.2). Please run "ironic-dbsync online_data_migrations" using the previous release.
~~~

Version-Release number of selected component (if applicable):
openstack-ironic-api-10.1.2-4.el7ost.noarch                 Mon Aug 27 09:57:30 2018
openstack-ironic-common-10.1.2-4.el7ost.noarch              Mon Aug 27 09:56:48 2018
openstack-ironic-conductor-10.1.2-4.el7ost.noarch           Mon Aug 27 09:57:30 2018
puppet-ironic-12.4.0-0.20180329034302.8285d85.el7ost.noarch Mon Aug 27 09:55:32 2018
python-ironic-inspector-client-3.1.1-1.el7ost.noarch        Mon Aug 27 09:56:22 2018
python-ironic-lib-2.12.1-1.el7ost.noarch                    Mon Aug 27 09:56:39 2018
python2-ironicclient-2.2.0-1.el7ost.noarch                  Mon Aug 27 09:56:23 2018


Actual result:
Because of the missing column, ironic-conductor container fails to start on the overcloud controllers

~~~
2018-09-01 03:23:40.837 1 ERROR oslo_service.service DBError: (pymysql.err.InternalError) (1054, u"Unknown column 'nodes.rescue_interface' in 'field list'") 
~~~