Description of problem:

Build RHOS_TRUNK-16.0-RHEL-8-20191022.n.2 .

tempest fails to ssh the nodes , for example
tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario failing.

According to the test vm console log, it was able to optain ip address and have
some metadata, but form undercloud it is not ssh- able.

No sign of tunneled traffic between the controller and compute.  


The issue maybe related some build /initialization time issue,
or version incompatibilities.

[heat-admin@controller-0 ~]$ sudo podman exec ovn-dbs-bundle-podman-0 ovn-nbctl show
2019-10-24T07:40:50Z|00001|ovsdb_idl|WARN|OVN_Northbound database lacks HA_Chassis table (database needs upgrade?)
2019-10-24T07:40:50Z|00002|ovsdb_idl|WARN|OVN_Northbound database lacks HA_Chassis_Group table (database needs upgrade?)
2019-10-24T07:40:50Z|00003|ovsdb_idl|WARN|Logical_Router table in OVN_Northbound database lacks policies column (database needs upgrade?)
2019-10-24T07:40:50Z|00004|ovsdb_idl|WARN|OVN_Northbound database lacks Logical_Router_Policy table (database needs upgrade?)
2019-10-24T07:40:50Z|00005|ovsdb_idl|WARN|Logical_Router_Port table in OVN_Northbound database lacks ha_chassis_group column (database needs upgrade?)
2019-10-24T07:40:50Z|00006|ovsdb_idl|WARN|Logical_Switch_Port table in OVN_Northbound database lacks ha_chassis_group column (database needs upgrade?)
2019-10-24T07:40:50Z|00007|ovsdb_idl|WARN|OVN_Northbound database lacks HA_Chassis table (database needs upgrade?)
2019-10-24T07:40:50Z|00008|ovsdb_idl|WARN|OVN_Northbound database lacks HA_Chassis_Group table (database needs upgrade?)
2019-10-24T07:40:50Z|00009|ovsdb_idl|WARN|Logical_Router table in OVN_Northbound database lacks policies column (database needs upgrade?)
2019-10-24T07:40:50Z|00010|ovsdb_idl|WARN|OVN_Northbound database lacks Logical_Router_Policy table (database needs upgrade?)
2019-10-24T07:40:50Z|00011|ovsdb_idl|WARN|Logical_Router_Port table in OVN_Northbound database lacks ha_chassis_group column (database needs upgrade?)
2019-10-24T07:40:50Z|00012|ovsdb_idl|WARN|Logical_Switch_Port table in OVN_Northbound database lacks ha_chassis_group column (database needs upgrade?)
switch ae04320d-cf2d-4519-81fe-2c0dc6f6d24e (neutron-a2667613-2943-403b-9f1f-023736d6c39a) (aka public)
    port provnet-a2667613-2943-403b-9f1f-023736d6c39a
        type: localnet
        addresses: ["unknown"]
    port d217d547-e3f4-4810-ae7b-68386b309b31
        type: localport
        addresses: ["fa:16:3e:97:a2:9d"]
[heat-admin@controller-0 ~]$ sudo podman exec ovn-dbs-bundle-podman-0 ovn-nbctl --version
ovn-nbctl (Open vSwitch) 2.11.1
DB Schema 5.16.0


neutron server log:
2019-10-23 15:28:31.622 26 ERROR ovsdbapp.backend.ovs_idl.transaction [req-c0776e3f-9761-4580-bec6-3db489bf9672 33a950414444426a97cb225a034949d4 449fa6f1fd1f4848a45b2c11414900b0 - default default] Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
    txn.results.put(txn.do_commit())
  File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 86, in do_commit
    command.run_idl(txn)
  File "/usr/lib/python3.6/site-packages/networking_ovn/ovsdb/commands.py", line 1042, in run_idl
    resource_id=self.name, resource_type=self.resource_type)
networking_ovn.common.exceptions.RevisionConflict: OVN revision number for ecdaa845-89b1-434c-8816-a1ff02b613f7 (type: ports) is equal or higher than the given resource. Skipping update