This is a timing issue with the test.
I am able to make it work if I make it wait a little between netcat listen and netcat client connect.
Will need to tweak it a bit to make it more reliable.
Please go through the Blocker Process if a new fix needs to be included in 16.1.2 blocker respin. All patches need to be landed for blocker respin on Sept 29th and blocker + flag is required.
Description of problem: Our unified job detect failure in neutron_tempest_plugin.scenario.test_port_forwardings.PortForwardingTestJSON.test_port_forwarding_to_2_servers the error: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py", line 123, in test_port_forwarding_to_2_servers self._test_udp_port_forwarding(servers) File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py", line 108, in _test_udp_port_forwarding server['id']))) File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/common/utils.py", line 77, in wait_until_true while not predicate(): File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py", line 105, in <lambda> lambda: _message_received(server, ssh_client, expected_msg), File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py", line 94, in _message_received constants.PROTO_NAME_UDP) File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/base.py", line 489, in nc_client self.assertEqual(0, result.exit_status) File "/usr/lib/python3.6/site-packages/testtools/testcase.py", line 411, in assertEqual self.assertThat(observed, matcher, message) File "/usr/lib/python3.6/site-packages/testtools/testcase.py", line 498, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: 0 != 1 Ncat: Version 7.70 ( https://nmap.org/ncat ) execute /usr/lib/python3.6/site-packages/neutron_tempest_plugin/common/shell.py:77 2020-06-30 04:20:34.819 673245 DEBUG neutron_tempest_plugin.common.shell [-] Executing command 'echo "knock knock" | nc -z -w 1 -u 10.0.0.212 1036' on local host (timeout=None)... execute_local_command /usr/lib/python3.6/site-packages/neutron_tempest_plugin/common/shell.py:127 testtools.matchers._impl.MismatchError: 0 != 1 > /usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py(124)test_port_forwarding_to_2_servers() -> self._test_udp_port_forwarding(servers) (Pdb) self._test_udp_port_forwarding(servers) 2020-06-30 04:20:40.102 673245 INFO tempest.lib.common.ssh [-] Creating ssh connection to '10.0.0.212:1026' as 'cirros' with public key authentication 2020-06-30 04:20:40.106 673245 INFO paramiko.transport [-] Connected (version 2.0, client dropbear_2015.67) 2020-06-30 04:20:40.176 673245 INFO paramiko.transport [-] Authentication (publickey) successful! 2020-06-30 04:20:40.177 673245 INFO tempest.lib.common.ssh [-] ssh connection to cirros.0.212 successfully created 2020-06-30 04:20:40.320 673245 DEBUG neutron_tempest_plugin.common.shell [-] Executing command 'ncat --version 2>&1' on local host (timeout=None)... execute_local_command /usr/lib/python3.6/site-packages/neutron_tempest_plugin/common/shell.py:127 2020-06-30 04:20:40.334 673245 DEBUG neutron_tempest_plugin.common.shell [-] Command 'ncat --version 2>&1' succeeded: stderr: stdout: Ncat: Version 7.70 ( https://nmap.org/ncat ) execute /usr/lib/python3.6/site-packages/neutron_tempest_plugin/common/shell.py:77 2020-06-30 04:20:40.335 673245 DEBUG neutron_tempest_plugin.common.shell [-] Executing command 'echo "knock knock" | nc -z -w 1 -u 10.0.0.212 1036' on local host (timeout=None)... execute_local_command /usr/lib/python3.6/site-packages/neutron_tempest_plugin/common/shell.py:127 *** testtools.matchers._impl.MismatchError: 0 != 1 (Pdb) n --Return-- > /usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py(124)test_port_forwarding_to_2_servers()->None -> self._test_udp_port_forwarding(servers) (Pdb) n testtools.matchers._impl.MismatchError: 0 != 1 > /usr/lib/python3.6/site-packages/testtools/testcase.py(719)_run_test_method() -> return self._get_test_method()() Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: