Bug 2181163

Summary: Ci failed the test_cross_tenant_traffic test
Product: Red Hat OpenStack Reporter: Attila Fazekas <afazekas>
Component: openstack-neutronAssignee: Miro Tomaska <mtomaska>
Status: NEW --- QA Contact: Eran Kuris <ekuris>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 16.2 (Train)CC: chrisw, mlavalle, scohen
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Attila Fazekas 2023-03-23 09:07:51 UTC 
Initializing random number generator... [    2.174826] random: dd urandom read with 13 bits of entropy available
done.
Starting acpid: OK
Starting network...
udhcpc (v1.23.2) started
Sending discover...
Sending select for 10.100.0.6...
Lease of 10.100.0.6 obtained, lease time 43200
route: SIOCADDRT: File exists
WARN: failed: route add -net "0.0.0.0/0" gw "10.100.0.1"
checking http://169.254.169.254/2009-04-04/instance-id
failed 1/20: up 2.25. request failed
failed 2/20: up 14.26. request failed
failed 3/20: up 26.28. request failed
failed 4/20: up 38.29. request failed
failed 5/20: up 50.31. request failed
failed 6/20: up 62.33. request failed
failed 7/20: up 74.35. request failed
failed 8/20: up 86.36. request failed
failed 9/20: up 98.38. request failed
failed 10/20: up 110.40. request failed
failed 11/20: up 122.42. request failed
failed 12/20: up 134.44. request failed
failed 13/20: up 146.45. request failed
failed 14/20: up 158.47. request failed
failed 15/20: up 170.49. request failed
failed 16/20: up 182.51. request failed
[  201.365826] random: nonblocking pool is initialized
failed 17/20: up 194.53. request failed
failed 18/20: up 206.55. request failed
failed 19/20: up 218.56. request failed
failed 20/20: up 230.58. request failed
failed to read iid from metadata. tried 20
failed to get instance-id of datasource
Top of dropbear init script
Starting dropbear sshd: failed to get instance-id of datasource


Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 113, in _get_ssh_connection
    sock=proxy_chan)
  File "/usr/lib/python3.6/site-packages/paramiko/client.py", line 437, in connect
    passphrase,
  File "/usr/lib/python3.6/site-packages/paramiko/client.py", line 676, in _auth
    self._agent = Agent()
  File "/usr/lib/python3.6/site-packages/paramiko/agent.py", line 379, in __init__
    self._connect(conn)
  File "/usr/lib/python3.6/site-packages/paramiko/agent.py", line 65, in _connect
    ptype, result = self._send_message(cSSH2_AGENTC_REQUEST_IDENTITIES)
  File "/usr/lib/python3.6/site-packages/paramiko/agent.py", line 83, in _send_message
    l = self._read_all(4)
  File "/usr/lib/python3.6/site-packages/paramiko/agent.py", line 88, in _read_all
    result = self._conn.recv(wanted)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/common/utils/__init__.py", line 90, in wrapper
    return f(*func_args, **func_kwargs)
  File "/usr/lib/python3.6/site-packages/tempest/scenario/test_security_groups_basic_ops.py", line 492, in test_cross_tenant_traffic
    self._verify_mac_addr(self.alt_tenant)
  File "/usr/lib/python3.6/site-packages/tempest/scenario/test_security_groups_basic_ops.py", line 447, in _verify_mac_addr
    access_point_ssh = self._connect_to_access_point(tenant)
  File "/usr/lib/python3.6/site-packages/tempest/scenario/test_security_groups_basic_ops.py", line 372, in _connect_to_access_point
    server=tenant.access_point)
  File "/usr/lib/python3.6/site-packages/tempest/scenario/manager.py", line 638, in get_remote_client
    linux_client.validate_authentication()
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 59, in wrapper
    six.reraise(*original_exception)
  File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 32, in wrapper
    return function(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 115, in validate_authentication
    self.ssh_client.test_connection_auth()
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 217, in test_connection_auth
    connection = self._get_ssh_connection()
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 129, in _get_ssh_connection
    password=self.password)
tempest.lib.exceptions.SSHTimeout: Connection to the 10.0.0.186 via SSH timed out.
User: cirros, Password: None

Reproducibility: Unknown
Comment 2 Attila Fazekas 2023-03-23 14:45:21 UTC 
Reproducibility: less than 20% in a regular job.
Comment 4 Miro Tomaska 2023-04-11 17:25:55 UTC 
Hi Attila,

Where is this log coming from?  Is that from the VM? I was not able to find it. Thanks!


Starting acpid: OK
Starting network...
udhcpc (v1.23.2) started
Sending discover...
Sending select for 10.100.0.6...
Lease of 10.100.0.6 obtained, lease time 43200
route: SIOCADDRT: File exists
WARN: failed: route add -net "0.0.0.0/0" gw "10.100.0.1"
checking http://169.254.169.254/2009-04-04/instance-id
failed 1/20: up 2.25. request failed
failed 2/20: up 14.26. request failed
failed 3/20: up 26.28. request failed
failed 4/20: up 38.29. request failed
failed 5/20: up 50.31. request failed
failed 6/20: up 62.33. request failed
failed 7/20: up 74.35. request failed
failed 8/20: up 86.36. request failed
failed 9/20: up 98.38. request failed
failed 10/20: up 110.40. request failed
failed 11/20: up 122.42. request failed
failed 12/20: up 134.44. request failed
failed 13/20: up 146.45. request failed
failed 14/20: up 158.47. request failed
failed 15/20: up 170.49. request failed
failed 16/20: up 182.51. request failed
[  201.365826] random: nonblocking pool is initialized
failed 17/20: up 194.53. request failed
failed 18/20: up 206.55. request failed
failed 19/20: up 218.56. request failed
failed 20/20: up 230.58. request failed
failed to read iid from metadata. tried 20
failed to get instance-id of datasource
Top of dropbear init script
Starting dropbear sshd: failed to get instance-id of datasource