Bug 1918616 - test_security_groups.NetworkSecGroupTest.test_multiple_ports_secgroup_inheritance is failing
Summary: test_security_groups.NetworkSecGroupTest.test_multiple_ports_secgroup_inherit...
Keywords:
Status: CLOSED DUPLICATE of bug 1908391
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-networking-ovn
Version: 16.2 (Train)
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Jakub Libosvar
QA Contact: Eran Kuris
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-01-21 08:54 UTC by Eran Kuris
Modified: 2021-03-15 10:15 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-03-15 10:15:12 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Eran Kuris 2021-01-21 08:54:44 UTC
Description of problem:
the test is failing with ssh error connetion:
72be19e41f04"],"description":"","binding:vnic_type":"normal","qos_policy_id":null,"port_security_enabled":true,"dns_name":"tempest-server-test-787355251","dns_assignment":[{"ip_address":"10.100.0.5","hostname":"tempest-server-test-787355251","fqdn":"tempest-server-test-787355251.openstackgate.local."}],"ip_allocation":"immediate","tags":[],"created_at":"2021-01-20T13:29:33Z","updated_at":"2021-01-20T13:29:44Z","revision_number":4,"project_id":"e9bae233c2b94a4da40766ea47a987ab"}]}'
2021-01-20 13:29:53,611 235495 INFO     [tempest.lib.common.rest_client] Request (NetworkSecGroupTest:test_multiple_ports_secgroup_inheritance): 201 POST http://10.0.0.105:9696/v2.0/floatingips 2.872s
2021-01-20 13:29:53,612 235495 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: {"floatingip": {"floating_network_id": "13882979-51cb-4151-9349-0373e27d3d59", "port_id": "95227b2c-c508-4796-ae3c-01d2d37ea7bd"}}
    Response - Headers: {'content-type': 'application/json', 'content-length': '604', 'x-openstack-request-id': 'req-c5ea72e3-8c21-4a35-aa3a-0542dae77e8d', 'date': 'Wed, 20 Jan 2021 13:29:53 GMT', 'connection': 'close', 'status': '201', 'content-location': 'http://10.0.0.105:9696/v2.0/floatingips'}
        Body: b'{"floatingip": {"id": "a320b08f-2ffd-460e-b03b-527a9efc8f51", "tenant_id": "e9bae233c2b94a4da40766ea47a987ab", "floating_ip_address": "10.0.0.247", "floating_network_id": "13882979-51cb-4151-9349-0373e27d3d59", "router_id": "e99fcd6f-7d7f-4ba3-bb12-4e1f4f85dc7f", "port_id": "95227b2c-c508-4796-ae3c-01d2d37ea7bd", "fixed_ip_address": "10.100.0.5", "status": "DOWN", "project_id": "e9bae233c2b94a4da40766ea47a987ab", "description": "", "dns_domain": "", "dns_name": "", "port_forwardings": [], "tags": [], "created_at": "2021-01-20T13:29:52Z", "updated_at": "2021-01-20T13:29:52Z", "revision_number": 0}}'
2021-01-20 13:29:53,619 235495 DEBUG    [neutron_tempest_plugin.scenario.base] NetworkSecGroupTest:test_multiple_ports_secgroup_inheritance begins to ping 10.0.0.225 in 120 sec and the expected result is reachable
2021-01-20 13:30:13,767 235495 DEBUG    [tempest.lib.common.utils.test_utils] Call ping returns true in 20.148275 seconds
2021-01-20 13:30:13,768 235495 DEBUG    [neutron_tempest_plugin.scenario.base] NetworkSecGroupTest:test_multiple_ports_secgroup_inheritance finishes ping 10.0.0.225 in 120 sec and the ping result is expected
2021-01-20 13:30:13,775 235495 INFO     [tempest.lib.common.ssh] Creating ssh connection to '10.0.0.225:22' as 'cloud-user' with public key authentication
2021-01-20 13:30:13,779 235495 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 ([Errno None] Unable to connect to port 22 on 10.0.0.225). Number attempts: 1. Retry after 2 seconds.
2021-01-20 13:30:16,283 235495 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 ([Errno None] Unable to connect to port 22 on 10.0.0.225). Number attempts: 2. Retry after 3 seconds.
2021-01-20 13:30:19,789 235495 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 ([Errno None] Unable to connect to port 22 on 10.0.0.225). Number attempts: 3. Retry after 4 seconds.
2021-01-20 13:30:24,296 235495 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 ([Errno None] Unable to connect to port 22 on 10.0.0.225). Number attempts: 4. Retry after 5 seconds.
2021-01-20 13:30:29,800 235495 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 ([Errno None] Unable to connect to port 22 on 10.0.0.225). Number attempts: 5. Retry after 6 seconds.
2021-01-20 13:30:36,302 235495 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 ([Errno None] Unable to connect to port 22 on 10.0.0.225). Number attempts: 6. Retry after 7 seconds.
2021-01-20 13:30:43,810 235495 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 ([Errno None] Unable to connect to port 22 on 10.0.0.225). Number attempts: 7. Retry after 8 seconds.
2021-01-20 13:30:52,320 235495 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 ([Errno None] Unable to connect to port 22 on 10.0.0.225). Number attempts: 8. Retry after 9 seconds.
2021-01-20 13:31:01,830 235495 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 ([Errno None] Unable to connect to port 22 on 10.0.0.225). Number attempts: 9. Retry after 10 seconds.
2021-01-20 13:31:12,335 235495 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 ([Errno None] Unable to connect to port 22 on 10.0.0.225). Number attempts: 10. Retry after 11 seconds.
2021-01-20 13:31:23,848 235495 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 ([Errno None] Unable to connect to port 22 on 10.0.0.225). Number attempts: 11. Retry after 12 seconds.
2021-01-20 13:31:36,364 235495 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 ([Errno None] Unable to connect to port 22 on 10.0.0.225). Number attempts: 12. Retry after 13 seconds.
2021-01-20 13:31:49,882 235495 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 ([Errno None] Unable to connect to port 22 on 10.0.0.225). Number attempts: 13. Retry after 14 seconds.
2021-01-20 13:32:04,400 235495 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 ([Errno None] Unable to connect to port 22 on 10.0.0.225). Number attempts: 14. Retry after 15 seconds.
2021-01-20 13:32:19,918 235495 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 ([Errno None] Unable to connect to port 22 on 10.0.0.225). Number attempts: 15. Retry after 16 seconds.
2021-01-20 13:32:36,439 235495 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 ([Errno None] Unable to connect to port 22 on 10.0.0.225). Number attempts: 16. Retry after 17 seconds.
2021-01-20 13:32:53,959 235495 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 ([Errno None] Unable to connect to port 22 on 10.0.0.225). Number attempts: 17. Retry after 18 seconds.
2021-01-20 13:33:12,464 235495 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 ([Errno None] Unable to connect to port 22 on 10.0.0.225). Number attempts: 18. Retry after 19 seconds.
2021-01-20 13:33:31,969 235495 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 ([Errno None] Unable to connect to port 22 on 10.0.0.225). Number attempts: 19. Retry after 20 seconds.
2021-01-20 13:33:52,491 235495 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 ([Errno None] Unable to connect to port 22 on 10.0.0.225). Number attempts: 20. Retry after 21 seconds.
2021-01-20 13:34:14,012 235495 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 ([Errno None] Unable to connect to port 22 on 10.0.0.225). Number attempts: 21. Retry after 22 seconds.
2021-01-20 13:34:36,539 235495 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 ([Errno None] Unable to connect to port 22 on 10.0.0.225). Number attempts: 22. Retry after 23 seconds.
2021-01-20 13:35:00,067 235495 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 ([Errno None] Unable to connect to port 22 on 10.0.0.225). Number attempts: 23. Retry after 24 seconds.
2021-01-20 13:35:24,611 235495 INFO     [paramiko.transport] Connected (version 2.0, client OpenSSH_8.0)
2021-01-20 13:35:24,720 235495 INFO     [paramiko.transport] Authentication (publickey) failed.
2021-01-20 13:35:24,725 235495 ERROR    [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cloud-user.0.225 after 23 attempts. Proxy client: no proxy client
2021-01-20 13:35:24.725 235495 ERROR tempest.lib.common.ssh Traceback (most recent call last):
2021-01-20 13:35:24.725 235495 ERROR tempest.lib.common.ssh   File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 112, in _get_ssh_connection
2021-01-20 13:35:24.725 235495 ERROR tempest.lib.common.ssh     sock=proxy_chan)
2021-01-20 13:35:24.725 235495 ERROR tempest.lib.common.ssh   File "/usr/local/lib/python3.6/site-packages/paramiko/client.py", line 446, in connect
2021-01-20 13:35:24.725 235495 ERROR tempest.lib.common.ssh     passphrase,
2021-01-20 13:35:24.725 235495 ERROR tempest.lib.common.ssh   File "/usr/local/lib/python3.6/site-packages/paramiko/client.py", line 691, in _auth
2021-01-20 13:35:24.725 235495 ERROR tempest.lib.common.ssh     self._agent = Agent()
2021-01-20 13:35:24.725 235495 ERROR tempest.lib.common.ssh   File "/usr/local/lib/python3.6/site-packages/paramiko/agent.py", line 379, in __init__
2021-01-20 13:35:24.725 235495 ERROR tempest.lib.common.ssh     self._connect(conn)
2021-01-20 13:35:24.725 235495 ERROR tempest.lib.common.ssh   File "/usr/local/lib/python3.6/site-packages/paramiko/agent.py", line 65, in _connect
2021-01-20 13:35:24.725 235495 ERROR tempest.lib.common.ssh     ptype, result = self._send_message(cSSH2_AGENTC_REQUEST_IDENTITIES)
2021-01-20 13:35:24.725 235495 ERROR tempest.lib.common.ssh   File "/usr/local/lib/python3.6/site-packages/paramiko/agent.py", line 83, in _send_message
2021-01-20 13:35:24.725 235495 ERROR tempest.lib.common.ssh     data = self._read_all(4)
2021-01-20 13:35:24.725 235495 ERROR tempest.lib.common.ssh   File "/usr/local/lib/python3.6/site-packages/paramiko/agent.py", line 88, in _read_all
2021-01-20 13:35:24.725 235495 ERROR tempest.lib.common.ssh     result = self._conn.recv(wanted)
2021-01-20 13:35:24.725 235495 ERROR tempest.lib.common.ssh ConnectionResetError: [Errno 104] Connection reset by peer
2021-01-20 13:35:24.725 235495 ERROR tempest.lib.common.ssh 
2021-01-20 13:35:24,728 235495 DEBUG    [neutron_tempest_plugin.scenario.base] Connection to the 10.0.0.225 via SSH timed out.
User: cloud-user, Password: None

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. run automation regression ci run
2.
3.

Actual results:

https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/network/view/networking-ovn/job/DFG-network-networking-ovn-16.2_director-rhel-virthost-3cont_2comp-ipv4-vlan/36/testReport/neutron_tempest_plugin.scenario.test_security_groups/NetworkSecGroupTest/test_multiple_ports_secgroup_inheritance_id_f07d0159_8f9e_4faa_87f5_a869ab0ad488_/
Expected results:


Additional info:

Comment 1 Jakub Libosvar 2021-01-26 16:29:03 UTC
The reason is that the VM was not able to reach the metadata API and thus the SSH public key authentication failed:

[  114.745985] cloud-init[873]: 2021-01-20 13:31:48,837 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [50/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f0dd7c36278>, 'Connection to 169.254.169.254 timed out. (connect timeout=50.0)'))]

I checked the metadata logs and seems like everything was set up correctly, however the queries are not observable in the metadat agent and not in haproxy - here I'm not sure if I'm checking the right place but the var/log/messages doesn't see any queries in haproxy from the time the instance was booting.

Comment 2 Elvira 2021-03-15 10:15:12 UTC
Closing this as it is a duplicate of BZ[0], fixed by OVN patch[1]. This was related to quickly deleting and provisioning back the metadata datapath.

[0] https://bugzilla.redhat.com/show_bug.cgi?id=1908391
[1] http://patchwork.ozlabs.org/project/ovn/patch/1608197000-637-1-git-send-email-dceara@redhat.com/

*** This bug has been marked as a duplicate of bug 1908391 ***


Note You need to log in before you can comment on or make changes to this bug.