Bug 1738279
| Summary: | Octavia Tempest test_mixed_ipv4_ipv6_members_traffic test fails | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Carlos Goncalves <cgoncalves> |
| Component: | python-octavia-tests-tempest | Assignee: | Gregory Thiemonge <gthiemon> |
| Status: | CLOSED ERRATA | QA Contact: | Carlos Goncalves <cgoncalves> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 15.0 (Stein) | CC: | bbonguar, jschluet, lhh, njohnston |
| Target Milestone: | z2 | Keywords: | Triaged, ZStream |
| Target Release: | 15.0 (Stein) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | python-octavia-tests-tempest-1.1.1-0.20190826060430.7140479.el8ost | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-03-05 12:02:11 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0710 |
Description of problem: Test octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest.test_mixed_ipv4_ipv6_members_traffic fails. It times out validating HTTP connectivity to member server. Looking inside the member server, one can see the HTTP service going out of memory on launch. Version-Release number of selected component (if applicable): RHOS_TRUNK-15.0-RHEL-8-20190725.n.1 How reproducible: 100% Steps to Reproduce: 1. Deploy OSP 15 2. Run test test_mixed_ipv4_ipv6_members_traffic Actual results: 2019-08-02 09:36:01.347 695988 DEBUG octavia_tempest_plugin.tests.test_base [-] Octavia Setup: webserver2_ip = 10.2.2.135 resource_setup /usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/test_base.py:594 [10/1910] 2019-08-02 09:36:01.348 695988 DEBUG octavia_tempest_plugin.tests.test_base [-] Octavia Setup: webserver2_ipv6 = fd77:1457:4cf0:26a8::2db resource_setup /usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/test_base.py:596 2019-08-02 09:36:01.348 695988 DEBUG octavia_tempest_plugin.tests.test_base [-] Octavia Setup: webserver2_public_ip = 10.0.0.231 resource_setup /usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/test_base.py:598 2019-08-02 09:36:01.370 695988 INFO tempest.lib.common.ssh [-] Creating ssh connection to '10.0.0.226:22' as 'cirros' with public key authentication 2019-08-02 09:36:01.376 695988 INFO paramiko.transport [-] Connected (version 2.0, client dropbear_2015.67) 2019-08-02 09:36:01.456 695988 INFO paramiko.transport [-] Authentication (publickey) successful! 2019-08-02 09:36:01.457 695988 INFO tempest.lib.common.ssh [-] ssh connection to cirros.0.226 successfully created 2019-08-02 09:36:01.457 695988 DEBUG tempest.lib.common.utils.linux.remote_client [-] Remote command: set -eu -o pipefail; PATH=$PATH:/sbin; sudo ip address add fd7b:f9f7:fff:4eca::1e5/64 dev eth0 exec_command /usr/lib/python3.6/site-packages/tempest/lib/common/utils/linux/remote_client.py:107 2019-08-02 09:36:01.458 695988 INFO tempest.lib.common.ssh [-] Creating ssh connection to '10.0.0.226:22' as 'cirros' with public key authentication 2019-08-02 09:36:01.461 695988 INFO paramiko.transport [-] Connected (version 2.0, client dropbear_2015.67) 2019-08-02 09:36:01.579 695988 INFO paramiko.transport [-] Authentication (publickey) successful! 2019-08-02 09:36:01.580 695988 INFO tempest.lib.common.ssh [-] ssh connection to cirros.0.226 successfully created 2019-08-02 09:36:01.590 695988 INFO tempest.lib.common.ssh [-] Creating ssh connection to '10.0.0.231:22' as 'cirros' with public key authentication 2019-08-02 09:36:06.684 695988 WARNING tempest.lib.common.ssh [-] Failed to establish authenticated ssh connection to cirros.0.231 ([Errno None] Unable to connect to port 22 on 10.0.0.231). Number attempts: 1. Retry after 2 seconds.: paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 10.0.0.231 2019-08-02 09:36:09.188 695988 WARNING tempest.lib.common.ssh [-] Failed to establish authenticated ssh connection to cirros.0.231 ([Errno None] Unable to connect to port 22 on 10.0.0.231). Number attempts: 2. Retry after 3 seconds.: paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 10.0.0.231 2019-08-02 09:36:12.695 695988 INFO paramiko.transport [-] Connected (version 2.0, client dropbear_2015.67) 2019-08-02 09:36:12.801 695988 INFO paramiko.transport [-] Authentication (publickey) successful! 2019-08-02 09:36:12.802 695988 INFO tempest.lib.common.ssh [-] ssh connection to cirros.0.231 successfully created 2019-08-02 09:36:12.802 695988 DEBUG tempest.lib.common.utils.linux.remote_client [-] Remote command: set -eu -o pipefail; PATH=$PATH:/sbin; sudo ip address add fd77:1457:4cf0:26a8::2db/64 dev eth0 exec_command /usr/lib/python3.6/site-pac kages/tempest/lib/common/utils/linux/remote_client.py:107 2019-08-02 09:36:12.802 695988 INFO tempest.lib.common.ssh [-] Creating ssh connection to '10.0.0.231:22' as 'cirros' with public key authentication 2019-08-02 09:36:12.805 695988 INFO paramiko.transport [-] Connected (version 2.0, client dropbear_2015.67) 2019-08-02 09:36:12.908 695988 INFO paramiko.transport [-] Authentication (publickey) successful! 2019-08-02 09:36:12.909 695988 INFO tempest.lib.common.ssh [-] ssh connection to cirros.0.231 successfully created 2019-08-02 09:36:12.919 695988 INFO tempest.lib.common.ssh [-] Creating ssh connection to '10.0.0.226:22' as 'cirros' with public key authentication 2019-08-02 09:36:12.923 695988 INFO paramiko.transport [-] Connected (version 2.0, client dropbear_2015.67) 2019-08-02 09:36:13.027 695988 INFO paramiko.transport [-] Authentication (publickey) successful! 2019-08-02 09:36:13.028 695988 INFO tempest.lib.common.ssh [-] ssh connection to cirros.0.226 successfully created 2019-08-02 09:36:13.607 695988 DEBUG tempest.lib.common.utils.linux.remote_client [-] Remote command: set -eu -o pipefail; PATH=$PATH:/sbin; sudo screen -d -m /dev/shm/httpd.bin -port 80 -id 1 exec_command /usr/lib/python3.6/site-packages /tempest/lib/common/utils/linux/remote_client.py:107 2019-08-02 09:36:13.609 695988 INFO tempest.lib.common.ssh [-] Creating ssh connection to '10.0.0.226:22' as 'cirros' with public key authentication 2019-08-02 09:36:13.611 695988 INFO paramiko.transport [-] Connected (version 2.0, client dropbear_2015.67) 2019-08-02 09:36:13.674 695988 INFO paramiko.transport [-] Authentication (publickey) successful! 2019-08-02 09:36:13.674 695988 INFO tempest.lib.common.ssh [-] ssh connection to cirros.0.226 successfully created 2019-08-02 09:36:13.682 695988 DEBUG tempest.lib.common.utils.linux.remote_client [-] Remote command: set -eu -o pipefail; PATH=$PATH:/sbin; sudo screen -d -m /dev/shm/httpd.bin -port 81 -id 2 exec_command /usr/lib/python3.6/site-packages /tempest/lib/common/utils/linux/remote_client.py:107 2019-08-02 09:36:13.682 695988 INFO tempest.lib.common.ssh [-] Creating ssh connection to '10.0.0.226:22' as 'cirros' with public key authentication 2019-08-02 09:36:13.685 695988 INFO paramiko.transport [-] Connected (version 2.0, client dropbear_2015.67) 2019-08-02 09:36:13.787 695988 INFO paramiko.transport [-] Authentication (publickey) successful! 2019-08-02 09:36:13.787 695988 INFO tempest.lib.common.ssh [-] ssh connection to cirros.0.226 successfully created 2019-08-02 09:36:13.800 695988 INFO octavia_tempest_plugin.tests.validators [-] Validate URL got exception: HTTPConnectionPool(host='10.0.0.226', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connectio n.HTTPConnection object at 0x7fd6cb1f6550>: Failed to establish a new connection: [Errno 111] Connection refused',)). Retrying. 2019-08-02 09:36:18.809 695988 INFO octavia_tempest_plugin.tests.validators [-] Validate URL got exception: HTTPConnectionPool(host='10.0.0.226', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connectio n.HTTPConnection object at 0x7fd6c8976908>: Failed to establish a new connection: [Errno 111] Connection refused',)). Retrying. [...] 2019-08-02 09:41:09.275 695988 INFO octavia_tempest_plugin.tests.validators [-] Validate URL got exception: HTTPConnectionPool(host='10.0.0.226', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd6c8976668>: Failed to establish a new connection: [Errno 111] Connection refused',)). Retrying. 2019-08-02 09:41:14.281 695988 INFO tempest.test [-] <class 'tempest.lib.exceptions.TimeoutException'> raised in IPv6TrafficOperationsScenarioTest.setUpClass. Invoking tearDownClass. 2019-08-02 09:41:16.766 695988 INFO tempest.lib.common.rest_client [req-0f08f4c5-33f1-48bf-9ad6-f6de05012a3b ] Request (IPv6TrafficOperationsScenarioTest:tearDownClass): 204 DELETE http://10.0.0.147:9696/v2.0/floatingips/9ef6a874-54ca-459 b-bff5-9cb16023ec3a 2.483s Additional info: $ sudo /dev/shm/httpd.bin -port 80 -id 1 fatal error: runtime: out of memory runtime stack: runtime.throw(0x7ce5ca, 0x16) /usr/lib/golang/src/runtime/panic.go:617 +0x72 fp=0x7ffc22ffa740 sp=0x7ffc22ffa710 pc=0x42c812 runtime.sysMap(0xc000000000, 0x4000000, 0xc7d478) /usr/lib/golang/src/runtime/mem_linux.go:170 +0xc7 fp=0x7ffc22ffa780 sp=0x7ffc22ffa740 pc=0x418527 runtime.(*mheap).sysAlloc(0xc5e9c0, 0x2000, 0xc5e9d0, 0x1) /usr/lib/golang/src/runtime/malloc.go:633 +0x1cd fp=0x7ffc22ffa828 sp=0x7ffc22ffa780 pc=0x40ba4d runtime.(*mheap).grow(0xc5e9c0, 0x1, 0x0) /usr/lib/golang/src/runtime/mheap.go:1222 +0x42 fp=0x7ffc22ffa880 sp=0x7ffc22ffa828 pc=0x4256c2 runtime.(*mheap).allocSpanLocked(0xc5e9c0, 0x1, 0xc7d488, 0x0) /usr/lib/golang/src/runtime/mheap.go:1150 +0x37f fp=0x7ffc22ffa8b8 sp=0x7ffc22ffa880 pc=0x4255af runtime.(*mheap).alloc_m(0xc5e9c0, 0x1, 0x2a, 0x6e43a318) /usr/lib/golang/src/runtime/mheap.go:977 +0xc2 fp=0x7ffc22ffa908 sp=0x7ffc22ffa8b8 pc=0x424c02 runtime.(*mheap).alloc.func1() /usr/lib/golang/src/runtime/mheap.go:1048 +0x4c fp=0x7ffc22ffa940 sp=0x7ffc22ffa908 pc=0x45452c runtime.(*mheap).alloc(0xc5e9c0, 0x1, 0x1002a, 0x7ffc22ffa9e0) /usr/lib/golang/src/runtime/mheap.go:1047 +0x8a fp=0x7ffc22ffa990 sp=0x7ffc22ffa940 pc=0x424eda runtime.(*mcentral).grow(0xc5f7c0, 0x0) /usr/lib/golang/src/runtime/mcentral.go:256 +0x95 fp=0x7ffc22ffa9d8 sp=0x7ffc22ffa990 pc=0x417fa5 runtime.(*mcentral).cacheSpan(0xc5f7c0, 0x7fe885e47000) /usr/lib/golang/src/runtime/mcentral.go:106 +0x2ff fp=0x7ffc22ffaa38 sp=0x7ffc22ffa9d8 pc=0x417aaf runtime.(*mcache).refill(0x7fe885e47008, 0x2a) /usr/lib/golang/src/runtime/mcache.go:135 +0x86 fp=0x7ffc22ffaa58 sp=0x7ffc22ffaa38 pc=0x417546 runtime.(*mcache).nextFree(0x7fe885e47008, 0xc5832a, 0x7fe885e47008, 0x7fe885e47000, 0x8) /usr/lib/golang/src/runtime/malloc.go:786 +0x88 fp=0x7ffc22ffaa90 sp=0x7ffc22ffaa58 pc=0x40c288 runtime.mallocgc(0x180, 0x7c7ae0, 0x1, 0xc7d4e0) /usr/lib/golang/src/runtime/malloc.go:939 +0x76e fp=0x7ffc22ffab30 sp=0x7ffc22ffaa90 pc=0x40cb9e runtime.newobject(0x7c7ae0, 0x4000) /usr/lib/golang/src/runtime/malloc.go:1068 +0x38 fp=0x7ffc22ffab60 sp=0x7ffc22ffab30 pc=0x40cfa8 runtime.malg(0x5b00008000, 0xc61030) /usr/lib/golang/src/runtime/proc.go:3220 +0x31 fp=0x7ffc22ffaba0 sp=0x7ffc22ffab60 pc=0x435931 runtime.mpreinit(...) /usr/lib/golang/src/runtime/os_linux.go:311 runtime.mcommoninit(0xc59480) /usr/lib/golang/src/runtime/proc.go:618 +0xc2 fp=0x7ffc22ffabd8 sp=0x7ffc22ffaba0 pc=0x42f5b2 runtime.schedinit() /usr/lib/golang/src/runtime/proc.go:540 +0x74 fp=0x7ffc22ffac30 sp=0x7ffc22ffabd8 pc=0x42f244 runtime.rt0_go(0x7ffc22ffad88, 0x5, 0x7ffc22ffad88, 0x4004d8, 0x686101, 0x0, 0x500000000, 0x7ffc22ffad88, 0x4561c0, 0x0, ...) /usr/lib/golang/src/runtime/asm_amd64.s:195 +0x11a fp=0x7ffc22ffac38 sp=0x7ffc22ffac30 pc=0x4562ea $ free -m total used free shared buffers Mem: 46 43 3 6 3 -/+ buffers: 39 6 Swap: 0 0 0