Bug 1394744

Summary: TestSecurityGroupsBasicOps.test_port_security_disable_security_group test gets halted by giving request timed out.
Product: Red Hat OpenStack Reporter: Chandan Kumar <chkumar>
Component: openstack-ironic-inspectorAssignee: Dmitry Tantsur <dtantsur>
Status: CLOSED ERRATA QA Contact: Raviv Bar-Tal <rbartal>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 10.0 (Newton)CC: afazekas, apevec, dmellado, dtantsur, jschluet, lhh, mburns, mmagr, slinaber
Target Milestone: rcKeywords: AutomationBlocker, Triaged
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-ironic-inspector-4.2.0-3.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-14 16:32:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Chandan Kumar 2016-11-14 11:25:39 UTC
Description of problem:
TestSecurityGroupsBasicOps.test_port_security_disable_security_group tempest test gets hautled on undercloud by giving a Request timed out following by  Details: Command: 'set -eu -o pipefail; PATH=$PATH:/sbin; ping -c1 -w1 -s56 10.100.0.9' executed on host '10.0.0.216'.

Version-Release number of selected component (if applicable):
Ironic-inspector:
openstack-ironic-inspector-4.2.0-2.el7ost.noarch
python-ironic-inspector-client-1.9.0-1.el7ost.noarch
Tempest:
puppet-tempest-9.4.0-1.el7ost.noarch
openstack-tempest-13.0.0-4.bafe630git.el7ost.noarch
python-tempest-13.0.0-4.bafe630git.el7ost.noarch

Steps to Reproduce:
[1.] Deploy OpenStack using RHOP-10 director on RHEL-7.3
[2.] Install openstack-tempest-13.0.0-4.bafe630git.el7ost.noarch on undercloud
[3.] configure tempest then try to run tempest list-plugins, make sure ironic-inspector is visible
[4.] run ostestr -r 'TestSecurityGroupsBasicOps.test_port_security_disable_security_group'
you will get the following traceback:
[root@undercloud-0 ~]# ( cd tempest-dir  ;  ostestr -r 'TestSecurityGroupsBasicOps.test_port_security_disable_security_group' )
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmp7ShEYC
{0} tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_port_security_disable_security_group [335.122332s] ... FAILED
 
==============================
Failed 1 tests - output below:
==============================
 
tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_port_security_disable_security_group[compute,id-7c811dcc-263b-49a3-92d2-1b4d8405f50c,network]
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/root/tempest-dir/tempest/test.py", line 152, in wrapper
        return func(*func_args, **func_kwargs)
      File "/root/tempest-dir/tempest/test.py", line 107, in wrapper
        return f(self, *func_args, **func_kwargs)
      File "/root/tempest-dir/tempest/scenario/test_security_groups_basic_ops.py", line 614, in test_port_security_disable_security_group
        should_succeed=False)
      File "/root/tempest-dir/tempest/scenario/test_security_groups_basic_ops.py", line 377, in _check_connectivity
        should_succeed), msg)
      File "/root/tempest-dir/tempest/scenario/manager.py", line 926, in _check_remote_connectivity
        1)
      File "/root/tempest-dir/tempest/lib/common/utils/test_utils.py", line 103, in call_until_true
        if func():
      File "/root/tempest-dir/tempest/scenario/manager.py", line 917, in ping_remote
        source.ping_host(dest, nic=nic)
      File "/root/tempest-dir/tempest/common/utils/linux/remote_client.py", line 141, in ping_host
        return self.exec_command(cmd)
      File "/root/tempest-dir/tempest/common/utils/linux/remote_client.py", line 36, in wrapper
        return function(self, *args, **kwargs)
      File "/root/tempest-dir/tempest/common/utils/linux/remote_client.py", line 92, in exec_command
        return self.ssh_client.exec_command(cmd)
      File "/root/tempest-dir/tempest/lib/common/ssh.py", line 140, in exec_command
        cmd, self.host))
    tempest.lib.exceptions.TimeoutException: Request timed out
    Details: Command: 'set -eu -o pipefail; PATH=$PATH:/sbin; ping -c1 -w1 -s56 10.100.0.9' executed on host '10.0.0.216'.


[5.] On disabling ironic-inspector tempest plugin from /usr/lib/python2.7/site-packages/ironic_inspector-4.2.0-py2.7.egg-info/entry_points.txt by commenting #ironic_inspector_tests = ironic_inspector.test.inspector_tempest_plugin.plugin:InspectorTempestPlugin, this problem just vanished.

It appears that there is some issue with the ironic-inspector tempest plugin.

Comment 1 Attila Fazekas 2016-11-14 11:29:49 UTC
This issue prevents us to run the shipped tempest on the undercloud node, as we normally do.

Comment 2 Dmitry Tantsur 2016-11-14 11:37:01 UTC
> It appears that there is some issue with the ironic-inspector tempest plugin.

I don't see any signs of it, sorry.

This is a known issue with SSH code in tempest, mtreinish has tried to fix it previously, but his attempt broke other people. I'm not entirely sure why it exhibit itself when inspector plugin is enabled, but I don't see how we can help.

Comment 3 Dmitry Tantsur 2016-11-14 11:38:04 UTC
For the record: we had similar problems in upstream CI since https://review.openstack.org/#/c/358610/ was merged. Try reverting it to see if it helps.

Comment 4 Daniel Mellado 2016-11-14 11:41:07 UTC
Hey dmitry. What makes me wonder is *why* this shows only up when ironic-inspector plugin is enabled. Is there any code within the plugin that would touch any ssh related setting?

Comment 5 Dmitry Tantsur 2016-11-14 11:44:22 UTC
We don't seem to. Actually our plugin [1] is pretty simple, I've no idea in what way it can cause configuration changes. I also don't think there should be any configuration in Tempest breaking SSH. But FWIW tempest is black magic for me and most of upstream contributors, so we *might* be missing something.

[1] https://github.com/openstack/ironic-inspector/tree/master/ironic_inspector/test/inspector_tempest_plugin

Comment 6 Daniel Mellado 2016-11-18 09:17:12 UTC
Ironic-Inspector monkey-patching eventlet on the path to the tempest_plugin is the source of this issue.

https://github.com/openstack/ironic-inspector/blob/master/ironic_inspector/test/__init__.py

Tempest doesn't support the usage of eventlet and if the eventlet monkey patch happens late (plugin load time) it causes this kind of issues.

Comment 7 Daniel Mellado 2016-11-18 10:05:06 UTC
Removing the eventlet import affecting the ironic-inspector tempest plugin solves the issue affecting another tests. Reassigning the bug to ironic-inspector in order to track the packaging.

Comment 8 Jon Schlueter 2016-11-18 14:49:37 UTC
openstack-ironic-inspector-4.2.0-3.el7ost has the remove eventlet monkeypatch from test
openstack-tempest-13.0.0-5.bafe630git.el7ost has the "Delete duplicated dvr tests " patch in it

Comment 13 errata-xmlrpc 2016-12-14 16:32:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html