Bug 1646195 - Undercloud Neutron Sanity Check validations doesn't work anymore
Summary: Undercloud Neutron Sanity Check validations doesn't work anymore
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-validations
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 14.0 (Rocky)
Assignee: Florian Fuchs
QA Contact: Andrey Fiodorov
URL:
Whiteboard:
: 1656101 1656106 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-11-05 11:04 UTC by Gaël Chamoulaud
Modified: 2019-03-18 13:03 UTC (History)
9 users (show)

Fixed In Version: openstack-tripleo-validations-9.3.1-0.20181008110760.4064fb7.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-03-18 13:03:13 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 623266 0 None None None 2018-12-07 14:55:26 UTC
OpenStack gerrit 625703 0 None None None 2019-01-17 11:11:08 UTC
OpenStack gerrit 628406 0 None None None 2019-01-04 11:38:39 UTC
OpenStack gerrit 631470 0 None None None 2019-01-17 11:12:19 UTC
Red Hat Product Errata RHBA-2019:0446 0 None None None 2019-03-18 13:03:22 UTC

Description Gaël Chamoulaud 2018-11-05 11:04:11 UTC
neutron-sanity-check script is present on the neutron containers but not anymore on the undercloud.

TASK [Run neutron-sanity-check] *******************************************************************************************************************************
task path: /usr/share/openstack-tripleo-validations/validations/undercloud-neutron-sanity-check.yaml:46
Using module file /usr/share/openstack-tripleo-validations/validations/library/neutron_sanity_check.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: stack
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /tmp/ansible-${USER}/ansible-tmp-1541415651.39-13156262441100 `" && echo ansible-tmp-1541415651.39-13156262441100="` echo /tmp/ansible-${USER}/ansible-tmp-1541415651.39-13156262441100 `" ) && sleep 0'
<localhost> PUT /home/stack/.ansible/tmp/ansible-local-992849hce7Md/tmpCnn3PU TO /tmp/ansible-stack/ansible-tmp-1541415651.39-13156262441100/neutron_sanity_check.py
<localhost> EXEC /bin/sh -c 'chmod u+x /tmp/ansible-stack/ansible-tmp-1541415651.39-13156262441100/ /tmp/ansible-stack/ansible-tmp-1541415651.39-13156262441100/neutron_sanity_check.py && sleep 0'
<localhost> EXEC /bin/sh -c 'sudo -H -S -n -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-mwscfpdljgmimuiqgcrhtffpxydbelbt; /usr/bin/python /tmp/ansible-stack/ansible-tmp-1541415651.39-13156262441100/neutron_sanity_check.py'"'"' && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /tmp/ansible-stack/ansible-tmp-1541415651.39-13156262441100/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_Jq42EX/ansible_module_neutron_sanity_check.py", line 100, in <module>
    main()
  File "/tmp/ansible_Jq42EX/ansible_module_neutron_sanity_check.py", line 79, in main
    stderr=subprocess.PIPE)
  File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

fatal: [undercloud]: FAILED! => {
    "changed": false, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_Jq42EX/ansible_module_neutron_sanity_check.py\", line 100, in <module>\n    main()\n  File \"/tmp/ansible_Jq42EX/ansible_module_neutron_sanity_check.py\", line 79, in main\n    stderr=subprocess.PIPE)\n  File \"/usr/lib64/python2.7/subprocess.py\", line 711, in __init__\n    errread, errwrite)\n  File \"/usr/lib64/python2.7/subprocess.py\", line 1327, in _execute_child\n    raise child_exception\nOSError: [Errno 2] No such file or directory\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "rc": 1
}

PLAY RECAP ****************************************************************************************************************************************************
undercloud                 : ok=1    changed=0    unreachable=0    failed=1

Comment 1 Florian Fuchs 2018-12-04 22:33:03 UTC
*** Bug 1656101 has been marked as a duplicate of this bug. ***

Comment 2 Florian Fuchs 2018-12-04 22:33:38 UTC
*** Bug 1656106 has been marked as a duplicate of this bug. ***

Comment 13 Andrey Fiodorov 2019-01-04 10:22:52 UTC
Still fails...

Task 'Run neutron-sanity-check' failed:
Host: controller-2
Message: Command `docker exec -u root neutron_ovs_agent /bin/bash -c neutron-sanity-check --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini` exited with code: 1: non-zero return code

stderr:
    2019-01-04 10:19:09.199 534131 INFO neutron.common.config [-] Logging enabled![00m
    2019-01-04 10:19:09.200 534131 INFO neutron.common.config [-] /usr/bin/neutron-sanity-check version 13.0.2.dev14[00m
    2019-01-04 10:19:09.589 534131 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', 'privsep-helper', '--config-file', '/etc/neutron/plugins/ml2/openvswitch_agent.ini', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmpKaxhjO/privsep.sock'][00m
    2019-01-04 10:19:10.018 534131 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap[00m
    2019-01-04 10:19:09.975 534304 INFO oslo.privsep.daemon [-] privsep daemon starting[00m
    2019-01-04 10:19:09.981 534304 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0[00m
    2019-01-04 10:19:09.986 534304 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_NET_ADMIN|CAP_SYS_ADMIN/CAP_NET_ADMIN|CAP_SYS_ADMIN/none[00m
    2019-01-04 10:19:09.986 534304 INFO oslo.privsep.daemon [-] privsep daemon running as pid 534304[00m
    2019-01-04 10:19:10.361 534131 ERROR neutron.agent.linux.utils [-] Exit code: 255; Stdin: ; Stdout: ; Stderr: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-arptables: No such file or directory
    [00m
    2019-01-04 10:19:10.362 534131 ERROR neutron.cmd.sanity_check [-] Bridge firewalling is not enabled. It may be the case that bridge and/or br_netfilter kernel modules are not loaded. Alternatively, corresponding sysctl settings may be overridden to disable it by default.[00m

Task 'Run neutron-sanity-check' failed:
Host: controller-1
Message: Command `docker exec -u root neutron_ovs_agent /bin/bash -c neutron-sanity-check --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini` exited with code: 1: non-zero return code

stderr:
    2019-01-04 10:19:09.922 531116 INFO neutron.common.config [-] Logging enabled![00m
    2019-01-04 10:19:09.923 531116 INFO neutron.common.config [-] /usr/bin/neutron-sanity-check version 13.0.2.dev14[00m
    2019-01-04 10:19:10.264 531116 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', 'privsep-helper', '--config-file', '/etc/neutron/plugins/ml2/openvswitch_agent.ini', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmp58SIjd/privsep.sock'][00m
    2019-01-04 10:19:10.717 531116 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap[00m
    2019-01-04 10:19:10.672 531687 INFO oslo.privsep.daemon [-] privsep daemon starting[00m
    2019-01-04 10:19:10.679 531687 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0[00m
    2019-01-04 10:19:10.685 531687 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_NET_ADMIN|CAP_SYS_ADMIN/CAP_NET_ADMIN|CAP_SYS_ADMIN/none[00m
    2019-01-04 10:19:10.685 531687 INFO oslo.privsep.daemon [-] privsep daemon running as pid 531687[00m
    2019-01-04 10:19:11.030 531116 ERROR neutron.agent.linux.utils [-] Exit code: 255; Stdin: ; Stdout: ; Stderr: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-arptables: No such file or directory
    [00m
    2019-01-04 10:19:11.031 531116 ERROR neutron.cmd.sanity_check [-] Bridge firewalling is not enabled. It may be the case that bridge and/or br_netfilter kernel modules are not loaded. Alternatively, corresponding sysctl settings may be overridden to disable it by default.[00m

Task 'Run neutron-sanity-check' failed:
Host: controller-0
Message: Command `docker exec -u root neutron_ovs_agent /bin/bash -c neutron-sanity-check --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini` exited with code: 1: non-zero return code

stderr:
    2019-01-04 10:19:09.542 544191 INFO neutron.common.config [-] Logging enabled![00m
    2019-01-04 10:19:09.542 544191 INFO neutron.common.config [-] /usr/bin/neutron-sanity-check version 13.0.2.dev14[00m
    2019-01-04 10:19:09.942 544191 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', 'privsep-helper', '--config-file', '/etc/neutron/plugins/ml2/openvswitch_agent.ini', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmp0OgAJg/privsep.sock'][00m
    2019-01-04 10:19:10.405 544191 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap[00m
    2019-01-04 10:19:10.353 544515 INFO oslo.privsep.daemon [-] privsep daemon starting[00m
    2019-01-04 10:19:10.357 544515 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0[00m
    2019-01-04 10:19:10.360 544515 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_NET_ADMIN|CAP_SYS_ADMIN/CAP_NET_ADMIN|CAP_SYS_ADMIN/none[00m
    2019-01-04 10:19:10.360 544515 INFO oslo.privsep.daemon [-] privsep daemon running as pid 544515[00m
    2019-01-04 10:19:10.751 544191 ERROR neutron.agent.linux.utils [-] Exit code: 255; Stdin: ; Stdout: ; Stderr: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-arptables: No such file or directory
    [00m
    2019-01-04 10:19:10.752 544191 ERROR neutron.cmd.sanity_check [-] Bridge firewalling is not enabled. It may be the case that bridge and/or br_netfilter kernel modules are not loaded. Alternatively, corresponding sysctl settings may be overridden to disable it by default.[00m

Task 'Fail' failed:
Host: controller-0
Message: 2019-01-04 10:19:10.751 544191 ERROR neutron.agent.linux.utils [-] Exit code: 255; Stdin: ; Stdout: ; Stderr: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-arptables: No such file or directory
2019-01-04 10:19:10.752 544191 ERROR neutron.cmd.sanity_check [-] Bridge firewalling is not enabled. It may be the case that bridge and/or br_netfilter kernel modules are not loaded. Alternatively, corresponding sysctl settings may be overridden to disable it by default.[00m

Task 'Fail' failed:
Host: controller-1
Message: 2019-01-04 10:19:11.030 531116 ERROR neutron.agent.linux.utils [-] Exit code: 255; Stdin: ; Stdout: ; Stderr: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-arptables: No such file or directory
2019-01-04 10:19:11.031 531116 ERROR neutron.cmd.sanity_check [-] Bridge firewalling is not enabled. It may be the case that bridge and/or br_netfilter kernel modules are not loaded. Alternatively, corresponding sysctl settings may be overridden to disable it by default.[00m

Task 'Fail' failed:
Host: controller-2
Message: 2019-01-04 10:19:10.361 534131 ERROR neutron.agent.linux.utils [-] Exit code: 255; Stdin: ; Stdout: ; Stderr: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-arptables: No such file or directory
2019-01-04 10:19:10.362 534131 ERROR neutron.cmd.sanity_check [-] Bridge firewalling is not enabled. It may be the case that bridge and/or br_netfilter kernel modules are not loaded. Alternatively, corresponding sysctl settings may be overridden to disable it by default.[00m

Failure! The validation failed for all hosts:
* controller-0
* controller-1
* controller-2

Comment 19 Mikey Ariel 2019-02-20 12:44:15 UTC
If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text.

If this bug does not require doc text, please set the 'requires_doc_text' flag to -.

Comment 20 Andrey Fiodorov 2019-02-21 15:15:07 UTC
Verified.

[stack@undercloud-0 ~]$ rpm -qa openstack-tripleo-validations
openstack-tripleo-validations-9.3.1-0.20190119052820.f400181.el7ost.noarch

Comment 22 errata-xmlrpc 2019-03-18 13:03:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0446


Note You need to log in before you can comment on or make changes to this bug.