Bug 1951573

Summary: OSP16.1 undercloud-neutron-sanity-check validation traceback
Product: Red Hat OpenStack Reporter: David Rosenfeld <drosenfe>
Component: openstack-neutronAssignee: Rodolfo Alonso <ralonsoh>
Status: CLOSED ERRATA QA Contact: Eran Kuris <ekuris>
Severity: medium Docs Contact:
Priority: medium    
Version: 16.1 (Train)CC: akatz, bcafarel, chrisw, ebarrera, ralonsoh, renich, scohen, tdoucet, uemit.seren
Target Milestone: z7Keywords: Triaged
Target Release: 16.1 (Train on RHEL 8.2)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-neutron-15.2.1-1.20210614073305.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-12-09 20:18:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description David Rosenfeld 2021-04-20 13:45:50 UTC
Description of problem: When openstack tripleo validator show run is executed for undercloud-neutron-sanity-check a traceback is generated.

openstack tripleo validator show run 4bcd3973-620e-4efb-b5f5-c53337942424
{
    "hosts": {
        "undercloud": {
            "_ansible_no_log": false,
            "action": "command",
            "changed": false,
            "cmd": [
                "podman",
                "exec",
                "-u",
                "root",
                "neutron_ovs_agent",
                "/bin/bash",
                "-c",
                "neutron-sanity-check --config-file /etc/neutron/neutron.conf --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/metadata_agent.ini --config-file /etc/neutron/dhcp_agent.ini --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --config-file /etc/neutron/l3_agent.ini -d"
            ],
            "delta": "0:00:02.703837",
            "end": "2021-04-20 13:24:33.458337",
            "failed": true,
            "invocation": {
                "module_args": {
                    "_raw_params": "podman exec -u root neutron_ovs_agent /bin/bash -c 'neutron-sanity-check --config-file /etc/neutron/neutron.conf --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/metadata_agent.ini --config-file /etc/neutron/dhcp_agent.ini --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --config-file /etc/neutron/l3_agent.ini -d'\n",
                    "_uses_shell": false,
                    "argv": null,
                    "chdir": null,
                    "creates": null,
                    "executable": null,
                    "removes": null,
                    "stdin": null,
                    "stdin_add_newline": true,
                    "strip_empty_ends": true,
                    "warn": true
                }
            },
            "msg": "non-zero return code",
            "rc": 1,
            "start": "2021-04-20 13:24:30.754500",
            "stderr": "2021-04-20 13:24:33.014 1036427 INFO neutron.common.config [-] Logging enabled!\u001b[00m\n2021-04-20 13:24:33.014 1036427 INFO neutron.common.config [-] /usr/bin/neutron-sanity-check version 15.2.1.dev12\u001b[00m\n2021-04-20 13:24:33.077 1036427 CRITICAL neutron [-] Unhandled error: oslo_config.cfg.NoSuchOptError: no such option SRIOV_NIC in group [DEFAULT]\n2021-04-20 13:24:33.077 1036427 ERROR neutron Traceback (most recent call last):\n2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/lib/python3.6/site-packages/oslo_config/cfg.py\", line 2197, in __getattr__\n2021-04-20 13:24:33.077 1036427 ERROR neutron     return self._get(name)\n2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/lib/python3.6/site-packages/oslo_config/cfg.py\", line 2631, in _get\n2021-04-20 13:24:33.077 1036427 ERROR neutron     value, loc = self._do_get(name, group, namespace)\n2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/lib/python3.6/site-packages/oslo_config/cfg.py\", line 2649, in _do_get\n2021-04-20 13:24:33.077 1036427 ERROR neutron     info = self._get_opt_info(name, group)\n2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/lib/python3.6/site-packages/oslo_config/cfg.py\", line 2849, in _get_opt_info\n2021-04-20 13:24:33.077 1036427 ERROR neutron     raise NoSuchOptError(opt_name, group)\n2021-04-20 13:24:33.077 1036427 ERROR neutron oslo_config.cfg.NoSuchOptError: no such option SRIOV_NIC in group [DEFAULT]\n2021-04-20 13:24:33.077 1036427 ERROR neutron \n2021-04-20 13:24:33.077 1036427 ERROR neutron During handling of the above exception, another exception occurred:\n2021-04-20 13:24:33.077 1036427 ERROR neutron \n2021-04-20 13:24:33.077 1036427 ERROR neutron Traceback (most recent call last):\n2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/bin/neutron-sanity-check\", line 10, in <module>\n2021-04-20 13:24:33.077 1036427 ERROR neutron     sys.exit(main())\n2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/lib/python3.6/site-packages/neutron/cmd/sanity_check.py\", line 415, in main\n2021-04-20 13:24:33.077 1036427 ERROR neutron     enable_tests_from_config()\n2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/lib/python3.6/site-packages/neutron/cmd/sanity_check.py\", line 399, in enable_tests_from_config\n2021-04-20 13:24:33.077 1036427 ERROR neutron     if cfg.CONF.SRIOV_NIC.physical_device_mappings:\n2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/lib/python3.6/site-packages/oslo_config/cfg.py\", line 2201, in __getattr__\n2021-04-20 13:24:33.077 1036427 ERROR neutron     raise NoSuchOptError(name)\n2021-04-20 13:24:33.077 1036427 ERROR neutron oslo_config.cfg.NoSuchOptError: no such option SRIOV_NIC in group [DEFAULT]\n2021-04-20 13:24:33.077 1036427 ERROR neutron \u001b[00m\nError: non zero exit code: 1: OCI runtime error",
            "stderr_lines": [
                "2021-04-20 13:24:33.014 1036427 INFO neutron.common.config [-] Logging enabled!\u001b[00m",
                "2021-04-20 13:24:33.014 1036427 INFO neutron.common.config [-] /usr/bin/neutron-sanity-check version 15.2.1.dev12\u001b[00m",
                "2021-04-20 13:24:33.077 1036427 CRITICAL neutron [-] Unhandled error: oslo_config.cfg.NoSuchOptError: no such option SRIOV_NIC in group [DEFAULT]",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron Traceback (most recent call last):",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/lib/python3.6/site-packages/oslo_config/cfg.py\", line 2197, in __getattr__",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron     return self._get(name)",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/lib/python3.6/site-packages/oslo_config/cfg.py\", line 2631, in _get",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron     value, loc = self._do_get(name, group, namespace)",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/lib/python3.6/site-packages/oslo_config/cfg.py\", line 2649, in _do_get",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron     info = self._get_opt_info(name, group)",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/lib/python3.6/site-packages/oslo_config/cfg.py\", line 2849, in _get_opt_info",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron     raise NoSuchOptError(opt_name, group)",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron oslo_config.cfg.NoSuchOptError: no such option SRIOV_NIC in group [DEFAULT]",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron ",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron During handling of the above exception, another exception occurred:",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron ",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron Traceback (most recent call last):",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/bin/neutron-sanity-check\", line 10, in <module>",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron     sys.exit(main())",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/lib/python3.6/site-packages/neutron/cmd/sanity_check.py\", line 415, in main",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron     enable_tests_from_config()",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/lib/python3.6/site-packages/neutron/cmd/sanity_check.py\", line 399, in enable_tests_from_config",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron     if cfg.CONF.SRIOV_NIC.physical_device_mappings:",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/lib/python3.6/site-packages/oslo_config/cfg.py\", line 2201, in __getattr__",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron     raise NoSuchOptError(name)",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron oslo_config.cfg.NoSuchOptError: no such option SRIOV_NIC in group [DEFAULT]",
                "2021-04-20 13:24:33.077 1036427 ERROR neutron \u001b[00m",
                "Error: non zero exit code: 1: OCI runtime error"
            ],
            "stdout": "",
            "stdout_lines": []
        }
    },
    "name": "Run neutron-sanity-check",
    "status": "FAILED"
}
{
    "hosts": {
        "undercloud": {
            "_ansible_no_log": false,
            "action": "fail",
            "changed": false,
            "failed": true,
            "msg": "2021-04-20 13:24:33.077 1036427 ERROR neutron Traceback (most recent call last):\n2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/lib/python3.6/site-packages/oslo_config/cfg.py\", line 2197, in __getattr__\n2021-04-20 13:24:33.077 1036427 ERROR neutron     return self._get(name)\n2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/lib/python3.6/site-packages/oslo_config/cfg.py\", line 2631, in _get\n2021-04-20 13:24:33.077 1036427 ERROR neutron     value, loc = self._do_get(name, group, namespace)\n2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/lib/python3.6/site-packages/oslo_config/cfg.py\", line 2649, in _do_get\n2021-04-20 13:24:33.077 1036427 ERROR neutron     info = self._get_opt_info(name, group)\n2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/lib/python3.6/site-packages/oslo_config/cfg.py\", line 2849, in _get_opt_info\n2021-04-20 13:24:33.077 1036427 ERROR neutron     raise NoSuchOptError(opt_name, group)\n2021-04-20 13:24:33.077 1036427 ERROR neutron oslo_config.cfg.NoSuchOptError: no such option SRIOV_NIC in group [DEFAULT]\n2021-04-20 13:24:33.077 1036427 ERROR neutron \n2021-04-20 13:24:33.077 1036427 ERROR neutron During handling of the above exception, another exception occurred:\n2021-04-20 13:24:33.077 1036427 ERROR neutron \n2021-04-20 13:24:33.077 1036427 ERROR neutron Traceback (most recent call last):\n2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/bin/neutron-sanity-check\", line 10, in <module>\n2021-04-20 13:24:33.077 1036427 ERROR neutron     sys.exit(main())\n2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/lib/python3.6/site-packages/neutron/cmd/sanity_check.py\", line 415, in main\n2021-04-20 13:24:33.077 1036427 ERROR neutron     enable_tests_from_config()\n2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/lib/python3.6/site-packages/neutron/cmd/sanity_check.py\", line 399, in enable_tests_from_config\n2021-04-20 13:24:33.077 1036427 ERROR neutron     if cfg.CONF.SRIOV_NIC.physical_device_mappings:\n2021-04-20 13:24:33.077 1036427 ERROR neutron   File \"/usr/lib/python3.6/site-packages/oslo_config/cfg.py\", line 2201, in __getattr__\n2021-04-20 13:24:33.077 1036427 ERROR neutron     raise NoSuchOptError(name)\n2021-04-20 13:24:33.077 1036427 ERROR neutron oslo_config.cfg.NoSuchOptError: no such option SRIOV_NIC in group [DEFAULT]\n2021-04-20 13:24:33.077 1036427 ERROR neutron \u001b[00m"
        }
    },
    "name": "Fail",
    "status": "FAILED"
}

Version-Release number of selected component (if applicable): RHOS-16.1-RHEL-8-20210415.n.0 


How reproducible: Every time


Steps to Reproduce:
1. Execute openstack tripleo validator run --validation undercloud-neutron-sanity-check
2. Execute openstack tripleo validator show run for the UUID found in step 1
3.

Actual results: openstack tripleo validator show run completes with a traceback


Expected results: openstack tripleo validator show run completes without a traceback


Additional info:

Comment 1 Bernard Cafarelli 2021-04-26 14:51:38 UTC
Probably related to recent https://review.opendev.org/c/openstack/neutron/+/780023 as it mentions cfg.CONF.SRIOV_NIC and this validator passes on previous deployment 20210323.n.0

Comment 15 Uemit Seren 2021-09-20 09:32:47 UTC
We recently upgraded to OSP 16.2 and we are running into the same issue.

Comment 20 Tommy Doucet 2021-10-20 19:27:14 UTC
Customer on case 03062025 getting this issue with 16.2

Comment 22 Rodolfo Alonso 2021-11-18 09:42:03 UTC
Hello Tommy:

U/S patch https://review.opendev.org/q/I2a5e1fe3dbc6f2f342feaec92f4c122cfccce6d1 is now in D/S repository, in 16.2 version.

The fix will be in 16.2.2 release.

Regards.

Comment 31 errata-xmlrpc 2021-12-09 20:18:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.7 (Train) bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3762