Bug 1704478 - switch-vlan validation fails on OSP15
Summary: switch-vlan validation fails on OSP15
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-validations
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: beta
: 15.0 (Stein)
Assignee: Gaël Chamoulaud
QA Contact: nlevinki
URL:
Whiteboard:
Depends On: 1710324
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-29 21:37 UTC by Alexander Chuzhoy
Modified: 2019-09-26 10:49 UTC (History)
10 users (show)

Fixed In Version: openstack-tripleo-validations-10.4.1-0.20190621220403.673148d.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-09-21 11:21:38 UTC
Target Upstream Version:


Attachments (Terms of Use)
debug_dir (62.87 KB, application/gzip)
2019-05-02 14:10 UTC, Alexander Chuzhoy
no flags Details
new debug_dir (768.70 KB, application/gzip)
2019-05-08 15:48 UTC, Alexander Chuzhoy
no flags Details


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 658821 0 None MERGED [Python3] Adds bytes to str conversion management 2020-02-26 14:26:06 UTC
OpenStack gerrit 661725 0 None MERGED [Python3] Adds bytes to str conversion management 2020-02-26 14:26:06 UTC
OpenStack gerrit 666603 0 None MERGED Use sorted dictionary before iteration 2020-02-26 14:26:07 UTC
OpenStack gerrit 666801 0 None MERGED Use sorted dictionary before iteration 2020-02-26 14:26:07 UTC
Red Hat Product Errata RHEA-2019:2811 0 None None None 2019-09-21 11:21:53 UTC

Description Alexander Chuzhoy 2019-04-29 21:37:00 UTC
switch-vlan validation fails on OSP15


Environment:
openstack-tripleo-validations-10.4.1-0.20190426070346.e774685.el8ost.noarch



Running:
ansible-playbook -vvv -i /usr/bin/tripleo-ansible-inventory /usr/share/openstack-tripleo-validations/validations/switch-vlans.yaml



Fails with "Error: too many values to unpack (expected 2)":

TASK [Check that switch vlans are present if used in nic-config files] ***********************************************************************************************************************
task path: /usr/share/openstack-tripleo-validations/validations/switch-vlans.yaml:25
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: stack
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /tmp/ansible-${USER}/ansible-tmp-1556573739.0759747-104081225904581 `" && echo ansible-tmp-1556573739.0759747-104081225904581="` echo /tmp/ansible-${USER}/ansible-tmp-1556573739.0759747-104081225904581 `" ) && sleep 0'                                                                                                      
Using module file /usr/share/openstack-tripleo-validations/validations/library/switch_vlans.py
<localhost> PUT /home/stack/.ansible/tmp/ansible-local-84625ucc885v0/tmp6vm33zfo TO /tmp/ansible-stack/ansible-tmp-1556573739.0759747-104081225904581/AnsiballZ_switch_vlans.py
<localhost> EXEC /bin/sh -c 'chmod u+x /tmp/ansible-stack/ansible-tmp-1556573739.0759747-104081225904581/ /tmp/ansible-stack/ansible-tmp-1556573739.0759747-104081225904581/AnsiballZ_switch_vlans.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3 /tmp/ansible-stack/ansible-tmp-1556573739.0759747-104081225904581/AnsiballZ_switch_vlans.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /tmp/ansible-stack/ansible-tmp-1556573739.0759747-104081225904581/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible-stack/ansible-tmp-1556573739.0759747-104081225904581/AnsiballZ_switch_vlans.py", line 114, in <module>
    _ansiballz_main()
  File "/tmp/ansible-stack/ansible-tmp-1556573739.0759747-104081225904581/AnsiballZ_switch_vlans.py", line 106, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/tmp/ansible-stack/ansible-tmp-1556573739.0759747-104081225904581/AnsiballZ_switch_vlans.py", line 49, in invoke_module
    imp.load_module('__main__', mod, module, MOD_DESC)
  File "/usr/lib64/python3.6/imp.py", line 235, in load_module
    return load_source(name, filename, file)
  File "/usr/lib64/python3.6/imp.py", line 170, in load_source
    module = _exec(spec, sys.modules[name])
  File "<frozen importlib._bootstrap>", line 618, in _exec
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/ansible_switch_vlans_payload_dwc0vyxv/__main__.py", line 225, in <module>
  File "/tmp/ansible_switch_vlans_payload_dwc0vyxv/__main__.py", line 209, in main
  File "/tmp/ansible_switch_vlans_payload_dwc0vyxv/__main__.py", line 208, in <dictcomp>
ValueError: too many values to unpack (expected 2)

fatal: [undercloud]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible-stack/ansible-tmp-1556573739.0759747-104081225904581/AnsiballZ_switch_vlans.py\", line 114, in <module>\n    _ansiballz_main()\n  File \"/tmp/ansible-stack/ansible-tmp-1556573739.0759747-104081225904581/AnsiballZ_switch_vlans.py\", line 106, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/tmp/ansible-stack/ansible-tmp-1556573739.0759747-104081225904581/AnsiballZ_switch_vlans.py\", line 49, in invoke_module\n    imp.load_module('__main__', mod, module, MOD_DESC)\n  File \"/usr/lib64/python3.6/imp.py\", line 235, in load_module\n    return load_source(name, filename, file)\n  File \"/usr/lib64/python3.6/imp.py\", line 170, in load_source\n    module = _exec(spec, sys.modules[name])\n  File \"<frozen importlib._bootstrap>\", line 618, in _exec\n  File \"<frozen importlib._bootstrap_external>\", line 678, in exec_module\n  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n  File \"/tmp/ansible_switch_vlans_payload_dwc0vyxv/__main__.py\", line 225, in <module>\n  File \"/tmp/ansible_switch_vlans_payload_dwc0vyxv/__main__.py\", line 209, in main\n  File \"/tmp/ansible_switch_vlans_payload_dwc0vyxv/__main__.py\", line 208, in <dictcomp>\nValueError: too many values to unpack (expected 2)\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

PLAY RECAP ***********************************************************************************************************************************************************************************
undercloud                 : ok=3    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

(undercloud) [stack@undercloud ~]$

Comment 1 Bob Fournier 2019-04-30 00:36:37 UTC
Moving to validations team to take a look.

Comment 2 Gaël Chamoulaud 2019-05-02 12:49:51 UTC
Alexander,

Could you please run this?

$ export ANSIBLE_KEEP_REMOTE_FILES=1
$ ansible-playbook -vvvv -i /usr/bin/tripleo-ansible-inventory /usr/share/openstack-tripleo-validations/validations/switch-vlans.yaml

Take note of the directory Ansible used to store modules on the host.
In the log you pasted in your comment #1, it should be this line,

<localhost> PUT /home/stack/.ansible/tmp/ansible-local-84625ucc885v0/tmp6vm33zfo TO **/tmp/ansible-stack/ansible-tmp-1556573739.0759747-104081225904581/AnsiballZ_switch_vlans.py**

Navigate to the directory you noted,

$ cd /tmp/ansible-stack/ansible-tmp-1556573739.0759747-104081225904581/

Run the wrapper’s explode command to turn the string into some python files that I can work with for debugging:

$ python3 /tmp/ansible-stack/ansible-tmp-1556573739.0759747-104081225904581/AnsiballZ_switch_vlans.py explode
$ cd debug_dir
$ ll
total 5412
drwxr-xr-x. 3 root root      65 Apr 30 13:41 ansible
-rw-r--r--. 1 root root    8179 Apr 30 14:24 ansible_module_switch_vlans.py
-rw-r--r--. 1 root root 5532748 Apr 30 13:40 args

Now that you should have a debug_dir created, could you please make a tarball and adding it as an attachment to this BZs

Thanks for your help,
Gaël

Comment 3 Alexander Chuzhoy 2019-05-02 14:10:05 UTC
Created attachment 1561680 [details]
debug_dir

Comment 4 Gaël Chamoulaud 2019-05-03 09:10:44 UTC
Hi Alexander,

You did the right process but on the wrong ansible task.

Could you please repeat the same process but with the right path coming from the failing task which call the switch_vlans.py module (ie TASK [Check that switch vlans are present if used in nic-config files])?

Thanks
Gaël

Comment 5 Alexander Chuzhoy 2019-05-08 15:48:24 UTC
This is the error:
TASK [Check that switch vlans are present if used in nic-config files] ***********************************************************************************************************************
task path: /usr/share/openstack-tripleo-validations/validations/switch-vlans.yaml:25
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: stack
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /tmp/ansible-${USER}/ansible-tmp-1557330056.4472654-224543919452709 `" && echo ansible-tmp-1557330056.4472654-224543919452709="` echo /tmp/ansible-${USER}/ansible-tmp-1557330056.4472654-224543919452709 `" ) && sleep 0'                                                                                                      
Using module file /usr/share/openstack-tripleo-validations/validations/library/switch_vlans.py
<localhost> PUT /home/stack/.ansible/tmp/ansible-local-6143332xsiz3bj/tmpow8oae8u TO /tmp/ansible-stack/ansible-tmp-1557330056.4472654-224543919452709/AnsiballZ_switch_vlans.py
<localhost> EXEC /bin/sh -c 'chmod u+x /tmp/ansible-stack/ansible-tmp-1557330056.4472654-224543919452709/ /tmp/ansible-stack/ansible-tmp-1557330056.4472654-224543919452709/AnsiballZ_switch_vlans.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3 /tmp/ansible-stack/ansible-tmp-1557330056.4472654-224543919452709/AnsiballZ_switch_vlans.py && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible-stack/ansible-tmp-1557330056.4472654-224543919452709/AnsiballZ_switch_vlans.py", line 262, in <module>
    _ansiballz_main()
  File "/tmp/ansible-stack/ansible-tmp-1557330056.4472654-224543919452709/AnsiballZ_switch_vlans.py", line 252, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/tmp/ansible-stack/ansible-tmp-1557330056.4472654-224543919452709/AnsiballZ_switch_vlans.py", line 120, in invoke_module
    imp.load_module('__main__', mod, module, MOD_DESC)
  File "/usr/lib64/python3.6/imp.py", line 235, in load_module
    return load_source(name, filename, file)
  File "/usr/lib64/python3.6/imp.py", line 170, in load_source
    module = _exec(spec, sys.modules[name])
  File "<frozen importlib._bootstrap>", line 618, in _exec
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/ansible_switch_vlans_payload_t9qfd9o2/__main__.py", line 225, in <module>
  File "/tmp/ansible_switch_vlans_payload_t9qfd9o2/__main__.py", line 209, in main
  File "/tmp/ansible_switch_vlans_payload_t9qfd9o2/__main__.py", line 208, in <dictcomp>
ValueError: too many values to unpack (expected 2)

fatal: [undercloud]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible-stack/ansible-tmp-1557330056.4472654-224543919452709/AnsiballZ_switch_vlans.py\", line 262, in <module>\n    _ansiballz_main()\n  File \"/tmp/ansible-stack/ansible-tmp-1557330056.4472654-224543919452709/AnsiballZ_switch_vlans.py\", line 252, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/tmp/ansible-stack/ansible-tmp-1557330056.4472654-224543919452709/AnsiballZ_switch_vlans.py\", line 120, in invoke_module\n    imp.load_module('__main__', mod, module, MOD_DESC)\n  File \"/usr/lib64/python3.6/imp.py\", line 235, in load_module\n    return load_source(name, filename, file)\n  File \"/usr/lib64/python3.6/imp.py\", line 170, in load_source\n    module = _exec(spec, sys.modules[name])\n  File \"<frozen importlib._bootstrap>\", line 618, in _exec\n  File \"<frozen importlib._bootstrap_external>\", line 678, in exec_module\n  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n  File \"/tmp/ansible_switch_vlans_payload_t9qfd9o2/__main__.py\", line 225, in <module>\n  File \"/tmp/ansible_switch_vlans_payload_t9qfd9o2/__main__.py\", line 209, in main\n  File \"/tmp/ansible_switch_vlans_payload_t9qfd9o2/__main__.py\", line 208, in <dictcomp>\nValueError: too many values to unpack (expected 2)\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

PLAY RECAP ***********************************************************************************************************************************************************************************
undercloud                 : ok=3    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   





So going to use:
/tmp/ansible-stack/ansible-tmp-1557330056.4472654-224543919452709/AnsiballZ_switch_vlans.py

Comment 6 Alexander Chuzhoy 2019-05-08 15:48:48 UTC
Created attachment 1565719 [details]
new debug_dir

Comment 7 Gaël Chamoulaud 2019-05-13 10:27:41 UTC
Here is a first statement about what happens here.

The culprit is the custom ansible lookup plugin called tht.py which should return
a beautiful list of tuples containing each files and their contents from the overcloud
swift container.

It well returns a list of something but not a list of tuples, that's why the error 
"ValueError: too many values to unpack (expected 2)"

It seems to be related to Python3 on RHEL 8. Trying to debug it and finding out a
proper solution ASAP.

Comment 12 Alexander Chuzhoy 2019-06-17 13:53:31 UTC
FailedQA

Environment:
openstack-tripleo-validations-10.4.1-0.20190608000407.ff86911.el8ost.noarch


  source /home/stack/stackrc
  export TRIPLEO_PLAN_NAME=my-templates
  ansible-playbook -vvv -i /usr/bin/tripleo-ansible-inventory /usr/share/openstack-tripleo-validations/validations/switch-vlans.yaml


TASK [Check that switch vlans are present if used in nic-config files] ********************************************************************************************************************************************
task path: /usr/share/openstack-tripleo-validations/validations/switch-vlans.yaml:25
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: stack
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /tmp/ansible-${USER}/ansible-tmp-1560779449.07787-35710612521861 `" && echo ansible-tmp-1560779449.07787-35710612521861="` echo /tmp/ansible-${USER}/ansible-tmp-1560779449.07787-35710612521861 `" ) && sleep 0'                                                                                                                                                         
Using module file /usr/share/openstack-tripleo-validations/validations/library/switch_vlans.py
<localhost> PUT /home/stack/.ansible/tmp/ansible-local-454071ta56ev4/tmp5cebg91c TO /tmp/ansible-stack/ansible-tmp-1560779449.07787-35710612521861/AnsiballZ_switch_vlans.py
<localhost> EXEC /bin/sh -c 'chmod u+x /tmp/ansible-stack/ansible-tmp-1560779449.07787-35710612521861/ /tmp/ansible-stack/ansible-tmp-1560779449.07787-35710612521861/AnsiballZ_switch_vlans.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3 /tmp/ansible-stack/ansible-tmp-1560779449.07787-35710612521861/AnsiballZ_switch_vlans.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /tmp/ansible-stack/ansible-tmp-1560779449.07787-35710612521861/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible-stack/ansible-tmp-1560779449.07787-35710612521861/AnsiballZ_switch_vlans.py", line 114, in <module>
    _ansiballz_main()
  File "/tmp/ansible-stack/ansible-tmp-1560779449.07787-35710612521861/AnsiballZ_switch_vlans.py", line 106, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/tmp/ansible-stack/ansible-tmp-1560779449.07787-35710612521861/AnsiballZ_switch_vlans.py", line 49, in invoke_module
    imp.load_module('__main__', mod, module, MOD_DESC)
  File "/usr/lib64/python3.6/imp.py", line 235, in load_module
    return load_source(name, filename, file)
  File "/usr/lib64/python3.6/imp.py", line 170, in load_source
    module = _exec(spec, sys.modules[name])
  File "<frozen importlib._bootstrap>", line 618, in _exec
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/ansible_switch_vlans_payload_kr3f9l85/__main__.py", line 225, in <module>
  File "/tmp/ansible_switch_vlans_payload_kr3f9l85/__main__.py", line 209, in main
  File "/tmp/ansible_switch_vlans_payload_kr3f9l85/__main__.py", line 208, in <dictcomp>
ValueError: too many values to unpack (expected 2)

fatal: [undercloud]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible-stack/ansible-tmp-1560779449.07787-35710612521861/AnsiballZ_switch_vlans.py\", line 114, in <module>\n    _ansiballz_main()\n  File \"/tmp/ansible-stack/ansible-tmp-1560779449.07787-35710612521861/AnsiballZ_switch_vlans.py\", line 106, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/tmp/ansible-stack/ansible-tmp-1560779449.07787-35710612521861/AnsiballZ_switch_vlans.py\", line 49, in invoke_module\n    imp.load_module('__main__', mod, module, MOD_DESC)\n  File \"/usr/lib64/python3.6/imp.py\", line 235, in load_module\n    return load_source(name, filename, file)\n  File \"/usr/lib64/python3.6/imp.py\", line 170, in load_source\n    module = _exec(spec, sys.modules[name])\n  File \"<frozen importlib._bootstrap>\", line 618, in _exec\n  File \"<frozen importlib._bootstrap_external>\", line 678, in exec_module\n  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n  File \"/tmp/ansible_switch_vlans_payload_kr3f9l85/__main__.py\", line 225, in <module>\n  File \"/tmp/ansible_switch_vlans_payload_kr3f9l85/__main__.py\", line 209, in main\n  File \"/tmp/ansible_switch_vlans_payload_kr3f9l85/__main__.py\", line 208, in <dictcomp>\nValueError: too many values to unpack (expected 2)\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

PLAY RECAP ********************************************************************************************************************************************************************************************************
undercloud                 : ok=3    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

(undercloud) [stack@undercloud temp1]$

Comment 13 Gaël Chamoulaud 2019-06-17 14:19:19 UTC
Hi Alexander, 

Sorry I didn't mention it sooner that during the Stein cycle we refactored all the tripleo-validations architecture.

All the contents of the validations directory are now obsolete and will be removed soon!

All the validations have been migrated into Ansible roles (/usr/share/openstack-tripleo-validations/roles/) and
you can run the playbooks from /usr/share/openstack-tripleo-validations/playbooks/.

Please use the run-validations.sh script which should be present in /bin of the undercloud. Otherwise, you can grab it 
here https://github.com/openstack/tripleo-validations/blob/master/scripts/run-validations.sh and run:

```
$ ./run-validations.sh --plan my-templates --validation-name switch-vlans
```

Sorry for the confusion!

Gaël

Comment 16 Alexander Chuzhoy 2019-06-29 02:41:14 UTC
Verified:

Environment: openstack-tripleo-validations-10.4.1-0.20190626140421.fae8a85.el8ost.noarch

The reported issue doesn't reproduce.
+ ansible-playbook -vvv -i /usr/bin/tripleo-ansible-inventory /usr/share/openstack-tripleo-validations/validations/switch-vlans.yaml
+ grep 'VLAN ID 193 not on attached switch'
    "msg": "VLAN ID 193 not on attached switch\nVLAN ID 193 not on attached switch\nVLAN ID 193 not on attached switch"
+ '[' 0 '!=' 0 ']'

Comment 20 errata-xmlrpc 2019-09-21 11:21:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811


Note You need to log in before you can comment on or make changes to this bug.