Description of problem: 16.2/17 featureset001 job failing with "Exporting Tempest provider: Subunit File: testrepository.subunit (1)" As per build history of job, failure started on 27th Aug:- Looks like issue started after merge of:- https://review.opendev.org/c/openstack/tripleo-quickstart/+/804399 Version-Release number of selected component (if applicable): 16.2/17 How reproducible: Everytime in component/integration line CI Additional info: ~~~ 2021-08-29 23:48:56.227828 | primary | TASK [os_tempest : Collecting data from tempest run] *************************** 2021-08-29 23:48:56.227834 | primary | Sunday 29 August 2021 23:48:56 -0400 (0:00:00.464) 2:22:45.878 ********* 2021-08-29 23:48:57.128030 | primary | fatal: [undercloud]: FAILED! => { 2021-08-29 23:48:57.128064 | primary | "changed": true, 2021-08-29 23:48:57.128069 | primary | "cmd": ". /home/zuul/stackviz_venv/bin/activate\nstackviz-export -f /var/log/tempest/testrepository.subunit --env /var/log/tempest/stackviz/data\n", 2021-08-29 23:48:57.129997 | primary | "delta": "0:00:00.277828", 2021-08-29 23:48:57.130030 | primary | "end": "2021-08-29 23:48:57.094189", 2021-08-29 23:48:57.130037 | primary | "rc": 1, 2021-08-29 23:48:57.130041 | primary | "start": "2021-08-29 23:48:56.816361" 2021-08-29 23:48:57.130045 | primary | } 2021-08-29 23:48:57.130050 | primary | 2021-08-29 23:48:57.130054 | primary | STDOUT: 2021-08-29 23:48:57.130058 | primary | 2021-08-29 23:48:57.130062 | primary | Exporting Tempest provider: Subunit File: testrepository.subunit (1) 2021-08-29 23:48:57.130067 | primary | 2021-08-29 23:48:57.130071 | primary | 2021-08-29 23:48:57.130074 | primary | STDERR: 2021-08-29 23:48:57.130078 | primary | 2021-08-29 23:48:57.130082 | primary | Traceback (most recent call last): 2021-08-29 23:48:57.130095 | primary | File "/home/zuul/stackviz_venv/bin/stackviz-export", line 8, in <module> 2021-08-29 23:48:57.130101 | primary | sys.exit(main()) 2021-08-29 23:48:57.130106 | primary | File "/home/zuul/stackviz_venv/lib/python3.6/site-packages/stackviz/export.py", line 277, in main 2021-08-29 23:48:57.130111 | primary | artifacts.extend(export_tempest(provider, args.path, args.gzip)) 2021-08-29 23:48:57.130115 | primary | File "/home/zuul/stackviz_venv/lib/python3.6/site-packages/stackviz/export.py", line 182, in export_tempest 2021-08-29 23:48:57.130120 | primary | output_dir, prefix, compress) 2021-08-29 23:48:57.130124 | primary | File "/home/zuul/stackviz_venv/lib/python3.6/site-packages/stackviz/export.py", line 85, in export_tempest_raw 2021-08-29 23:48:57.135835 | primary | converted = tempest_subunit.convert_stream(subunit, strip_details=True) 2021-08-29 23:48:57.135898 | primary | File "/home/zuul/stackviz_venv/lib/python3.6/site-packages/stackviz/parser/tempest_subunit.py", line 274, in convert_stream 2021-08-29 23:48:57.135905 | primary | result_stream.run(result) 2021-08-29 23:48:57.135911 | primary | File "/home/zuul/stackviz_venv/lib/python3.6/site-packages/subunit/v2.py", line 313, in run 2021-08-29 23:48:57.135916 | primary | raise Exception("Non subunit content", content) 2021-08-29 23:48:57.135921 | primary | Exception: ('Non subunit content', b"'") 2021-08-29 23:48:57.135926 | primary | ~~~
I don't see how this is related to tempest ... first this bug should have gone to upstream bug tracker, it has nothing to do with downstream openstack-tempest component. From the logs I can see that the issue happened during openstack-ansible-os_tempest execution so this should be reported there. Based on the readme: https://opendev.org/openstack/openstack-ansible-os_tempest The bug tracker for openstack-ansible-os_tempest is: https://bugs.launchpad.net/openstack-ansible The linked traceback shows failing stackviz due to missing subunit data - the subunit is missing because tempest wasn't executed successfully in a prior step: 2021-08-29 23:48:46.529733 | primary | TASK [os_tempest : Execute tempest tests] ************************************** 2021-08-29 23:48:46.529739 | primary | Sunday 29 August 2021 23:48:46 -0400 (0:00:00.040) 2:22:36.183 ********* 2021-08-29 23:48:50.102045 | primary | fatal: [undercloud]: FAILED! => { 2021-08-29 23:48:50.102087 | primary | "changed": false, 2021-08-29 23:48:50.103138 | primary | "cmd": "set -e\nif [ -d /openstack/venvs/tempest-untagged/bin ];\nthen\n. /openstack/venvs/tempest-untagged/bin/activate\nfi\ntempest run --concurrency 4 --exclude-list /home/zuul/tempest/etc/tempest_excludelist.txt --include-list /home/zuul/tempest/etc/tempest_includelist.txt > /var/log/tempest/tempest_run.log\n", 2021-08-29 23:48:50.104147 | primary | "delta": "0:00:03.244151", 2021-08-29 23:48:50.104160 | primary | "end": "2021-08-29 23:48:50.077525", 2021-08-29 23:48:50.105033 | primary | "rc": 1, 2021-08-29 23:48:50.105058 | primary | "start": "2021-08-29 23:48:46.833374" 2021-08-29 23:48:50.105063 | primary | } 2021-08-29 23:48:50.105068 | primary | 2021-08-29 23:48:50.105072 | primary | STDERR: 2021-08-29 23:48:50.105076 | primary | 2021-08-29 23:48:50.105080 | primary | [Errno 2] No such file or directory: '/home/zuul/tempest/etc/tempest_includelist.txt' 2021-08-29 23:48:50.105085 | primary | 2021-08-29 23:48:50.105089 | primary | 2021-08-29 23:48:50.105093 | primary | MSG: 2021-08-29 23:48:50.105097 | primary | 2021-08-29 23:48:50.105103 | primary | non-zero return code Seems like the openstack-ansible-os_tempest expects having tempest_includelist.txt in /home/zuul/tempest/etc dir, however, there is just the exclude file there - https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-baremetal-rhos-17/d0bf974/logs/undercloud/home/zuul/tempest/etc/ I don't see any latest changes in os_tempest which would go near these parameters so I assume the real issue is somewhere even before os_tempest is called - the job definition maybe? I'm closing this as NOT A BUG due to above mentioned reasons.