Bug 1999107

Summary: 16.2/17 featureset001 job failing with "Exporting Tempest provider: Subunit File: testrepository.subunit (1)"
Product: Red Hat OpenStack Reporter: Sandeep Yadav <sandyada>
Component: openstack-tempestAssignee: Chandan Kumar <chkumar>
Status: CLOSED NOTABUG QA Contact: Martin Kopec <mkopec>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 16.2 (Train)CC: apevec, lhh, slinaber, udesale
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-08-30 18:30:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sandeep Yadav 2021-08-30 12:39:13 UTC 
Description of problem:

16.2/17 featureset001 job failing with "Exporting Tempest provider: Subunit File: testrepository.subunit (1)"

As per build history of job, failure started on 27th Aug:-


Looks like issue started after merge of:-

https://review.opendev.org/c/openstack/tripleo-quickstart/+/804399

Version-Release number of selected component (if applicable):

16.2/17 

How reproducible:

Everytime in component/integration line CI


Additional info:

~~~
2021-08-29 23:48:56.227828 | primary | TASK [os_tempest : Collecting data from tempest run] ***************************
2021-08-29 23:48:56.227834 | primary | Sunday 29 August 2021  23:48:56 -0400 (0:00:00.464)       2:22:45.878 *********
2021-08-29 23:48:57.128030 | primary | fatal: [undercloud]: FAILED! => {
2021-08-29 23:48:57.128064 | primary |     "changed": true,
2021-08-29 23:48:57.128069 | primary |     "cmd": ". /home/zuul/stackviz_venv/bin/activate\nstackviz-export -f /var/log/tempest/testrepository.subunit --env /var/log/tempest/stackviz/data\n",
2021-08-29 23:48:57.129997 | primary |     "delta": "0:00:00.277828",
2021-08-29 23:48:57.130030 | primary |     "end": "2021-08-29 23:48:57.094189",
2021-08-29 23:48:57.130037 | primary |     "rc": 1,
2021-08-29 23:48:57.130041 | primary |     "start": "2021-08-29 23:48:56.816361"
2021-08-29 23:48:57.130045 | primary | }
2021-08-29 23:48:57.130050 | primary |
2021-08-29 23:48:57.130054 | primary | STDOUT:
2021-08-29 23:48:57.130058 | primary |
2021-08-29 23:48:57.130062 | primary | Exporting Tempest provider: Subunit File: testrepository.subunit (1)
2021-08-29 23:48:57.130067 | primary |
2021-08-29 23:48:57.130071 | primary |
2021-08-29 23:48:57.130074 | primary | STDERR:
2021-08-29 23:48:57.130078 | primary |
2021-08-29 23:48:57.130082 | primary | Traceback (most recent call last):
2021-08-29 23:48:57.130095 | primary |   File "/home/zuul/stackviz_venv/bin/stackviz-export", line 8, in <module>
2021-08-29 23:48:57.130101 | primary |     sys.exit(main())
2021-08-29 23:48:57.130106 | primary |   File "/home/zuul/stackviz_venv/lib/python3.6/site-packages/stackviz/export.py", line 277, in main
2021-08-29 23:48:57.130111 | primary |     artifacts.extend(export_tempest(provider, args.path, args.gzip))
2021-08-29 23:48:57.130115 | primary |   File "/home/zuul/stackviz_venv/lib/python3.6/site-packages/stackviz/export.py", line 182, in export_tempest
2021-08-29 23:48:57.130120 | primary |     output_dir, prefix, compress)
2021-08-29 23:48:57.130124 | primary |   File "/home/zuul/stackviz_venv/lib/python3.6/site-packages/stackviz/export.py", line 85, in export_tempest_raw
2021-08-29 23:48:57.135835 | primary |     converted = tempest_subunit.convert_stream(subunit, strip_details=True)
2021-08-29 23:48:57.135898 | primary |   File "/home/zuul/stackviz_venv/lib/python3.6/site-packages/stackviz/parser/tempest_subunit.py", line 274, in convert_stream
2021-08-29 23:48:57.135905 | primary |     result_stream.run(result)
2021-08-29 23:48:57.135911 | primary |   File "/home/zuul/stackviz_venv/lib/python3.6/site-packages/subunit/v2.py", line 313, in run
2021-08-29 23:48:57.135916 | primary |     raise Exception("Non subunit content", content)
2021-08-29 23:48:57.135921 | primary | Exception: ('Non subunit content', b"'")
2021-08-29 23:48:57.135926 | primary |
~~~
Comment 2 Martin Kopec 2021-08-30 18:30:00 UTC 
I don't see how this is related to tempest ... first this bug should have gone to upstream bug tracker, it has nothing to do with downstream openstack-tempest component.

From the logs I can see that the issue happened during openstack-ansible-os_tempest execution so this should be reported there.
Based on the readme: https://opendev.org/openstack/openstack-ansible-os_tempest
The bug tracker for openstack-ansible-os_tempest is: https://bugs.launchpad.net/openstack-ansible

The linked traceback shows failing stackviz due to missing subunit data - the subunit is missing because tempest wasn't executed successfully in a prior step:

2021-08-29 23:48:46.529733 | primary | TASK [os_tempest : Execute tempest tests] **************************************
2021-08-29 23:48:46.529739 | primary | Sunday 29 August 2021  23:48:46 -0400 (0:00:00.040)       2:22:36.183 *********
2021-08-29 23:48:50.102045 | primary | fatal: [undercloud]: FAILED! => {
2021-08-29 23:48:50.102087 | primary |     "changed": false,
2021-08-29 23:48:50.103138 | primary |     "cmd": "set -e\nif [ -d /openstack/venvs/tempest-untagged/bin ];\nthen\n. /openstack/venvs/tempest-untagged/bin/activate\nfi\ntempest run   --concurrency 4   --exclude-list /home/zuul/tempest/etc/tempest_excludelist.txt   --include-list /home/zuul/tempest/etc/tempest_includelist.txt > /var/log/tempest/tempest_run.log\n",
2021-08-29 23:48:50.104147 | primary |     "delta": "0:00:03.244151",
2021-08-29 23:48:50.104160 | primary |     "end": "2021-08-29 23:48:50.077525",
2021-08-29 23:48:50.105033 | primary |     "rc": 1,
2021-08-29 23:48:50.105058 | primary |     "start": "2021-08-29 23:48:46.833374"
2021-08-29 23:48:50.105063 | primary | }
2021-08-29 23:48:50.105068 | primary |
2021-08-29 23:48:50.105072 | primary | STDERR:
2021-08-29 23:48:50.105076 | primary |
2021-08-29 23:48:50.105080 | primary | [Errno 2] No such file or directory: '/home/zuul/tempest/etc/tempest_includelist.txt'
2021-08-29 23:48:50.105085 | primary |
2021-08-29 23:48:50.105089 | primary |
2021-08-29 23:48:50.105093 | primary | MSG:
2021-08-29 23:48:50.105097 | primary |
2021-08-29 23:48:50.105103 | primary | non-zero return code


Seems like the openstack-ansible-os_tempest expects having tempest_includelist.txt in /home/zuul/tempest/etc dir, however, there is just the exclude file there - https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-baremetal-rhos-17/d0bf974/logs/undercloud/home/zuul/tempest/etc/

I don't see any latest changes in os_tempest which would go near these parameters so I assume the real issue is somewhere even before os_tempest is called - the job definition maybe? 

I'm closing this as NOT A BUG due to above mentioned reasons.