Bug 1728646

Summary: Functional tests are failing on RHEL8 with python36 (subunit.parser error with stestr-3)
Product: Red Hat OpenStack Reporter: Michal Pryc <mpryc>
Component: openstack-novaAssignee: Michal Pryc <mpryc>
Status: CLOSED EOL QA Contact: OSP DFG:Compute <osp-dfg-compute>
Severity: medium Docs Contact:
Priority: high    
Version: 15.0 (Stein)CC: cyril, dasmith, eglynn, jhakimra, jschluet, kchamart, lyarwood, mbooth, sbauza, sgordon, stchen, vromanso
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-09-30 20:05:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Michal Pryc 2019-07-10 10:56:50 UTC
Functional tests on OSP15 with stestr-3 on RHEL8 are failing with subunit error:

ft1.1: subunit.parsertesttools.testresult.real._StringException: Binary content:
  Packet data (application/octet-stream)

Parser Error: {{{Bad checksum - calculated (0x4d492833), stored (0x3031392d)}}}
ft1.2: subunit.parsertesttools.testresult.real._StringException: lost connection during failure report of test 'subunit.parser'


This looks similar to upstream https://bugs.launchpad.net/nova/+bug/1832164, however backporting only this patch to OSP15 is not resolving issue.

The root cause is too much information in Exception causing subunit to fail with that amount of stderr.

In upstream there were number of patches to reduce such stderr deprecation warnings, e.g.:

https://bugs.launchpad.net/nova/+bug/1813147

https://bugs.launchpad.net/nova/+bug/1629047

Comment 1 Michal Pryc 2019-07-11 09:09:31 UTC
Those are top deprecation warning messages:

-----

b'/usr/lib/python3.6/site-packages/paste/deploy/loadwsgi.py:22: DeprecationWarning: Parameters to load are deprecated.  Call .resolve and .require separately.'
b'  return pkg_resources.EntryPoint.parse("x=" + s).load(False)'



----- This one is fixed by codeeng 175704-----

b'/home/zuul/src/code.engineering.redhat.com/nova/nova/db/sqlalchemy/api.py:1879: SADeprecationWarning: The joinedload_all() function is deprecated, and will be removed in a future release.  Please use method chaining with joinedload() instead'



----- Below two can be 'fixed' by downgrading SQLAlchemy to version 1.2.18 -----

b'/usr/lib/python3.6/site-packages/migrate/changeset/databases/visitor.py:76: SADeprecationWarning: The Engine.contextual_connect() method is deprecated.  This method is an artifact of the threadlocal engine strategy which is also to be deprecated.   For explicit connections from an Engine, use the Engine.connect() method.'

-----

b'/usr/lib64/python3.6/site-packages/sqlalchemy/orm/base.py:220: SADeprecationWarning: The Query.with_lockmode() method is deprecated and will be removed in a future release.  Please refer to Query.with_for_update(). '

Comment 2 Michal Pryc 2019-07-11 14:25:06 UTC
There were number of failed runs and from what I can see the common pattern is that for all failed tests the following one was inprogress:


nova.tests.functional.notification_sample_tests.test_instance.TestInstanceNotificationSampleOldAttachFlow.test_instance_action

Comment 3 Michal Pryc 2019-07-11 14:52:56 UTC
Looks like removing test_instance_action causes tests to pass, which means the following upstream fix did not resolve issue completely:

The following commit was trying to fix the issue:
   https://github.com/openstack/nova/commit/93027713ab614b59ebb66f0617f85d43c956427c

Comment 6 Cyril Roelandt 2020-03-23 15:43:06 UTC
We have a similar issue in Glance, and Abhishek Kekane told me that the recently released subunit 1.4.0 fixes this issue.

There is already a patch in RDO to start using this new version in U (https://review.rdoproject.org/r/#/c/26033/).

Comment 7 stchen 2020-09-30 20:05:15 UTC
Closing EOL, OSP 15 has been retired as of Sept 19, 2020